Large language models possess remarkable capabilities in language comprehension, code generation, and various other tasks. However, a crucial question arises: How well can they abstract problem-solving rules and algorithms? More importantly, how can we effectively test this ability? A research by the London Institute for Mathematical Sciences (LIMS), conducted using Nebius infrastructure, helps to understand the causes of LLM imperfections.

Let’s consider a simple example: the classic blocks world planning problem. This problem involves swapping colored blocks stacked on the tabletop. The task is to list the moves required to change a current state to a target state. We add a constraint similar to manipulating real blocks: at each step, you can only remove the top block and set it aside, or return a previously set-aside block to the top. When presented with this problem in natural language, GPT-4 achieves a modest success rate of about 40%.

Now, imagine we transform the concept of blocks, colors, and actions into something entirely different — say, elements from a fantasy world. The blocks become magical objects, colors represent various magical properties, and actions are spells. The fundamental task remains unchanged; we’ve merely relabeled the objects, properties, and actions. Surprisingly, GPT-4’s performance plummets dramatically to around 1% in this scenario.

If we take it a step further and replace the names with arbitrary character sequences of the same length, the performance drops to zero. Other models exhibit similar results.

This example highlights a reasoning weakness of LLMs where they fail, and humans significantly outperform AI. People can abstract the solution algorithm and apply it even to meaningless objects and actions. The ability to abstract is particularly crucial in mathematics. For instance, when working with a function of a variable, we aim to compute it for any value, not just those used during model training.

The blocks world example raises concerns about the capabilities of modern large language models in solving problems, which require reasoning and abstract thinking. Intrigued by this, researchers at LIMS sought to test how well the architecture of large language models can abstract rules. This inquiry focuses on the transformer architecture — the foundation of modern LLMs, which we’ve discussed in one of the previous blog posts. The LIMS scientists tackled this challenge using Nebius AI.

Elementary cellular automata emerged as an ideal testing tool for this purpose. These strictly defined mathematical objects are primitive enough that applying learning algorithms to them allows us to understand two things. First, can the model abstract the task when dealing with such a simple object? And second, if it can, how does it do it — in other words, how does the neural network learn to “distill” the rules?

A cellular automaton operates on a simple principle:

Why is this process relevant to the problem we formulated above? The middle part of the animation describes the rule by which the cellular automaton operates (which can be abstracted). We see the first row and the second row derived from it using the given rule. This process continues: the third row is generated from the second using the same rule, and so on, creating a sequence of binary strings that describe the automaton’s temporal evolution.

A well-known example of such a sequence is the Sierpinski triangle:

In the triangle’s first row, only one symbol is shaded. To form this image from a series of rows, a simple rule is applied:

Other examples exist, including those where the image evolution becomes seemingly unpredictable — each subsequent row appears to be generated randomly:

However, even in these cases, the rule remains strictly deterministic, illustrating the mathematical concept of “deterministic chaos.”

Now that we understand how cellular automata work, let’s explore the model chosen by our LIMS colleagues for testing and the principles behind its training.

They selected BERT, the first truly significant encoder transformer model, as the most suitable candidate. While less advanced than current models, BERT is ideal for testing whether transformers can abstract tasks.

Let’s examine a sequence generated by a rule with a radius of 1 (r1), where values are added within a distance of 1 from the central cell, resulting in 3 input variables:

Here, the rule 1100111 and row t = 0 are randomly generated, and each subsequent row up to row 5 is derived from the previous one by applying chosen rule. In LIMS’s first task, the BERT model is trained to predict the rule from these six consequent states. With 6 rows total, each 84 digits long, this variant is dubbed r1s84T6. Such dataset allows for 256 rules in total, presenting a relatively simple computational challenge.

LIMS researchers then increased the complexity. In the r2s84T6 task, they expanded the radius to 2, resulting in 5 variables:

This expansion allows for approximately 4 billion possible rules.

Pushing further, when the radius is increased to 3 with 7 variables, the number of possible rules skyrockets to about 10³⁸. The space of potential rules grows combinatorically. Consider the third variant, r3s24T20:

In this setup, the model trains on about a million different rules and is then tested on rules absent from the training set. During analysis, the model encounters each rule once in the training set but never sees them in the test set.

Here are some of the samples used for training:

Can you, by looking at these samples, try to guess which rule has been used to generate them? This shows the complexity of the task.

The results obtained by LIMS are shown on the graph below:

Getting a validation accuracy of more than 0.93, we can see that BERT can indeed successfully predict the rule, synthesizing from input data a program that generates this data. This implies that more advanced modern transformer neural networks are also capable of such abstraction.

While the blocks world example mentioned at the beginning remains valid, we now understand that LLMs’ inability to abstract tasks isn’t due to their architecture. Rather, it’s a consequence of their training specifics. We’re pleased that this scientifically significant result was achieved using the Nebius AI infrastructure, contributing to our understanding of AI capabilities and limitations.

How large models can abstract rules: a research by LIMS was originally published in Nebius on Medium, where people are continuing the conversation by highlighting and responding to this story.

The global network of ML engineers is divided into two parts: industrial and academic. The flow of information, methods of interaction, and especially events follow unique unspoken rules in different parts. If you belong to the first world, then the second may seem distant and even irrelevant to you.

Indeed, it is possible to build AI products for end users without venturing into research territory. However, understanding the second world and at least occasionally participating in AI research conferences will boost your perspective, make you a more versatile specialist, and give you new ideas that can be directly applied in your work.

This article will help ML engineers from companies learn how the world of AI research is structured. We will cover all the main topics, including articles, conferences, datasets, and the transfer of scientific ideas into services.

For researchers in the field of computer science (which includes not only the AI domain, of course), the primary indicator of success is the publication of their work at one of the top international conferences. This represents the initial “checkpoint” in gaining recognition for their research. For example, key AI conferences traditionally include the International Conference on Machine Learning (ICML) and the Conference on Neural Information Processing Systems (NeurIPS, formerly known as NIPS). For the most interesting and relevant ones in 2024, see the list curated by ML experts at Nebius AI. There are also many conferences dedicated to specific areas within ML, such as computer vision, information retrieval, speech technology, machine translation, and others.

The many pros of sharing

Those outside the computer science field might believe that the most valuable ideas should be kept secret to capitalize on their uniqueness. However, the reality is quite the opposite. A scientist’s reputation is judged by the significance of their research and how frequently their papers are cited by others (the citation index). This is a crucial aspect of their career. Researchers advance in their professional hierarchy and earn respect in their community only if they consistently produce influential work that is published, widely recognized, and forms the foundation for the work of other scientists. One example of such foundational, highly-cited study is this paper on the EfficientNet family of models, presented at ICML 2019 (over 13,000 citations).

Many top studies, and possibly the majority, are the result of international collaborations between researchers from various universities and companies around the world. A significant and highly valued milestone in a researcher’s career is when they gain the experience to independently identify and evaluate ideas. However, even beyond this point, the invaluable assistance of colleagues remains essential. Scientists aid each other in refining ideas and co-author papers. The greater a researcher’s contribution to science, the easier it becomes for them to connect with like-minded peers.

Lastly, due to the vast density and accessibility of information today, different researchers often independently develop very similar (and indeed valuable) scientific ideas simultaneously. If you don’t publish your idea, it’s almost certain that someone else will. The “winner” in these cases is usually not the one who conceived the innovation slightly earlier, but rather the one who published it first. Or it could be the one who presented the idea in the most comprehensive, clear, and persuasive manner.

Describing an achievement in text and code

An academic article is constructed around the core idea proposed by the researcher, which represents their contribution to computer science. The article begins with a description of this idea, succinctly formulated in a few sentences. Following this is an introduction that outlines the range of problems addressed by the proposed innovation. The description and introduction are generally written in simple language, understandable to a broad audience. After the introduction, it’s necessary to formalize the presented problems in mathematical language and introduce precise definitions.

The researcher then must present a clear and comprehensive explanation of the essence of their innovation, highlighting how it differs from previous, similar methods. All theoretical expositions must be supported by references to existing proofs or independently verified by the author. This might involve certain assumptions, such as providing a proof for a scenario where there is an infinite amount of training data (clearly an unachievable situation) or where the data is completely independent. Nearing the end of the paper, the researcher discusses the experimental results they have achieved. Here’s a great example of a paper that follows this structure, presented at NeurIPS 2019 by the developers of PyTorch.

For an article to be favorably received by reviewers invited by conference organizers, it must possess one or more key attributes. The most critical factor increasing the likelihood of approval is the scientific novelty of the proposed idea.

Often, this novelty is measured against existing ideas, and it is the responsibility of the article’s author, not the reviewers, to make this assessment. Ideally, the author should thoroughly discuss existing methods in the article and, if feasible, demonstrate how they are special cases of the proposed method. By doing so, the scientist demonstrates that conventional approaches are not always sufficient, having expanded and proposed a more comprehensive, adaptable, and thus more effective theoretical framework. If the novelty is indisputable, reviewers tend to be less stringent in their assessment of other aspects, such as overlooking poor English.

To reinforce the novelty, it’s best to include comparisons with existing methods using one or several open datasets that are recognized in the academic community. Examples of such datasets include the ImageNet image repository and databases from institutions like the Modified National Institute of Standards and Technology (MNIST) and the Canadian Institute For Advanced Research (CIFAR). A complication arises because these “academic” datasets often differ structurally from real-world data encountered in the industry. As a result, the effectiveness of the proposed method can vary with different datasets. Researchers, especially those with ties to the industry, are mindful of this discrepancy and might include disclaimers indicating different outcomes on their proprietary data versus publicly available datasets.

Occasionally, a method is so specifically tailored to an open database that it underperforms with real-world data. A solution to this prevalent issue is the introduction of new, more representative datasets. However, this often involves private content that companies cannot legally disclose. In such cases, they may resort to data anonymization, removing identifiable details like faces and numbers in photographs. Furthermore, for a dataset to be widely accepted and become a benchmark in the scientific community, it requires more than just making it available; it necessitates a separate, well-cited publication detailing its merits and advantages.

When open datasets are absent in the research topic, reviewers are left to take the author’s results at face value. Theoretically, an author could exaggerate these results and remain undetected, but such practices are unlikely in the academic community, where the majority of scientists are genuinely committed to advancing science.

In several ML fields it’s also customary to attach code links (usually to GitHub) to papers. For instance, one of the 2022 OpenAI papers on language models using human feedback refers to a corresponding InstructGPT page on GitHub. The papers themselves usually contain little or no code, or just pseudocode. Here again, complications arise if the paper is written by a researcher from a company rather than a university, as corporate or startup code is by default protected by NDA. Researchers and their colleagues often have to put in significant effort to separate the code related to the described idea from internal and private repositories.

The likelihood of publication also depends on the relevance of the chosen subject. Relevance is largely dictated by products and services: if a corporation or startup is interested in developing a new service or improving an existing one based on the idea from the paper, it’s always a huge advantage.

As previously mentioned, papers in computer science are rarely written solo. However, typically one of the authors invests much more time and effort than the others, making their contribution to the scientific novelty the greatest. This individual is listed first among the authors, and in future citations of the paper, only their name might be mentioned (for example, “Smith et al, ” which translates from Latin as “Smith and others”). Nevertheless, the contributions of the other authors are also extremely valuable — otherwise, it would be impossible for them to be listed as authors.

Peer review

Submissions for articles usually close several months prior to the conference. After an article is submitted, reviewers typically have 3–5 weeks to read, evaluate, and comment on it. This process follows either a single-blind system, where the authors do not see the reviewers’ names, or a double-blind system, where the reviewers also do not see the authors’ names. The latter is considered more impartial: various studies have demonstrated that an author’s reputation can influence a reviewer’s decision. For instance, they might assume that a scientist with numerous published articles is inherently deserving of a higher evaluation.

Even in a double-blind review, however, a reviewer can often guess the author if they are in the same field. Additionally, the article may already be published in the arXiv database — the largest repository of scientific papers — by the time it undergoes review. Conference organizers do not prohibit this, but they recommend using a different title and abstract for the arXiv publication. Nonetheless, if the article has been posted there, locating it is still relatively straightforward.

Evaluating an article always inovles multiple reviewers. One of them is assigned the role of a meta-reviewer, whose job is to overview the verdicts of their colleagues and make the final decision. If there’s a divergence in the reviewers’ assessments, the meta-reviewer may also read the paper for a more comprehensive understanding.

Sometimes, after reviewing the ratings and comments, authors have the opportunity to engage in discussion with the reviewer; there’s even a chance to persuade them to change their decision (though this system is not in place at all conferences, and influencing the final verdict is even rarer). In these discussions, authors cannot refer to other scientific works, except for those already cited in their paper. The goal is simply to help the reviewer better understand the content of the article.

Where to publish

Computer science articles are more commonly submitted to conferences than to scientific journals. The reason is that journal publications have more stringent requirements and the review process can take months or even years. Computer science is a rapidly evolving field, so authors are usually unwilling to wait that long for publication. However, a paper accepted at a conference can later be expanded (e.g., with more detailed results) and published in a journal where there are less strict limitations on length.

What to expect when visiting

The format of the authors’ presence for accepted papers at the conference is determined by the reviewers. If a paper is given the green light, authors are most often allotted a space for a poster presentation. A poster is a static slide summarizing the paper with illustrations. Parts of the conference hall are filled with long rows of poster stands. Authors spend a significant amount of time near their posters, interacting with scientists interested in their work.

A slightly more prestigious form of participation is a quick presentation, or a “lightning talk.” If reviewers deem the paper worthy of a lightning talk, the author is given about three minutes to speak in front of a broad audience. On one hand, a lightning talk is a great opportunity to present the idea not only to those who showed interest in the poster but also to a wider audience. On the other hand, visitors who approach the poster are typically more prepared and deeply immersed in your specific topic than the average listener in the hall. Therefore, during a lightning talk, it’s crucial to efficiently introduce the audience to the subject matter.

Usually, at the end of their lightning talk, authors mention their poster number, enabling interested listeners to find it and gain a deeper understanding of the paper.

And finally, the most prestigious option is a combination of a poster and a full presentation of the idea, allowing for a more relaxed and detailed discussion.

Participating in conferences as an author of an accepted paper leads to impressive results. For instance, in 2010, Yutaka Matsuo, a distinguished AI researcher from Japan, developed an algorithm capable of detecting early earthquake signs by monitoring Twitter mentions of tremors. He showcased this work at the annual WWW conference, significantly boosting his career. In 2019, he became the first AI specialist added to the board of the Japanese technology giant Softbank, reflecting the impact of his study and the attention it garnered from major IT corporations.

However, scientists, including paper authors, attend conferences not solely for self-promotion. Firstly, they naturally seek out posters related to their field for obvious reasons. Secondly, it’s important for them to expand their contact list with the aim of future collaborative academic work. This is not hunting — or at least, it’s just the very first stage of what is typically followed by a mutually beneficial exchange of ideas, findings, and joint work on one or several papers.

At the same time, productive networking at a top conference is challenging due to the total lack of free time. If a scientist still has energy left after a full day spent at lectures and discussions around posters, and has overcome jet lag, they might head to one of the numerous parties. These are often organized by corporations and, as a result, these gatherings sometimes have more of a hunting vibe. However, many attendees use these events not for job hunting but for networking. In the evening, with no lectures or posters, it’s easier to “catch” the specialist you’re interested in. As an example, OpenAI hosted a party during the days of NeurIPS 2022.

Computer science (again, including AI/ML) is one of the few fields where the interests of corporations and startups are closely intertwined with the academic realm. At conferences like NeurIPS, ICML, and others, numerous industry professionals attend, not just those from universities. This is typical for computer science but quite the opposite for most other sciences.

Perhaps the most cited example of a scientific work whose ideas quickly found application in services is ‘Attention Is All You Need’ from 2017. This paper marked the beginning of the famous transformers, which underlie many modern AI products — we recently discussed this story in detail. The paper was presented at NIPS 2017, later renamed to NeurIPS.

On the other hand, not all ideas presented in papers are immediately applied to the creation or improvement of services. The cycle of transferring into production can vary and sometimes spans decades. For instance, convolutional neural networks, whose roots go all the way back to 1980, only gained real prevalence in the 2010s. Their use continues today: for example, Midjourney utilizes this type of network, among others. There might not be such long ‘breaks’ anymore: advancements in the ML field, as you know, were too long slowed down by a lack of computing power and data. The current pace of development seems unlikely to allow such a situation to recur, but it’s impossible to say for sure: for example, quantum computing has not yet reached the average consumer.

Even within a single company, a researcher may propose a scientifically groundbreaking idea to their service team colleagues and be denied implementation for various reasons. The famous Xerox PARC laboratory made numerous incredible discoveries in the 70s and 80s, but the parent company did not always use these innovations in products. As a result, for example, the windowed GUI, largely invented at PARC, was ‘borrowed’ and more rapidly integrated into operating systems by Apple and Microsoft. Returning to the topic of modern machine learning, one of the reasons why it’s challenging to implement ideas in services is the (already mentioned) discrepancy between ‘academic’ and actual real-world datasets. Additionally, bringing the idea to life could be delayed, require extensive resources, or improve only one metric at the expense of others.

The situation is mitigated by the fact that many developers and industrial ML engineers are also somewhat researchers. They attend conferences, speak the same language as academics, propose ideas, sometimes participate in creating papers (for example, in coding), or even author them themselves. If a developer is immersed in the academic process, keeps up with what’s happening in the research department — in short, if they demonstrate a reciprocal movement towards scientists, the cycle of transforming scientific ideas into new service capabilities is shortened.

Joining AI research community: overview for industry experts was originally published in Nebius on Medium, where people are continuing the conversation by highlighting and responding to this story.

The beginning of the year is a good time to make plans, right? If you’re an ML engineer, researcher, or technical manager, now is the perfect moment to start thinking about which conferences to attend this year.

Recently, we had an insightful discussion with our machine learning experts at Nebius AI. From their wealth of hands-on experience in developing LLMs, generative networks, and other cutting-edge tech, they’ve curated this list of must-attend conferences. Participating in even a few of these is a surefire way to deepen your knowledge and network with like-minded professionals.

You’ve heard of most of them, and you may have even attended some of them. However, in recent years, there have been more significant players among conference organizers, with OpenAI being the best example. Meanwhile, some other events have been a little overshadowed in their fields. We’ve tried to include both fundamental events aimed more at researchers, as well as very practical, workshop-filled venues in our list.

AAAI Conference (Association for the Advancement of Artificial Intelligence)

Vancouver, Canada, February 20–27, 2024
Attendance: In-person only
aaai.org/aaai-conference

A hub for AI exploration, the AAAI Conference unites experts to discuss AI’s latest trends. The agenda includes technical papers, keynote speeches, and interactive workshops, covering everything from AI’s foundational theories to its practical applications. As an academic-first event, AAAI is ideal for professionals eager to expand their AI expertise.

This year’s AAAI would be particularly noteworthy for its pronounced emphasis on topics like AI in social disciplines, responsible AI and the so-called “Collaborative Bridge,” marking an ongoing collaboration between different communities, both within and beyond the field of AI.

NVIDIA GTC (GPU Technology Conference)

San José, California, March 18–21, 2024
Attendance: In-person and remote
nvidia.com/gtc

Centered on AI, deep learning, and GPU advancements, NVIDIA GTC is the place to be for anyone interested in AI’s technological backbone. It features a mix of experts’ keynotes, technical sessions, and hands-on training. The conference provides a platform to learn about the latest breakthroughs in AI technologies, practical applications, and best practices in leveraging GPUs, like the ones Nebius AI has, for machine learning.

GTC 2024 will feature high-profile speakers from companies like Meta, Amazon, Google DeepMind, OpenAI, LinkedIn, and NVIDIA itself, as well lots of academic institutions.

ICLR (International Conference on Learning Representations)

Vienna, Austria, May 7–11, 2024
Attendance: In-person and remote
iclr.cc

Focusing on data representation within machine learning models, the ICLR is essential for those interested in neural networks and deep learning. The conference is known for its emphasis on deep learning and its applications, making it a key event for ML engineers interested in the latest advancements in neural networks and representation learning. ICLR encourages the submission of papers that propose new methodologies, demonstrate empirical results, or offer theoretical insights in deep learning.

ICML (International Conference on Machine Learning)

Vienna, Austria, July 21–27, 2024
Attendance: In-person and remote
icml.cc

The ICML, a leading global conference focusing exclusively on machine learning, is known for its high-quality, innovative papers. It’s the perfect venue for ML engineers to explore cutting-edge research. The conference includes presentations on a wide range of ML topics, workshops, and tutorials, making it a great event for both academia and industry professionals. Attendees get to exchange ideas, discover new tools and methodologies, and connect with their fellow researchers. Without a doubt, both ICLR and ICML will showcase researchers from all prominent organizations in our field — just like they always do.

TechCrunch Disrupt

San Francisco, California, October 28–20, 2024
Attendance: In-person and remote
techcrunch.com/events/techcrunch-disrupt-2024

Disrupt is one of the most anticipated technology conferences, known for spotlighting startups and new technologies. Just as the organizer, TechCrunch itself, is a coveted platform where startups are discovered by a wider audience, the conference is perfect for those interested in the intersection of ML and entrepreneurship. The event features keynotes, fireside chats, and panels with industry leaders, along with the famous Startup Battlefield competition where new companies pitch to top investors.

OpenAI Dev Day

2024 location, date, and format: TBA
devday.openai.com

Organized by OpenAI, the OpenAI Dev Day is designed for developers and engineers in AI and machine learning. It includes talks, workshops, and hands-on sessions on tools developed by OpenAI’s latest tools, such as GPT-4, DALL-E, and Codex.

The conference aims to provide a deep understanding of these tools and how they can be leveraged in various applications, from natural language processing to generative models. Attendees will gain in-depth knowledge of these tools and their applications, alongside opportunities to network with AI experts.

MLOps World

2024 location, date, and format: TBA
mlopsworld.com

Dedicated to machine learning operations, MLOps World covers model management, ML infrastructure, and ML systems’ continuous integration and delivery. It brings together AI practitioners, and thought team leaders to discuss best practices, share case studies, and explore new tools and technologies in the MLOps space.

MLOps World is the conference to attend if you’re an ML engineer looking to enhance your skills in managing and operationalizing ML models.

NeurIPS (Conference on Neural Information Processing Systems)

2024 location, date, and format: TBA
nips.cc

The NeurIPS is one of the most prestigious events in machine learning and computational neuroscience, featuring peer-reviewed papers, invited talks, workshops, and tutorials on various cutting-edge topics. Renowned for its high academic standards, the NeurIPS is a prime venue for those seeking firsthand insights into advanced research and trends in ML subfields like deep learning and reinforcement learning.

While this year’s program will be announced much later due to the paper review process, NeurIPS will undoubtedly remain one of the most (if not the most) significant and game-changing events in the realm of AI.

Which AI conferences to attend in 2024? was originally published in Nebius on Medium, where people are continuing the conversation by highlighting and responding to this story.

Yandex Station 2 is our newest device that is becoming the centerpiece of the smart home with Alice, our voice assistant. We have completely overhauled the interior and exterior design, added an LED screen on the top panel, and tried to learn from the previous generation’s experience when it came to many details that may not be immediately obvious.

I want to share the inner workings of this process. Below, you will find several stories about different aspects of hardware development: we will discuss the study of the room’s shape with microphones, the scattering of light in transparent materials, generative animations, and the unexpected benefits of FPGAs.

A Story About Sound

Initially, we planned for Station 2 to be smaller than the first Station. However, our acoustic flagship — Station Max — had already been launched by the time we started engineering Station 2. It was almost impossible for a smaller device to compete with the Max when it came to sound. To counter this, we decided to make the sound different — not directed at a person but evenly distributed throughout the room (the so-called “360-degree sound”). In addition, we sought to strike a balance between all frequencies so that users could play voice compositions or podcasts without the annoyance of overwhelming bass.

The 360-degree mode is already used in our smaller speakers. In Station Mini, the surround feeling is achieved by directing the speaker downwards and dispersing the sound to the sides. Yet Station 2 was supposed to be a more powerful device with much better sound quality than the Mini. Early experiments demonstrated that a downward-facing speaker could not provide the quality we were looking for.

In addition, we wanted to preserve the hallmark of the first Station and Max version — stereo sound capabilities.

As a result of many experiments and testing a variety of sound schemes, we came up with the one shown in the above illustration: two full-range speakers are directed forward and backward. On the right and left sides, they are supported by passive radiators responsible for the low frequencies.

The foremost advantage of this arrangement is that you can rotate the device however you like. The speakers can point forward and backward or left and right — the latter positioning slightly enhances the stereo effect. But herein lies the problem: few people put the speaker in the center of the room. In the real world, one of the speakers — or sometimes both — can end up close to a wall, and the sound deteriorates due to constant reflections. Two years ago, we would have given up and opted for a safer sound scheme. But conveniently, amid designing the acoustics for Station 2, we had some encouraging results while playing around with Room Correction. (I will tell you what it is and how it affects the sound in the following story.) After seeing these results, we decided to take our chances.

We had to do some tinkering with the speakers — they needed to be broadband, with the best possible amplitude-frequency response, but also very compact. In addition, we discovered an interesting aspect: with this arrangement, you need to take the position of the device’s center of gravity into account. The higher the center of gravity, the less stable the device. The speakers use specially shaped magnets, additionally weighted to ensure stability. And yes, our speakers are traditionally custom-made — it is impossible to find such speakers on the market.

An unexpected advantage of this symmetrical layout is that high-volume sound does not give rise to excess vibrations throughout the device’s body. They tend to negatively affect everything, including the quality of voice recognition by Alice, because microphone membranes vibrate along with the device itself.

A Story About Room Correction

In their reviews for the first-gen Station and Station Max, users often pointed out that there is too much bass. Low frequencies dominated, even in instrumental tracks and orchestral music. We found that, at least partly, this perception depended on where exactly the device was placed. A small study showed that many people have the Station on a recessed shelf or niche, and some users even put it on the floor in the very corner of the room. Walls and room partitions in close proximity to the device would create acoustic barriers. The resulting changes in frequency ratio often meant precisely what our users wrote about: the overabundance of bass.

This issue can be solved with an equalizer — we launched our own in December last year. But this function is directed at music lovers more so than the broader userbase. We wanted the Station to sound great out-of-the-box: even if put in a closet.

On the one hand, there are solutions on the market that help you adapt the speaker system to a specific room. On the other hand, this primarily applies to multi-speaker setups. The sound quality is optimized for the listener located at a particular spot, like on a sofa in the center of the room.

To automatically adjust settings depending on the environment, the device needs to be aware of its surroundings. Unlike self-driving cars, a speaker does not have a lidar (laser sensors): it cannot determine the shapes of nearby objects. What it does have are microphones.

Using Station 2’s microphones, we recorded the sounds it produced in an acoustic chamber and used them as reference. When the speaker plays the same sounds in the user’s actual environment, it gauges the subtle differences between what it “hears” and the prerecorded reference. This way, the device can get a rough understanding of how the environment distorts the sounds being played. Suppose that some frequencies are louder than the reference by 2–3 decibels: then the Station dials down the volume at these frequencies.

At first, we tried to regulate the sound at all frequencies. We had spent many weeks on tests, but in the end, we realized that the perception of the “highs” depends on not just the speaker’s physical location but also the listener’s. We have not yet learned to detect where a person is in relation to the Station; besides, there can often be several listeners. Nevertheless, we are pleased with how the bass and part of the midrange can adapt to the surrounding room.

A Story About Design

Coming up with a design for a whole new generation of a device is not an easy undertaking. It is essential that users familiar with previous Yandex Stations can see continuity in the new smart speaker and feel that it belongs to a larger family of products. On top of that, we had to develop something entirely new.

As a start, we decided to find out what exterior features people associate with Yandex Stations the most. These features were:

• Simple shapes: simpler than those of, say, the Amazon Echo.
• Lights: LEDs around the volume controls on the first Station, a glowing Alice logo on the Station Mini, etc.
• Body fabric.

All three features had to stay. Station 2 is the second device since the Yandex Module that uses our new design language, which we plan to keep on using in the future. The core property of this language is its two-dimensionality. The main part is responsible for function (in this case, it is covered up with fabric, and the function is sound), and the top part is responsible for emotion. If you look at Yandex Module or the power adapter for the new Station 2, you will notice this two-dimensional principle in action.

In Station 2, we strived for a more elegant look compared to the first generation. To achieve this elegance, we gave the base a sleeker shape. The smoothness creates a subtle illusion: it almost looks like the Station floats above the table. In one of the prototypes, the device’s sides seamlessly transitioned into the base. This prototype could tip over with a faint push, so we decided to slightly complicate the lower part while preserving the rounded corners. To compare our prototypes by unsteadiness, we placed them on an incline and observed the angles at which the speakers would start falling over. Eventually, we reached an improvement of 4–⁠5 degrees. The final design of the base is quite intricate:

Picking the suitable fabric was a challenge of its own. We wanted to take a step forward and make the station a part of the home interior: it had to look good in most environments. After experimenting with different fabrics, we realized that mélange makes for the best solution. You may be familiar with this type of fabric, as it is often seen in clothing and upholstered furniture:

Mélange implies that the thread is woven from fibers of different colors, and the cloth itself is made from this thread. Having added 30% of gray thread, we got a dark Anthracite color with lighter inclusions:

But when picking among earlier Stations, most customers gravitated toward the light gray option. It would not mesh well with the new speaker’s black top panel. We solved this by coming up with a Sand-colored variation. Blue cobalt, in turn, is practically black, but with a gentle twist — it is the kind of look people often go after.

A Story About the Top Panel

The idea of lights, which I listed among the main features shared by all Yandex Stations, is taken to an extreme with Station 2. We chose LED as the most suitable technology for our purpose: to illuminate the top panel from edge to edge. Many difficulties lay ahead, only a handful of which we had anticipated.

The concept of light scattering can occasionally be found in consumer devices, although not very often. Typically, one or more lenses are placed above the light sources. We ran experiments with different types of LEDs, constantly altering their number and mutual arrangement. It became clear that we would need way more than 50 diodes. With so many diodes, it was impractical to use lenses, as that would make our device too expensive. We had to develop a panel that would diffuse the light all by itself — and look gorgeous while doing it.

We tried laying out translucent pieces of plastic on top of the LED board. It was important that one diode does not illuminate the entire panel but also does not look too accentuated, like a bright dot. We wanted something in between so that the light flows smoothly between the diodes. Moreover, the picture had to be clearly visible in the sun. Glossy surfaces glare more, and matte surfaces can affect saturation. Even the matting method plays a role.

An additional complication was equipping the panel with a central unit — an “island” with touch buttons and microphones. We considered the option of lighting the entire panel, including through the printed circuit board of the island. It was supposed to be made either from a transparent film or fiberglass so thin that it is translucent. Alas, not everything is meant to be transparent: during trials, the glowing diodes would highlight the tracks, microcircuits, resistors, and capacitors, making them visible. In the end, we decided to allocate some dedicated space for the island. We carefully chose the central unit’s size to avoid users accidentally hitting the microphones when pressing buttons. By the way, you can switch between tracks by swiping across the three buttons.

The presence of this central unit imposed certain limitations, prompting us to add additional LEDs to the Alice logo, the light from which did actually need to stand out prominently. We went so far with this in one of the prototypes that several distinct dots would shine through the logo. In another mockup, the light from the diodes would trickle into the neighboring microphone openings; those cannot be completely isolated.

In the final version, the island’s components are hidden under the opaque central area of the film that covers the entire top panel. The scattering of light inside the plastic (polycarbonate, to be precise) is achieved by adding a special diffusing agent, the composition of which is almost as secret as the Coca-Cola recipe. The edges of the top panel are visible from the sides: the light reaches them, too. Even if the speaker is above eye level, it will still give you a visual hint at what action is being performed. 84 LEDs provide the backlight — this is the number we eventually settled on.

Brainstorming and implementing the glowing top was one of the lengthiest steps in the development of Station 2. Producing such complex “sandwiches” with an IML film of a sufficiently large area is by no way a trivial task. We found only one company in China that could produce panels of sufficient quality.

A Story About Generative Animations

The screen described above cannot display any specific objects or indicators. It would seem that 84 LEDs are enough, but in practice, this roughly corresponds to a resolution of 7x12 pixels minus the island: less than in early color-screen phones. For comparison, the display of the Nokia 3310 had a resolution of 84x48, tallying 4032 pixels.

But how do we show mesmerizing animations while playing music with only 84 lights? The solution came to light as a result of close cooperation between a developer and a tech-savvy designer. We represent the audio track as a raw data sequence, cut off the outlier bursts, and smooth the data out. At each moment, the renderer “sees” a frame with a set of data distributed by frequencies and their total quantity. You may picture it as such: the tops have one volume, the mids have a different one, and the subs — another volume of their own. Next, we distribute data across the LEDs and determine which group of diodes is responsible for which frequencies. If there are lots of subs, the corresponding diodes flash with higher intensity, and so on.

However, in this mode, the animations tended to look a bit too simplistic. To combat this, we came up with a layer of mathematical modification of the data. From each point, we would build the so-called Manhattan distance. This allows us to light up not just one LED but a kind of circle around it, made up of several diodes. As a final touch, we randomized the borders of this circle a bit. This results in splashes of ambiguous shapes that continuously “flow” into each other — the picture changes as the track plays.

Alright, I told you how we generate shapes on the top panel. But how does the Station choose which colors to use? The answer is that our designer drew a large gradient map with a complex, multi-colored, and non-linear gradient. At each moment in time, a set of colors for LEDs is taken from a certain fragment of this map, then from a neighboring one, and so on. This is how images that pop up during the animation transform not only in shape but also in color. Unique combinations are generated each time, even when you replay the same track.

A Story About Chipageddon and FPGAs

During development of Station Mini, our previous device, we firstly experienced Chipageddon, a hardware shortage caused by the pandemic. That experience prepared us to brace for problems with finding the right components for Station 2.

The Chipageddon impacted the availability of three nodes especially hard: the Type-C controller, STM32 (more on that below), and the amplifier. The latter had to be substituted during development; initially, we designed the device with a different amplifier model in mind. It turned out that class D amplifiers with digital input and built-in signal processing are represented by just a few SKUs in the entire market, and their tuning varies significantly from one manufacturer to another. So, we had about two months to bring the sound level on the new amplifier to the specified requirements.

Our previous speakers used a standard STM32 as the LED controller, and the early versions of Station 2 were no exception. But at the beginning of last year, STMicroelectronics, who used to produce these controllers by the millions, had stopped supplying them in that amount. The sharp decrease in production volumes would soon trigger a chain reaction: many companies began panic-buying whatever was left of STM32s. As a result, the chips not only went up in price but disappeared from the market altogether. At that moment, Chinese analogs with the characteristics we needed (memory size and frequency) were available. But there were few of them; they cost about 2.5 times more than the STM chips we were familiar with and lacked in-depth documentation. We also looked at domestic microcontrollers: they turned out to be even more expensive. The possibility of producing hundreds of thousands of chips in the required (short) time frame was also murky.

Then one of our distributors suggested a more affordable FPGA (Field-Programmable Logic Integrated Circuit) as a replacement for the STM32. Powerful circuits of this type are used for complex calculations and fast processing, such as Fourier transform or video transcoding. Simple FPGAs, in turn, are often used to control simple devices. We were won over when the distributor let us know they could quickly prepare an MVP of such a solution. In the meantime, we had an opportunity to gain expertise in an area that was not entirely familiar to us. The thing is, working with FPGAs is not the same as working with STM32. Yes, you write code in the IDE in both cases, but FPGA programming is based on a completely different logic.

We had figured out controlling LEDs previously in Station Max: there were 400 single-color diodes. In Station 2, there are 84 four-color LEDs (RGBW), which makes for 336 channels. Remembering that the FPGA is all about parallel processing, we decided to take advantage of the opportunities presented by this logic. The STM32 controlled four hundred diodes through shift register chips connected in a sequence into one 400-bit register. In comparison, the FPGA controls seven 48-bit chains in parallel — plus one shorter chain for the logo LEDs. This made it possible to increase the frame rate to several hundred per second, making the animation smoother.

But LEDs are easy to figure out, even when involving a frame buffer and 256 brightness levels per channel. It was a lot trickier to implement touch controls.

While the STM32 has a built-in touch interface controller, the FPGA does not. It might seem that there is a large selection of touch microcircuits to choose from, made by many manufacturers. But the Chipageddon took it all away. The necessary chips were practically absent — the last remaining batch of several hundred thousand that we had managed to order had to be reserved for Station Mini.

Therefore, we began building a touch input controller on an FPGA. It is enough to connect the touch zone with two FPGA legs and one capacitor. But due to the design of Station 2, the tracks between the integrated circuit and the touch controls had to be run all over the board, through connectors, and along a flexible cable. There were a lot of interfering LED control signals around. The first firmware with touch functionality worked fairly well…right until we turned on the animation. That was when false positives started occurring: the system would register a key press even when no one touched the button.

We had to collect raw data from sensors, analyze it and build digital filters while also selecting parameters for reducing noise. Then we found that the average signal level from the touch zone changes slowly over time, so the response threshold must be made adaptive. But that was not the end either. It turned out that the quality of the signal depends on the specific microcircuit. About 17 out of 20 test batch speakers worked reliably. With the remaining three, we observed false positives. We had tweaked the filtering parameters, and three “bad” devices stopped misbehaving. But four of the “good” ones broke.

This effect was caused by the fact that every microcircuit has individual delays in each cell: the signal transmission speed slightly differs. We are talking nanoseconds — but even such minuscule differences matter. Finally, we managed to rewrite the measurement algorithm so that the difference between the microcircuits did not affect anything. But by this time, we had already added a special touch signal quality meter to the FPGA firmware and introduced it into the testing procedure during manufacture. Now we make sure that the touch controls work properly on every Station.

A Story About Zigbee

Previously, Alice would manage smart home devices only through the clouds. The “turn on the light bulb” request would be sent to the Yandex smart home cloud, then to the light bulb manufacturer’s cloud. Finally, the manufacturer’s cloud signaled for the bulb to turn on: that is how it worked for most smart devices. This scheme is unstable and fragile, which often leads to delays in the execution of commands.

The modern Zigbee protocol allows you to send commands locally, directly from the speaker to a light bulb. This is possible when there is a Zigbee chip installed on both devices. Such chips can operate on battery power for several years — dozens of times longer than Wi-Fi modules. While Zigbee’s bandwidth is much more modest, it is plenty for a smart home. In addition, a Zigbee network is a mesh network in the sense that devices with constant power (such as sockets) can act as repeaters and automatically expand the coverage. Newly installed devices choose the closest repeater to connect to, and if it fails, they quickly adapt to the changed topology.

It was crucial to find a place for the Zigbee module somewhere within the speaker’s body. As a result, it shares the board with the top panel’s LEDs located right under the touch controls. It is positioned at the maximum possible distance from all the metal components that could negatively impact the module’s antenna.

The smart speaker still relies on the cloud for voice recognition, but for us, the emergence of Zigbee is a step towards a fully local smart home.

A Story About a Full-Fledged Type-C

What was the difficulty in implementing the Type-C port in first-gen Stations? When users see USB Type-C, they expect that they can power the device with any adapter, such as a phone charger. This is where a task arises: we need to determine whether the connected adapter is suitable. The USB Power Delivery (PD) protocol was developed precisely to tackle this.

Two special chips are needed for the protocol to work correctly: one in the adapter and another in the device. The adapter uses a chip to tell you what voltage and current it can provide. The device responds by communicating its needs for voltage and current, and the connection occurs with the parameters suitable for both parties.

When we were designing Station 2’s power system, we agreed on affordable PD controllers for both the adapter and the device — for the first time ever. The power supply was exclusively designed for this speaker. It outputs the exact parameters needed but is also explicitly adapted for playing music, capable of providing brief on-demand surges of much stronger current. And, of course, you can use it to charge any other devices that support PD.

However, the greatest difficulty was not in developing a suitable adapter but in handling situations where the power supply either does not have a PD chip or has one but cannot provide enough power. We made a volitional decision at this point: we let the Station boot up from unsupported power supplies and quietly alert the user that a more suitable adapter should be used. If the power supply supports the [15V 3A] PD profile, the speaker is fully functional.

Station 2 is one of the most technologically sophisticated Yandex devices to date — we had faced many challenges during its development and hope that we were able to make a smart speaker that users will love.

Station 2. Stories Behind Developing One of the Most Sophisticated Yandex Devices was originally published in Yandex on Medium, where people are continuing the conversation by highlighting and responding to this story.