Understanding AI as a “Concept Information Processor”
The reason why conversational AI can achieve natural conversation similar to that of humans cannot be explained merely by its acquisition of natural language or its learning of vast knowledge.
For natural conversation to be successful, the AI must internally retain information about various concepts embedded in the conversation and accurately adjust that information according to the flow of the dialogue. It must then generate responses based on that adjusted information.
For example, when you ask conversational AI, “I want to go from Tokyo Station to Shibuya Station, but without taking the train,” it replies, “There is a way to get from Tokyo Station to Shibuya Station without taking the train by using the bus.”
If this query were entered into a search engine, it would return pages showing how to access Shibuya Station by train or sites offering train route guides. This is because search engines do not comprehend the concept of “without taking the train,” and instead return results matching the keywords.
Conversational AI, by conceptualizing the part that says “without taking the train,” avoids suggesting train options and provides alternative methods.
In this article, I will focus on the concept information processing capabilities of conversational AI and explore the essence of its mechanism. Therefore, I will analyze conversational AI as a general concept information processor, comparing it to traditional general-purpose computers.
From this perspective, it becomes clear that the essence of generic concept processing lies in contextualized, patterned processing.
Context-dependence means not only reflecting the latest state of the concept but also continuously incorporating intermediate states of concept information generated internally while processing input or generating output sentences.
Patterned processing refers to the idea that the process itself does not change during processing. The same rules and flow are applied throughout the process.
Considering the flexible concept processing of conversational AI, this may seem somewhat surprising. However, flexible concept information processing can be sufficiently realized through patterned processing, and this method is actually more suited to general-purpose concept information processing.
Transformer Model
Current conversational AIs are implemented based on a technology called the Transformer model.
The Transformer model has three major characteristics. The first is its chained processing structure.
The Transformer model processes the words included in the input sentence and the words in the output sentence being generated by conversational AI, one by one. While generating the output sentence, it performs chained processing by outputting one word, using that word for the next process, and then outputting the next word.
The second characteristic of the Transformer model is that it generates large-sized data, called embeddings and hidden states, for each word being processed.
When viewing generative AI as a general-purpose concept information processor, this data corresponds to concept information. This data is inaccessible from outside the AI, making it impossible for others to understand what it means.
The third major characteristic is that when processing a word, it generates the corresponding concept information using all the previously processed concept information.
This mechanism is known as the attention mechanism. The paper introducing the Transformer model was titled “Attention Is All You Need,” and this attention mechanism is considered the core of the Transformer model.
Common Rules for Unchanging Concept Processing
We will not delve into the technical details of the attention mechanism or the Transformer model here. Although these details are important for implementing conversational AI, they do not represent the essence of the general-purpose concept information processing mechanism that this article seeks to explore.
The crucial point here is how the attention mechanism utilizes the concept information from previously processed words when generating new concept information.
The concept information of the processed words is used to modify the concept information of the word currently being processed. This modification is applied to all previously processed words in the same way.
During this process, the modification rules remain the same for all words. Of course, even though the rules are the same, the content of the modification depends on the concept information of both the processed words and the word being processed.
In traditional computer terms, the concept information of processed words acts as the program, and the concept information of the word being processed acts as the data to be modified. The common rules of concept information processing correspond to the processor’s specifications.
By making each piece of concept information function as a program to modify another piece of concept information, the common rules of concept information processing can be fixed, much like the processor in a traditional computer.
This explains why conversational AI can flexibly converse with fixed parameters after pre-training and adjusting the parameters of its internal neural network.
Comparison with Traditional Computers
In a traditional computer, the processor’s specifications are designed by humans. In contrast, the common rules for concept processing in conversational AI as a general-purpose concept information processor are determined through training in machine learning.
During the training process, the conversational AI processes vast amounts of text repeatedly, adjusting the parameters of its internal neural network. From the perspective of a general-purpose concept information processor, this can be seen as learning the common rules of concept processing.
Despite this difference, the fact that the processor’s specifications do not change during operation is the same as in traditional computers. This demonstrates that even if the processor’s specifications remain unchanged, highly complex and useful processing can be achieved if the program becomes sophisticated enough.
Additionally, in traditional computers, individual instructions in the program specify which data in memory to modify. Conversely, in conversational AI, the concept information of one word modifies the concept information of all subsequent words. However, the attention mechanism of the Transformer model changes the degree of modification.
Get katoshi’s stories in your inbox
Join Medium for free to get updates from this writer.
Despite these differences, the basic idea that programs modify data remains common. This highlights that, fundamentally, the processor’s role is to process information.
Significant Differences from Traditional Computers
A key difference between traditional computers and conversational AI as a general-purpose concept information processor is the pre-determination of programs.
In traditional computers, programs are basically determined at the start of processing. While it is possible to modify and change the program during processing (since the program is also information), it is not common.
In contrast, the Transformer model of conversational AI, by design, generates and modifies concept information corresponding to the program during processing.
First, as mentioned in the first characteristic of the Transformer model, it processes output words in a chained manner, generating concept information as it goes. This means it is repeatedly outputting programs and executing them during processing.
Additionally, as mentioned in the third characteristic, the attention mechanism generates concept information for the word being processed using all previously processed concept information. This means programs generate other programs.
The dynamic generation of programs during processing and the influence of other programs on the generated programs are major distinctions between conversational AI as a general-purpose concept information processor and traditional computer processing.
Patterned Processing
By generating concept information corresponding to a program while the process progresses, even with fixed rules and patterned processing, highly complex tasks can be accomplished.
This becomes clearer when considering a similar model in its minimal configuration.
In general, AI, including conversational AI, is characterized by analog information processing, machine learning, and the complex structure of neural networks. These features do indeed differ from conventional computer information processing, but they are not the essential points of concept information processing.
Even if the information being handled is limited to just one bit of digital information, composed of zeros and ones, and even if the processing follows a simple set of pre-determined common rules, a processing structure that treats processed information as instructions for subsequent operations should still be able to generate extremely complex output patterns.
Of course, simple common rules cannot precisely handle real-world concept information. However, it is clear that the structure allowing complex output to be generated from tiny rules is key to conversational AI’s ability to achieve the complexity necessary for natural conversation with humans, even while adhering to fixed rules.
Additionally, extracting general-purpose concept processing as compact common rules in comparison to the complexity of processing demonstrates that such a system can possess significant adaptability and applicability.
Context-Dependent Processing
Processing based on fixed rules also applies to phenomena that can be represented by equations, such as in physics.
In the case of motion equations for one or two objects, providing the initial state and elapsed time allows one to analytically calculate the state after a certain period of time.
However, as known in the three-body problem, when the number of objects increases to three or more, the same equation applies, but the solution cannot be obtained analytically. Instead, only by dividing time into increments and simulating each step sequentially can the progression of time be understood.
In such simulation-based processing, when the latest state and rules are provided, the next moment’s state can be calculated. This can be referred to as state-dependent processing.
Similarly, the characteristic of conversational AI as a concept information processor is that it not only relies on the latest concept information but also on previously processed concept information to determine the next step. The collection of processed concept information can be called the context. Thus, conversational AI as a concept information processor can be described as engaging in context-dependent processing rather than state-dependent processing.
The interactions between a few objects, which can be solved analytically, never surpass the complexity suggested by the equations.
On the other hand, the motion of many objects, solvable only through simulation, is a form of state-dependent processing. This exhibits complexity far beyond what the equations might suggest. Sometimes, phenomena like self-organization or emergence occur, where order arises within the complexity, seemingly following entirely new rules.
Context-dependent processing amplifies the complexity and order-forming traits found in state-dependent processing. The context complicates processing while also contributing to the formation of order.
Generative AI, as a general-purpose concept information processor, utilizes this characteristic of context-dependent processing to realize complex, highly ordered concept processing capable of conversing with humans.
Conclusion
From these considerations, I believe, as mentioned at the start of this article, that context-dependent patterned processing is the essential mechanism of conversational AI as a general-purpose concept information processor.
The Transformer model embodies this essential structure. On top of that, with clever, machine-learning-enabled detailed structures like the attention mechanism, and the inclusion of vast parameter sizes, conversational AI capable of natural dialogue was realized.
Today’s conversational AI not only processes concept information in a general manner but also encapsulates knowledge about various concepts. This means that conversational AI learns and includes information beyond the common rules necessary for concept processing within its parameters.
Conversational AI also has the ability to use knowledge provided through input, even if it has not learned that specific knowledge. In other words, if there is a mechanism to provide some concept knowledge externally, there is no issue if conversational AI forgets that knowledge internally.
The minimum required information for a general-purpose concept information processor may be very small compared to the vast knowledge stored by generative AI. Therefore, by adjusting the parameters of generative AI to forget unnecessary knowledge while retaining the common rules needed for concept processing, it may be possible to consolidate these into a small set of parameters.
Since the common rules do not need to change, once they are consolidated into such a small parameter set, it could be possible to develop hardware chips that improve processing efficiency and energy efficiency. In that case, another mechanism to store knowledge about many concepts will be necessary.
Thus, I believe that the future direction of conversational AI research will shift toward identifying parameters sufficient for general-purpose concept processing and developing concept information memory technology capable of storing and utilizing the knowledge that was previously held within the Transformer model’s internal parameters.
