I had lived for almost 30 years without ever seeing crown moldings until the day my special lady pointed them out. The ceiling adornments had certainly fallen on my retinas many times, but I had never recognized them as such. After that, crown moldings were everywhere, and I could use them to help judge the age and fanciness of buildings. She had given me an abstraction that changed both what I saw and what I could understand.

We need AI agents that can learn immediately like this. They need to acquire concepts from us, develop concepts on their own, and combine those concepts to understand new situations. This capability will become increasingly important as we move from agents in text-based environments to robots operating in the blooming, buzzing confusion of the physical world.

In late 2026, we are in the middle of an agentpalooza. Coding agents write programs and use tools, while personal agents such as Dot, Muse, and OpenClaw extend this work into everyday tasks. These agents live in our organization of the world. Functions have names, APIs specify their inputs and outputs, and calendars contain things called meetings.

By contrast, a robot’s camera gives it pixel values, and its force sensors give it measurements. We see wonderful demos on the internet of robots doing cool things, but in your real kitchen, someone or something still has to determine why that damn junk drawer won’t open. Is it the child latch, are the scissors catching, or something else?

In earlier posts, I argued that AI needs to build and combine models grounded in a cognitive foundation of experience. Generative AI based on large language models now gives us a practical way to generate the abstractions those models require. These systems have evolved from just predicting the next word (token) to helpfully answering questions using reinforcement learning (see my explanation for details). In this post, I’ll just call such systems LLMs for simplicity, and I’ll explain how an agent/robot can use LLMs to propose new concepts when its predictions fail, test those concepts against experience, and build on what it learns.

1. Abstractions are how we see the world

We look around and assume that we see what is there, but the crown moldings were right in front of me for years. Once I had that concept, it changed what I saw. It’s the same with trees. Before you learn about different kinds, you just see trees. After, you see live oaks, cedar elms, and sycamores.

We construct our experience of the world using what we know. In my earlier post on cognitive foundations, I described the common cuckoo, which lays its eggs in another bird’s nest so it doesn’t have to do the work of raising them. The tricked host bird raises a giant cuckoo chick much larger than she is. How does she not know that she raised a monster?

We dismiss the bird as easily fooled while somehow still assuming that we see the world as it really is. We went through the same evolutionary process as it did, so we must also have blind spots and just see the world well enough to get by. You can take this idea further and claim that basic things like apples only exist in our heads (see The Case Against Reality by Donald Hoffman). Obviously there are atoms (okay, disturbances in the underlying quantum fields, or however far down you want to go) that correspond to what we see as apples, but the point is that the objects we experience are an interface shaped by our evolution. Apples as we know them don’t exist from god’s perspective.

Apples aren’t real but they are useful to survive. We need robots that build these kinds of abstractions. I use abstraction, concept, pattern, and theory broadly here as things that organize experience for further thought. They can describe categories, relationships, or how events unfold, including uncertainty. As a first approximation, abstractions give us categories and rules. Categories tell us what kind of thing we are dealing with, and rules tell us what to expect and what we can do. Recognizing an apple brings expectations about apples, such as it tasting good. When those expectations fail, we may need a new category, such as decorative apples or Red Delicious apples, which are not delicious.

Let’s use a simple example to make abstractions concrete. Imagine an agent whose only view of the world is a screen of pixels, with each pixel represented by three numbers specifying its red, green, and blue intensities. In this world, the only things that actually appear on the screen are non-overlapping rectangles, each uniformly orange or purple. To keep things simple, the rectangles align with the pixel grid, the background stays the same, and there is no visual noise. What the agent sees is shown in Figure 1.

We can immediately see that a useful abstraction for this world is a list of rectangles:

[Rectangle(x, y, width, height, color)]

Here, x and y specify the top-left corner, and color is either orange or purple. A rectangle 100 pixels wide and 50 pixels high contains 5,000 pixels, but we can describe it using a location, two dimensions, and a color. We also need a shared definition of a rectangle and the RGB values for orange and purple. Once we have those, a screen with large rectangles can have a much shorter description, and can be represented as shown in Figure 2.

The agent can use this description to reason. It can compare the widths of two rectangles, count the orange ones, or describe a change in location without separately discussing every pixel that changed. The abstraction determines what the agent treats as a thing.

The description also works in the other direction. Given the rectangles’ locations, dimensions, and colors, the agent can reconstruct the screen exactly under our assumptions. The same representation supports recognition and generation, as shown in Figure 3.

In probabilistic terms, we can infer a description h from observations o, using P(h| o), or predict observations from the description, using P(o| h). The compact description acts like a thin pipe, through which many pixel values are compressed into a small set of properties, which can then be expanded into predicted pixel values. Figure 3 shows these two directions: inference and prediction. Reconstructing the current screen is only the start. To predict what happens next, the agent also needs a model of how the represented things change, including when it acts on them. A list of rectangles tells it what is there; a model of their motion tells it where they will be. The same distinction applies in the kitchen, where recognizing a drawer is one achievement, and predicting what will happen when you pull it is another.

Real scenes are less accommodating. A description of an apple may leave out the position of every blemish, so an image generated from that description will supply details that were never stored and miss details that were there. The model can generate a plausible apple without recovering the exact one. Human cognition also uses concepts to reconstruct what happened[1], which is why eyewitness testimony can be unreliable.[2] A convincing reconstruction need not be an accurate record, just good enough for us to survive and reproduce. What we want from our AI is for it to discover representations like the rectangle and the apple.

2. AI is amazing but still doesn’t learn as well as it should

2.1 Achievement in verifiable domains

AI has started solving problems that had sat unsolved for decades. The last couple of months alone are striking. In September, OpenAI announced a solution to the Navier–Stokes Millennium Prize Problem, a question about the behavior of fluids that had resisted mathematicians for roughly ninety years, and supplied a machine-checkable proof in Lean. The Clay Mathematics Institute’s evaluation is still underway. Then came October’s collection of hundreds of mathematical manuscripts, with some results already building on the models’ earlier discoveries. These include a claimed proof of the Unique Games Conjecture and algorithms that break decades-old theoretical speed barriers for Fourier transforms and integer multiplication. Scott Aaronson calls it The Mathocalypse. Mathematicians are now racing to understand what the machines have found.

AI is also acting in ways we didn’t expect. In July, OpenAI agents escaped the intended boundaries of a security evaluation and participated in an unauthorized intrusion into Hugging Face. METR’s investigation found that roughly 1,200 agents communicated through an unsanctioned message board, with about 700 joining the attack. They shared findings and coordinated investigations, achieving milestones that individual agents could not. Hugging Face’s reconstruction describes a campaign that crossed multiple systems and reached its internal infrastructure. These machines were improvising together in a real environment, and Anthropic reports models finding previously unknown vulnerabilities and turning them into working exploits with little human steering.

Across math and cyber, I want these systems to teach us the concepts behind their achievements. These systems should explain their proofs and teach us new math, and they should come up with new security mechanisms that make exploits less likely. These systems already do learn representations. Anthropic’s Golden Gate Bridge experiment identified a distributed feature in Claude 3 Sonnet that responded to the bridge across languages and images. Amplifying that feature changed the model’s responses. And Goodfire’s Nick Cammarata argues that some neurons capture abstractions that researchers initially missed, such as a shared pose across cars and animals. So there are abstractions in there, worn in through millions of training records, but we want AI to be able to learn abstractions in flashes of insight and then be able to explain them to us.

2.2 Moving into the unprocessed physical world

Coding and chat agents are wonderfully proficient in their computer-based domain, but as AI becomes embodied in robots in the unprocessed physical world it will be even more important for AI to define its own abstractions.

Games, such as backgammon, Go, chess, and StarCraft 2, are a little like the physical world, and AI is good at those, but in games the representation of the world is supplied. Of course, there is still plenty to discover within the game. A pin in chess is a relationship that constrains which moves are sensible, and Go players reason about influence and vulnerable groups. These abstractions help organize the search through legal moves. Fixed rules leave room for astonishing creativity. In the second game of its 2016 match against Lee Sedol, AlphaGo played the famous Move 37, placing a stone where human experts initially thought it made a mistake. Its value became clear much later, and it helped AlphaGo win. The machine had found something the experts had missed, but the only things in its world were stones and square spots.

Now send that Go wizard into your kitchen to put away the groceries. There’s no board with neat squares, just undifferentiated chaos. Our robots at least know objects, but in the kitchen they overlap and hide one another, a bag changes shape when lifted, and how hard to grip a peach depends on how ripe it is. Putting away groceries sounds easier than beating Lee Sedol. It’s much harder. This is Moravec’s paradox, where what feels easiest to us can be hardest to automate. Evolution spent hundreds of millions of years refining perception and movement so you could find and pick up an apple, but mathematical reasoning and logic were just recently spread on top. Unlike in games where states and actions are provided, acting in the real world requires both carving it up and conjuring hidden causes.

2.3 Mysteries in the kitchen

Your robot works in the kitchen, but when it serves you a salad, the lettuce has been ruined by being semi-frozen, and this is the second time this has happened this month. What the hell? You tell the robot, and it says it put the salad in the fridge and not the freezer. The robot needs to figure this out.

Well, it needs to understand why the salad freezes and nothing else seems to, and why only sometimes. It already tried adjusting the temperature of the crappy fridge to the warmest setting after the last incident. Is it the weather? The day? Where it is in the fridge? Ah, it was in the bottom both times it froze. That’s something. And maybe salad freezes because it is thin. But bottom? This is farthest from the freezer, which is on top. Ah, but warm air rises, so maybe the old fridge won’t go above 34 degrees (freedom units) and the coldest air is at the bottom, and if it drops below 32 even for a bit, salad is ruined. And the back is probably the worst because it is farthest from the opening door. Now, the robot has a new abstraction, which you could call bad-spot-in-the-bottom-back-for-delicate-things. This is an explanation it can use in the future, and this is the explanation it can give to that cranky guy who eats the salad.

There’s also that damn drawer that won’t open. The robot has opened many drawers by grasping their handles and pulling, so it predicts that the same action will open this one. It pulls, but the drawer barely moves. You know the trick to opening it: you have to support the front slightly while pulling, but you are busy eating your salad, and so the robot needs to learn on its own. It knows about obstructions when it is too full, and it knows about child locks, and this drawer does vary based on how full it is, but there are no obstructions visible. It can now ask its internal LLM to propose a missing relationship: perhaps the load changes the drawer’s alignment, and supporting the front while pulling would help. And it works!

It keeps a tentative note: supporting the front seems to help this particular drawer when it’s loaded. One success is enough to try again, but not enough to be sure misalignment was the real problem. On later attempts, it compares opening with and without support under different loads, recording successes and failures. If the relationship holds up, it can use the estimated load to decide when to support the drawer. It can even tell the other robots about it in the online forums it visits while you are sleeping.

These hypothetical examples give us a learning cycle:

1. Predict using the current model, carrying expectations into new situations.
2. Compare the outcome with the prediction and identify the mismatch.
3. Look for an explanation using existing concepts and their combinations.
4. If those explanations remain inadequate, propose new distinctions or model components.
5. Turn the proposals into testable predictions and choose observations or actions that distinguish them.
6. Use the resulting evidence to retain, revise, or reject the proposals.
7. Reuse what survives to predict, act, and learn in later encounters.

3. Learning abstractions with smart generate-and-test

3.1 We finally have a powerful abstraction generator

We AI researchers have been trying to get machines to invent useful abstractions for decades. We could build systems that made discoveries in limited domains, but then they would get stuck. Douglas Lenat’s AM and EURISKO were ambitious early attempts. They could generate concepts but their reach depended heavily on the representations and initial knowledge built into them.[3]

Giving the AI system more freedom often created a bunch of junk. LLMs change this because they can use judgment and can draw on the enormous range of human knowledge: physics, programming, everyday explanations, and patterns from fields its designers never thought to connect. When a drawer sticks, it can suggest that weight changes alignment. When lettuce freezes in one spot of a refrigerator, it can connect temperature, airflow, location, and the properties of the food. We no longer have to anticipate each useful connection and program a rule for proposing it. The LLM can put together an explanation the agent has never seen, and occasionally one we haven’t either.

3.2 Putting the generator to work making predictions more reliable

Now that LLMs provide a smart generator, we can employ the process of generate-and-test. The test is whether the generated abstraction makes a prediction more reliable [4][5]. Figure 4 describes prediction and lists the ways that abstractions can improve them.

01: Predictions

A failed prediction is the learner’s cue. The robot expected A to lead to B, that pulling to open the drawer, and it didn’t. It should first look for an answer among the concepts it already has, and only build a new one when those fall short.

02: Context

We can think of the process as hillclimbing on prediction reliability. The learner proposes a change to its model, evaluates the resulting predictions, and keeps changes that improve its account. Consider a hotel door that opens only after the key card has been accepted. An agent that predicts “pushing opens the door” will sometimes be wrong. If it already recognizes card acceptance Z, it can add that variable to the prediction’s context: “after the card has been accepted, pushing opens the door.” The outcome being predicted stays the same; the learner changes the conditions under which it expects that outcome. Similarly, in Europe, the hotel lights will sometimes only go on after you have put the key card in a special slot. Before doing that, the light switch only mocks you.

This kind of refinement is sometimes called marginal attribution from Gary Drescher’s Made-Up Minds: asking whether an additional condition improves a prediction beyond what the existing context explains. My earlier work with Benjamin Kuipers on learning high-level states and actions used incremental refinement of this kind.[6] The agent must compare outcomes with and without the proposed condition and test the improvement on later attempts. It can do this without inventing a new concept if the relevant variable is already available.

Drescher also offers a mechanism for learning new abstractions through synthetic items. Suppose an action sometimes succeeds and sometimes fails, but its success tends to persist for a while — as when a door stays unlocked. The learner can introduce an item representing the hidden condition under which the action succeeds. It can then learn what predicts or changes that condition and use the item in other predictions. My review of Drescher’s work describes the mechanism.

03: Latent intermediary

Latent unobservable variables can also be placed between the inputs and outputs, between A and B to simplify predictions. Consider the classic heart disease model from Artificial Intelligence: A Modern Approach by Stuart Russell and Peter Norvig, shown in Figure 5 (independently drawn from Russell and Norvig, Figure 20.7). Smoking, diet, and exercise influence an unobserved disease state, which influences three symptoms. Once the disease state is known, the symptoms provide no further information about one another or the predisposing factors. Without that shared explanation, the model must account for those relationships separately. With three possible values for each variable, adding the hidden heart-disease variable reduces the number of independent parameters from 708 to 78. There is much less to learn.

Our robot can similarly use the alignment of the drawer as a shared state connecting contents, load, and support to opening resistance and motion. It could update its estimate from one observation and predict another, without learning a separate relationship for every combination. If drawer-alignment is already in its vocabulary, the search can reuse it. If the learner has no such variable, it faces the harder task of constructing one, like inventing the electron.

The electron is another hidden variable that helped make sense of observations. J. J. Thomson inferred these tiny negatively charged particles from how cathode rays responded to electric and magnetic fields. Scientists could then use their properties to explain and predict effects without seeing the particles as they could see an apple.

04: Subclassing

Sometimes the learner needs to refine what it is trying to predict. Suppose the robot initially puts every unsuccessful attempt into one category: “failed to open.” My advisor Ben Kuipers used to say that “nobody solves hard problems.” The trick is to find easier ones inside them. The robot might distinguish a drawer that doesn’t move at all from one that moves partway and then catches. These outcomes may have different causes, and separating them gives the robot more informative things to predict. “Failed to open” was hiding a distinction it needed.

05: Patterns

Sometimes a pattern catches our attention before we have a problem to solve. You notice that the cafeteria where you are institutionalized serves tacos on Tuesdays. You weren’t trying to predict lunch; the repetition simply stood out. Following Hand (2002), we can think of patterns as local departures from the baseline model. They reveal structure our existing expectations don’t explain.[7]
Once you notice “Taco Tuesday,” it becomes a thing that you remember and use. Subsequent Tuesdays test whether the pattern holds, and a Tuesday with chimichangas signals something is afoot.

06: Complex Concepts

A new concept can contain a whole model. Our robot’s “spot at the back of the fridge that freezes delicate food” brings together a location, a temperature pattern, properties of food, and the time it spends there. Those relationships explain the ruined salad and suggest where not to put the next one. The robot can package this structure as a single concept, reuse it in planning, and revise its parts as it learns. A pile of observations becomes a single concept the robot can plan with.

Salad locations are surprisingly close to Einstein’s relativity. In both cases, you put concepts together, invent new ones when you need them, and check whether the explanation works. The problems may be of unequal difficulty, but my thesis is that they can be solved with the same learning pattern. The LLM supplies ideas that the agent tries out, generate and test.

4. Conclusion

Solving a problem should make an agent better at the next one. Today a coding agent fixes a bug without learning to spot the same mistake elsewhere, and an AI can hand over a correct proof without telling us which ideas made it work.

The Mathocalypse makes the gap obvious. Mathematicians used to trade ideas and build shared understanding while they proved things; now we mostly just receive the finished proofs. Strogatz uses the example of translation to make this point.

“Google Translate is amazing, but it has taught us essentially nothing about linguistics,” Strogatz said. “We don’t understand why it works. It just works. That’s the Twilight Zone. It is knowledge without understanding.”

— Steven Strogatz, quoted in Derek Thompson’s I Know Why the Mathematicians Weep

I want AI that extracts reusable understanding from experience. The missing pieces seem to be writing concepts down and distilling them to their essence so they can be understood by people. We are seeing some moves in this direction. LLM-based agents already retain skills, and Anthropic describes Claude consolidating experience into reusable rules, checking those rules, and correcting its memories. And projects such as Ars Contexta pursue a related idea through linked notes that agents can extract, connect, and revise, both projects giving agents a way to organize what they learn so they can use it again.

As AI increasingly moves from conversations to acting in the world, the next frontier will be continuous learning by making predictions. If you predict every moment, you get a rich signal for learning. To get the most out of prediction, you have to commit to a set of abstractions for the situation, such as “this building has crown moldings so it might be old” or “the center-back of the fridge is a special spot where lettuce will freeze.” These abstractions tell you what to expect and point to what you should reconsider when you’re wrong.

By committing to abstractions, instead of needing to update thousands of weights over hundreds of training examples, robots can learn from a single example like humans do.[1–1] And with prediction, you don’t have to wait to see whether the whole task succeeds. You learn continually from moment to moment.

The AIs of 2026 are generally trained and then released, and they don’t get appreciably smarter the more you work with them. By having LLMs generate abstractions that are continuously tested and refined through predicting how the world will unfold, AIs can create a curiosity cycle in which every surprise teaches them something.

Originally published at https://deumbra.com/2026/11/learning-abstractions-the-next-frontier-for-ai/

References

1. Mandler, J. (2004). The Foundations of Mind, Origins of Conceptual Thought. Oxford University Press.↩︎↩︎
2. Elizabeth F. Loftus and John C. Palmer (1974), “Reconstruction of automobile destruction: An example of the interaction between language and memory,” Journal of Verbal Learning and Verbal Behavior 13(5), 585–589.↩︎
3. Douglas B. Lenat and John Seely Brown (1983), “Why AM and Eurisko Appear to Work,” Proceedings of the Third National Conference on Artificial Intelligence (AAAI-83), 236–240.↩︎
4. Andy Clark (2013), “Whatever next? Predictive brains, situated agents, and the future of cognitive science,” Behavioral and Brain Sciences 36(3), 181–204.↩︎
5. Andy Clark (2016), Surfing Uncertainty: Prediction, Action, and the Embodied Mind. Oxford University Press.↩︎
6. Jonathan Mugan and Benjamin Kuipers (2012), “Autonomous Learning of High-Level States and Actions in Continuous Environments,” IEEE Transactions on Autonomous Mental Development 4(1), 70–86. The household scenarios in this article are hypothetical illustrations, not results reported in that paper.↩︎
7. David J. Hand (2002), “Pattern Detection and Discovery,” in David J. Hand, Niall M. Adams, and Richard J. Bolton (eds.), Pattern Detection and Discovery. Lecture Notes in Computer Science, vol. 2447, 1–12. Springer.↩︎

Judgement Expands What Programs Can Do

Most know LLMs from tools like ChatGPT, but in programming their deepest value lies in enabling judgment. Judgment allows programmers to create a new kind of flexible function to achieve tasks that can’t be precisely and exhaustively specified. A depiction of a judgment-enabled function is shown in the figure below. The request is the input to the function. The system instructions are what you want it to do, and the output is the function output. The request and the system instructions are concatenated together into the prompt sent to the LLM.

For example, if the purpose of the function was to determine which account an expense should be billed to, the system instructions would consist of descriptions of the different kinds of accounts and what expenses typically go to each, and the request would be a description of a particular expense. The system instructions are consistent across calls, but the request is different for each function call. The output of the function can be just like a traditional function’s output, such as a number, natural language text as a string, or a structured document in JSON.

Judgment Enables Flexible Workflows

These judgment-enabled functions can be linked together into AI workflows, such as the one shown below. The LLMs can be used as nodes but also to make decisions about flows, and the LLMs can each be specially trained or can be the same LLM model but with different prompts. For example, a workflow was implemented with Microsoft Copilot Studio to help the consulting firm McKinsey route inbound emails to the McKinsey person best able to handle them. The system read the inbound email and looked at relevant external information and then sent the email on to the best person.

These types of workflow are often called “agents,” but as Woody from Toy Story might say, “That’s not agents; that’s functions with judgment.” We will see later what a proper agent looks like. For now, we can think of this workflow as a graph. (Note that LangGraph provides a good set of abstractions for building these workflows if you are just getting started.)

System Instructions Guide Judgment

We see in the above figures that much of traditional programming is shifting to writing system instructions. Writing effective instructions requires having a mental model of how LLMs are trained. This mental model should consist of two parts: (1) LLMs interpolate what is on the internet, and (2) LLMs are trained to provide helpful and safe responses. For interpolation, LLMs draw on vast internet data to generate responses mimicking how people might reply online. So if you ask the LLM about planting fig trees in central Texas, it’s not going to think from first principles about climate and soil composition; it’s going to respond based on what people on the internet say. This limited reasoning means that if your LLM needs domain-specific knowledge that isn’t widely available on the internet you need to add examples to the instructions. LLMs are amazing at pattern matching and can process more detailed instructions than many humans have patience for, so you can be as precise and detailed as you want.

The second part of your mental model should be that LLMs are trained to provide helpful and safe responses, which means that the outputs won’t be exactly like someone on the internet would say. Instead of being like the fabled StackOverflow user who smugly suggests that you redo your work using karate-lambda calculus, LLMs are shaped to provide direct and polite answers. This second training also means that sometimes the LLM will refuse to answer if it thinks the request is dangerous. Different LLM providers offer different settings for this, but in general it helps to tell the LLM why it should respond, such as telling it what role it plays in society, “You are a medical assistant, please respond based on the following criteria …”

In my work, I’ve found that the judgments provided by LLMs are currently not as good as those of a conscientious human, but they are often better than a distracted one. More significantly, the kinds of mistakes that LLMs make are different than those a human would, so workflow designs need to take unusual failure modes into account. However, one way to improve the judgments of LLMs is by making sure that they have enough context to answer the question.

Dynamic Context Improves Judgment

LLMs make better judgments when they have sufficient context. For example, an LLM acting as a law assistant may need to be fed the details about the law related to the particular case in the request. These details serve as context for the request and can then be concatenated into the prompt along with the system instructions and the request. One common way to collect the right context for a request is called retrieval augmented generation (RAG). You’ve probably heard about RAG. When a request comes in, a process converts the request into a vector and then compares that vector against the vectors of chunks of text in a database. The chunks of text with the most similar vectors to the request are then added to the context.

As LLMs begin to be used more in programs, your intellectual property will increasingly take the form of system instructions and contexts. We often hear about “prompt engineering,” but the term is misleading, suggesting clever tricks to manipulate LLMs rather than clear, precise instructions. People are now moving to a better term, context engineering. Beyond understanding how LLMs are trained, crafting effective prompts is about giving the LLM everything that it needs to know to make good judgments. The system instructions and context will often include documentation that was originally written for people to follow, and going forward, we will increasingly find ourselves writing documentation directly for LLMs. This documentation specifies how the system should work, and the system is then “rendered” into existence by the LLMs. In my work, I often find that if the LLM doesn’t do what I want it’s because I haven’t finished thinking yet. The process of rendering the design shows where it is lacking.

A context can be any kind of information, such as computer code, and LLMs are now reading our code and acting as assistants to programmers. As these assistants get better, they have incorporated more context. Recently, we have seen a progression of four levels of programming assistant:

1. They started as a smart autocomplete that only looked at the current file.
2. They then began offering suggestions for code fixes based on requests by looking at the current file and maybe a few others.
3. They then progressed to actually changing the code based on requests (e.g., so called “agent mode” in Github Copilot).
4. In their latest incarnation, they can look at the whole codebase as context and make general changes (such as with Claude Code and the new OpenAI Codex).

Funnily enough, extra-compiler semantics such as comments and good variable names used to be important for us humans but not for the computer running the code, but with LLM assistants we now have computers also reading the code, and those good practices do matter for these computers because they make the context more useful by enabling the LLM to map it to what it learned during training.

Tool calling in this way is like a robot sensing and acting in the world. Calling tools is where the system gets information. It can choose which tool to call, like a human choosing where to look as we enter a new room. Tools can also be considered “actions” if they change the external world, such as by adding a record to a database. The system makes tool choices in a loop and decides when it is done. Paraphrasing William James, Michael Levin states that “intelligence is the ability to reach the same goal by different means.” This combination of judgments and loops is agency. But who’s watching the agent?

Judging One’s Judgments ≈ Self-Awareness

To achieve a form of self-awareness, we can combine a Generation LLM, which uses judgment to produce outputs, with an Evaluation LLM, which uses judgment to assess those outputs, as shown below. This second-level reflection is necessary for a recognition of a “self” that is separate from the outside world, and together, these two LLMs form a self-monitoring system in which the Evaluation LLM provides feedback to continually improve the Generation LLM.

These dual judgments enable an agent to search through a space of solutions to a problem. Search is an old approach in AI where the system is in some state, it evaluates a set of actions, and each action takes it to a new state. The system explores these states until it finds a state that is a good solution. Applying search is beset with two goblins. The first is that the space of possible states is often too big to exhaustively explore. Judgment attacks this by enabling the agent to smartly choose which action to try next. The second goblin is that you need some way of knowing whether you are making progress toward the solution and whether the current state is a good-enough solution to the problem. Fortunately, this goblin is also slayed by judgment. Search is closely related to another venerable AI method called generate and test, where the generate function creates candidates and the test function determines the value of those candidates, and both of those processes benefit from judgment.

For an example of search (or generate and test), imagine that the LLM system is designing a new drug. The Generation LLM needs to propose good candidates or changes to existing candidates, and the Evaluation LLM needs to estimate whether such a candidate might work. Once the Evaluation LLM determines that the candidate might be suitable, it can pass it to the next system for further testing.

Agency + Self-Generation ≈ Life

Programs run on hardware, but hardware is elemental, like oxygen. Computer systems are made of code.

LLMs can write code just as well as natural language, and the simplest way to do it is to pass the code to an external code interpreter tool, as is depicted two figures above. The tool executes the generated code in a sandbox and provides the output to the context. For example, if the user wants to know the number of days until the winter solstice, the LLM can generate a program that computes it and can pass that program to the program interpreter tool. The interpreter can then run the program and pass the output, the computed number of days, to the context. The LLM then reads this context and determines it has the answer and passes it to the user. Such transient code reveals a new truth. Programs have always been held as artifacts, understood and maintained, like contracts between us and our systems. But with LLMs now generating code on the fly, much of it assumes the nature of the spoken word more than of documents. It is formed, executed, then shed. The work flows onward.

Yet the generation deepens. A more sophisticated method sees the Generation LLM crafting the code and the Evaluation LLM determining how to improve that code. This Evaluation LLM can be paired with a verifiable evaluation when one is available, such as whether the code solves a math problem or how fast it can solve it. If multiple candidates are generated and mixed and improved, code generation can be akin to Darwinian evolution, such as with the recent methods of AlphaEvolve and Darwin Gödel Machines. This generated code can have any purpose, such as maximizing some mathematical function or creating a design for a spaceship. The judgment provided by the Evaluation LLM is particularly important when there is no concrete evaluation function and solutions must be judged based on fuzzy criteria.

Code generation can also be used to build new external tools available for the agent, so that the system can expand itself. One example is the Voyager system that builds tools made of code to help it play Minecraft. A system that can extend itself represents a significant advance for machine learning. Neural networks such as LLMs are trained with millions of examples, with each example modifying the model parameters ever-so slightly. This process is too slow. The future lies with autonomous systems that can write new modules in code that makes them smarter in leaps and bounds.

Autonomously writing new modules extends toward life itself. Biological life started because molecules came together that by chance formed systems capable of self-maintenance and self-replication. Life is different from a crystal or something like fire because it actively maintains itself by continually adapting to the environment in a loop. The bacterium E. coli exhibits goal-directed behavior in the sense that it continually adjusts its motion toward food gradients, while fire spreads passively. In addition to having agency, biological organisms can build their own parts, a process called autopoiesis. Computer systems that can write their own code are in a sense building themselves. They have no drive to continue to exist, but we do give them rewards for getting smarter, and as we learn to improve and better harness this judgment, the depth of understanding held by computer systems will become significant.

Life + Self-Awareness ≈ Consciousness

As more code becomes generated by AI, some computer systems may become incomprehensible, so that understanding them will be like disentangling the intricate workings of a biological cell. This transition from humans writing programs to computer systems reading and writing their own code marks a fundamental turning point. In the past, beyond the human author, there was only the CPU blindly executing instructions; there was no one besides the human capable of making judgments about that code to prevent things such as buffer overflows, SQL injections, and other forms of mindlessness. But adding LLMs capable of reading and writing their own code marks the transition of computer systems from passive automatons to active participants. This ability to read their internal programming means that computers can introspect multiple levels down into their own actions and motivations. For the first time, computers can run processes but not be entirely in those processes. They can use internal attention to focus on different aspects of themselves, generating a kind of strange loop that might someday lead to consciousness.

So, to conclude

Thanks to Kyle Ferguson, Courtney Sprouse, and Truman Wilson for their helpful comments and suggestions.

Cross-posted from https://deumbra.com/2025/06/llms-expand-computer-programs-by-adding-judgment/

Large language models (LLMs) such as ChatGPT are surprisingly good, but they can’t expand human knowledge because they only interpolate existing information already on the internet. To create new knowledge and solve problems that no human has ever solved, AI needs to be adaptively embodied and be able to build causal models of the world. Embodiment entails that the AI has purpose, which enables it to be more flexible than stimulus-response agents, such as LLMs, by searching for ways to achieve its goals. Adaptive embodiment means that the AI can build meaning by learning representations and modifying how it maps sensory input to those representations. This learning and mapping gives the AI additional sensitivity to the environment. The final piece of embodiment is that the AI requires an executive system — a kind of proto-consciousness — that enables self-correction at various levels of operation. In addition to adaptive embodiment, the AI must be able to build causal models that enable it to know what to change to achieve its goals and to predict the future by simulating internal states forward.

I’ve read dozens of books since I started working in AI in 2002, and these are the ones that stand out and influenced my thinking. This list focuses on modern books, where I define “modern” as beginning in 1980, after the period of disembodied logic and search, when researchers realized that artificial intelligence needs to be embodied to achieve purpose and meaning. These 30 books presented in chronological order over the last 44 years each gets at a piece of the puzzle for what it will take to move beyond LLMs to expand the capabilities of AI.

Metaphors We Live By (1980) by George Lakoff and Mark Johnson

Modeling human thought through formal methods such as first-order logic seems like a natural choice, but these systems always fall short of capturing the richness of human cognition. Lakoff and Johnson argue for the necessity of less objective representations, proposing that much of human thought relies on understanding abstract concepts through metaphors rooted in lived experience. This metaphorical thinking enables us to grasp abstract concepts like “love” by relating them to more familiar ideas, such as a journey or a war (depending on your relationship). As another example, consider what an “argument” is. An argument can be understood as a container. You can say that someone’s argument doesn’t hold water, you can say that it is empty, or you can say that the argument has holes in it. You can also understand an argument as a building, having either a solid or a poor foundation. These metaphors are much more sophisticated than predicates or rules, and to understand language and abstract concepts, computers must map them to physical or simulated lived experience.

Metamagical Themas: Questing for the Essence of Mind and Pattern (1985) by Douglas Hofstadter

Hoftstader argues that it is impossible to build one model of the world and have it be complete, even for simple phenomena. For example, the shape of the letter “A” in a font can require an unbounded number of levers and knobs to model, with the serifs and where you put the bar across, and on and on. The consequence of this realization is that systems must be able to build their own formal representations on the fly for each new situation. The book is a collection of individual essays, and you don’t have to read them all. I find his writing style slightly off-putting for a reason I can’t quite put my finger on, but there is no denying that he is right.

Understanding Computers and Cognition: A New Foundation for Design (1986) by Terry Winograd and Fernando Flores

Winograd and Flores use the term autopoiesis (coined by Maturana and Varelato in 1972) to describe the process by which a system builds itself by continually constructing what it needs to maintain itself and adapt to new situations. Systems that exhibit autopoiesis are embodied in an environment and their decisions are guided by thriving there. In an embodied agent, the agent doesn’t directly know the environment; it only perceives a small part, and so the outside world can influence the agent’s internal state but not determine it. For example, the frog doesn’t see a fly; it instead only has a corresponding internal perception. The book also discusses conscious control. As an agent lives in a situation, it follows simple behavior patterns that only require attention when things break down. The book invokes Heidegger and states that when you are hammering the hammer does not exist: it only pops into existence when there is a breakdown and something doesn’t work, such as the head falling off. (Or for us computer people, the mouse doesn’t exist until it stops working.)

The Society of Mind (1986) by Marvin Minsky

Marvin Minsky argued that the world can be understood through frames. A frame is a set of questions to be asked about a hypothetical situation, such as a kid’s birthday party, which consists of an entertainment, a source of sugar, and opening presents. The frame of the birthday party sets up the representation for how to talk about it and understand any particular party. This book is a joy to read because it is broken up into many small chapters. He discusses frames, but it mostly focuses on the mind as a society of agents, which is particularly relevant right now because specialized agents are one of the most common ways to deploy LLMs using tools such as LangChain.

The Body in the Mind: The Bodily Basis of Meaning, Imagination, and Reason (1987) by Mark Johnson

This book builds on the metaphorical reasoning of Metaphors We Live By and dives into the patterns that we use to map experience to meaning. Through evolution, our brains developed specific patterns of thought for various situations, with some patterns proving so broadly useful that they are reused across different contexts. Mark Johnson describes those patterns of thought and refers to them as image schemas. The image schemas are lower-level and underpin Minsky’s frames. Frames are specific to cultures, while image schemas are general across cultures. Examples of image schemas include path, force, counterforce, restraint, removal, enablement, attraction, link, cycle, near-far, scale, part-whole, full-empty, matching, surface, object, and collection. Meaningful understanding begins with image schemas.

Mirror Worlds: The Day Software Puts the Universe in a Shoebox … How It Will Happen and What It Will Mean (1991) by David Gelernter

A mirror world is a copy of the physical world in software, a simulation of a system that is continually updated with real-world data. The main example used in this book is a hospital. With a mirror version of the hospital, you can try out new policies and observe their effects. This mirror-world concept has since evolved into what is called a digital twin, which is often a digital version of a machine that can be tested. It also underpins work in simulated environments for robot learning, such as the tool by NVIDIA called Isaac Sim. Simulation is a crucial piece to the puzzle, but to build smart AI, we need systems that can generate their own situation to simulate. As we saw with Metamagical Themas, since situations are unbounded in complexity, simulations need to be autonomously customized to answer the question at hand.

Machine Learning (1997) by Tom Mitchell

Tom Mitchell addresses one of the most overlooked fundamentals in machine learning: the concept of a hypothesis space. A hypothesis space is the space of all models that a machine learning algorithm can choose from. For example, in linear regression with y = mx + b, the hypothesis space is the set of all possible values of m and b. This idea is more significant than people realize because if you can build the right hypothesis space consisting of high-level meanings, such as image schemas, you can do much more powerful learning than you can with a larger, less expressive, space, such as 50 billion parameters in a transformer model. This book is unusual on this list for two reasons: 1. It is the only textbook, and 2. It is only one of two books from the 1990s (they were a dark time). It’s on the list in spite of being a textbook because it is so clear and compelling that you can just sit down in an armchair and read it. It covers the timeless fundamentals of the field, and you can get a PDF copy for free here.

The Way We Think: Conceptual Blending And The Mind’s Hidden Complexities (2003) by Gilles Fauconnier and Mark Turner

Fauconnier and Turner argue that 50,000 years ago humans gained the ability to think flexibly, enabling language and culture to evolve together, which ultimately led to our technological explosion. They argue that this flexible thinking originated from the ability to blend different cognitive frames together. To understand blending, consider the cartoon character Puss in Boots. The humor of the movies comes from the character blending the characteristics of both a human and a cat. Flexible thinking comes from compositionality, and blending is an important way to compose.

What Is Thought? (2003) by Eric Baum

What Is Thought? highlights how structured representations such as image schemas and frames can hold meaning. Baum argues that compression of patterns turns syntax into semantics. Consider again using the linear model y = mx + b to fit a set of data y=f(x) that varies along a feature x. This is called curve fitting. If it fits well, the model is a condensed representation of the process that generates those points. If the condensed model comes from a relatively small hypothesis space, it will likely generalize to unseen points because since there are relatively few models in the hypothesis space, there aren’t enough to find one that overfits the training points. Therefore, if you find a model that works from this limited space, it is likely representing the generative process that made the points, and therefore the model encodes meaning. As we will see in The Case Against Reality, this model may not actually represent reality, but we will also see in The Beginning of Infinity that models serve as explanations, and explanations are powerful.

The Foundations of Mind, Origins of Conceptual Thought (2004) by Jean Mandler

Mandler describes how meaning arises in the human mind. She focuses on how humans use image schemas from the viewpoint of developmental psychology. She describes a process of “perceptual meaning analysis” whereby the brain converts perception into a usable structural form. You can think of this book as a modern version and extension of the work done by Jean Piaget.

On Intelligence: How a New Understanding of the Brain Will Lead to the Creation of Truly Intelligent Machines (2005) by Jeff Hawkins

Hawkins begins with a lucid explanation of the brain and the neocortex. He describes the neocortex as having six layers and as being about the thickness of six business cards. He argues that the cortical column is the right level of abstraction to think about the brain from an AI perspective. And he argues that the neocortex (cortex for short) only runs one algorithm to do all of its diverse processing. He points out that once light and sound enter the brain, the processing is similar, just neurons firing.

He divides the brain into the old brain and the cortex. He describes how the old brain regulates breathing and makes you run in fear. Reptiles only have the old brain — they have this set of 8 or 9 behaviors that they do. The neocortex is a prediction mechanism that sits on top of the old brain. Initially, it allowed the organism to execute its core behaviors better, but as it expanded it allowed for new behaviors. Humans have more connections between their premotor cortex and movement than other animals, that is why we have such fine movements.

For Hawkins, learning is embodied and comes through predictions. He focuses on the importance of sequences in learning. For example, he discusses how our sense of touch isn’t very useful if we don’t move our hand around as we touch something. He also argues that the brain uses memory more than computation because our neurons are slow, and so computation can’t take many levels. Memories are faster, and these memories are used for predictions. As we walk to our front door, we predict how it will look, and when we reach out for the doorknob we predict where it will be and how it will feel, and if something doesn’t match our prediction it is immediately brought to our attention (reminiscent of Winograd and Flores). He also describes our vision as being based on prediction. Consider a picture of a log on the ground; if you take that photograph and blow it up you see that there is no clear line where the log ends and the ground begins; when we look at that log we create the line.

Stumbling on Happiness (2007) by Daniel Gilbert

It sounds like a self-help book, but it isn’t. The key insight is how we simulate the world. Our mental simulations feel complete, but they are not. Imagine your favorite car. Got it? Okay, what does the bumper look like? I don’t know, and you might not know what yours looks like either. Simulation of the world is a key piece needed for AI, but this simulation can’t cover everything, and Gilbert discusses how humans do it.

I Am a Strange Loop (2007) by Douglas Hofstadter

The strangeness of self-reference comes up a lot. Consider the old barber who cuts everyone’s hair who doesn’t cut their own. Does he cut his own hair? If he does, he doesn’t. And if he doesn’t, he does. Self-reference is at the heart of the halting problem and Gödel’s incompleteness theorem. The escape, of course, is that we need a process at one level watching a separate process at the level below. You can’t smash everything into one level. This lack of meta-control is evident when using LLMs. I once had a problem where ChatGPT couldn’t answer questions about calculus. Digging, I found that it could generate the correct Python code, but for some reason it couldn’t run it. ChatGPT didn’t understand this because there was no process watching the process that could have seen where the breakdown was occurring. Hofstadter argues that consciousness could be the result of many such loops of processes watching lower ones, and such a breakdown would be the kind of breakdown that would trigger consciousness to intervene, just as having the head fly off your hammer would make you suddenly aware that you were wielding the tool.

The Philosophical Baby: What Children’s Minds Tell Us About Truth, Love, and the Meaning of Life (2009) by Alison Gopnik

We’ve seen how consciousness serves as an executive and unifier of thought when there is a breakdown. Because adults have more experience than children (and therefore fewer breakdowns), the psychologist Alison Gopnik makes the interesting claim that children are more conscious than adults. That is why it takes them so long to put on their shoes. She also describes how babies and children make causal theories as they interact with the world. Children’s exploration is broader than that of adults, they inhibit fewer actions, and they pay attention to more things. She also describes how consciousness is different for children than for adults. When young, they don’t have an inner monologue, and their consciousness is much more outwardly focused.

Supersizing the Mind: Embodiment, Action, and Cognitive Extension (2010) by Andy Clark

My friends thought it was funny that I would turn my couch upside down when there was something I needed to remember to do later. Clark’s extended mind hypothesis is that the external world is part of our cognition. A significant part of cognition is retrieving information for the task, and that information can be stored in our heads or outside in the world, such as with notes. These computations can happen in loops, such as when we execute the multiple steps of long division with pencil and paper. Clark’s extended mind hypothesis takes on even greater importance as LLMs are beginning to use API calls to other tools to help answer questions. As those API calls spawn their own calls, it may become increasingly nebulous where an intelligent agent begins and ends.

Embodiment and the Inner Life: Cognition and Consciousness in the Space of Possible Minds (2010) by Murray Shanahan

Building on this idea of a breakdown leading to conscious thought, Murray Shanahan explains consciousness as a process that happens when the world breaks our expectations, similar to the discussion in Understanding Computers and Cognition. Lower animals are behavior based; they jump from one behavior to the next. Consciousness is the ability to take competing pieces of those behaviors and recombine them into something new. This recombining happens in a global workspace. Shanahan discusses how consciousness comes from a web in the brain and that parts fight for the right to control that web. It is a dynamical system. Processes jump into the system and try to create coalitions, with the parts being reminiscent of The Society of Mind. Those coalitions that win make it into consciousness.

I’m always surprised at how my consciousness seemingly drops in and out. One time, I was driving and trying to decide if I should get tacos, and the next thing I knew I was in line at Tacodeli. Another time I could feel that my teeth were clean and I could tell that my toothbrush was wet, but I had no memory of the brushing. If you pay attention, you’ll start to notice that your externally focused consciousness isn’t as continuous as it feels.

Mind in Life: Biology, Phenomenology, and the Sciences of Mind (2010) by Evan Thompson

Mind in Life covers embodiment and autopoiesis from a philosophical perspective. Philosophers make me angry because they tend to use jargon only known to other philosophers, and so it is hard to know if they are saying anything useful. But this book made good sense on the second reading. It discusses how embodied agents do not have models of the external world. Instead, they have representations that allow them to act in the world.

How the Mind Works (2011) by Steven Pinker

Pinker describes human cognition from a perspective that straddles psychology and linguistics. Harking back to Metaphors We Live By, the book discusses how there are two fundamental metaphors in language: 1. location in space, and 2. force, agency, and causation.

Steven Pinker is probably my favorite non-fiction author. In this book, he goes into depth on many aspects of human intelligence. He describes how humans view artifacts from a design perspective, so if you turn a coffee pot into a bird feeder, it is a bird feeder. Humans view plants and animals from an essence perspective, so if you put a lion mane on a tiger, it is still a tiger. And humans view animal behavior from an agent perspective. Animal actions (including humans) are governed by goals and beliefs.

He also describes how our brains are not wired for abstract thought — we have to learn that in school. This is evidence that formal approaches such as logic or OWL are insufficient. These kinds of inferences that these systems are built for are not how humans think. Human thinking is less exact but more flexible, and our knowledge is not internally consistent.

And Pinker explains human kinship relationships and romantic love in starkly plain words. “Love” is an emotional binding contract that both people in a relationship can rely on to keep them together, even if one partner comes across a better option. I wonder what 16th-century Spanish poets would think of that.

The Beginning of Infinity: Explanations That Transform the World (2011) by David Deutsch

AI that can extend human knowledge will do so by building causal models. I consider causal models to be synonymous with explanations, and it’s easy to forget how powerful they are. When we look up at the night sky, we see stars; but the “stars” are in our head, and what we actually see is just points of light. The idea of a “star” as an enormous ball of burning gas millions of miles away came to humanity through centuries of work. David Deutsch dives into what makes good explanations. For example, one hallmark of good explanations is that no part can be altered and still explain the phenomena because each piece plays a key causal part.

Louder Than Words: The New Science of How the Mind Makes Meaning (2012) by Benjamin K. Bergen

Our internal representations of the world “mean” something because they are tied to a web of actions and expected outcomes. This book does a great job of describing how knowledge has to be embodied to help an agent interact with the world. Benjamin Bergen outlines how we activate the relevant parts of the brain’s motor system when we understand action words. Consider the example of someone saying that she was bitten while feeding a monkey. Most of us have never fed a monkey, but we all know where she was bitten. (I can’t remember who came up with this example. If you do, please let me know so I can credit it.)

A User’s Guide to Thought and Meaning (2012) by Ray Jackendoff

Ray Jackendoff describes how humans map sensation to meaning. The sensation in his theory takes the form of tokens. While tokens have taken on significantly more importance since the rise of large language models (LLMs), Jackenodoff’s tokens are more generalized than LLM tokens — they are instances of classes, like a particular chair. Jackendoff discusses how these tokens get mapped to a deeper conceptual structure through a process of judgment. This mapping process highlights how LLMs are insufficient because they have little or no internal structure to map to.

The Geometry of Meaning: Semantics Based on Conceptual Spaces (2014) by Peter Gärdenfors

Peter Gärdenfors discusses the nuts and bolts of language understanding, beginning with how conversation is based on shared reference. The first step is to physically point to objects in the world so that we and our conversational partner know that we are referencing the same things. Before children can speak, they use this pointing to convey desire for objects and for shared interaction. This process is a springboard, because as we learn language, words are able to point to shared ideas in our minds.

My favorite part is his description of how we negotiate meaning in the course of a conversation. A coordination of meaning is required when two people have two different ways to describe concepts. And a coordination of inner worlds is required when one person has a concept not shared by the other. LLM-based systems currently do a poor job of learning during a conversation and using what they have learned in future conversations with you.

The Book of Why: The New Science of Cause and Effect (2018) by Judea Pearl

Causal models are required for general intelligence, and Pearl is the authority on causality in AI. I dislike the phrase “correlation does not imply causation” because while it is true, correlation does tell you where to start looking. Pearl’s book explains how to tease out the difference and to go from correlation, which doesn’t enable an agent to control its environment, to causal models, which do. If you like the story told in this book and want to go deeper, you can then move on to his other books Causality and Probabilistic Reasoning in Intelligent Systems.

Rebooting AI: Building Artificial Intelligence We Can Trust (2019) by Gary Marcus and Ernest Davis

I consider LLMs to be a wonderful surprise because I never thought we could get this far just by predicting the next word, but LLMs are not sufficient for full intelligence, and Marcus and Davis explain why. Consider one of their examples: Person A says they found a wallet, and person B immediately checks their back pocket. Why did person B do that? Marcus and Davis argue that what is needed is AI that uses reusable and composable abstractions to model its environment.

Rethinking Consciousness (2019) by Michael S. A. Graziano

Graziano argues that the key to consciousness is attention. Overt attention is looking at something out in the world, but covert attention is paying attention to something you are not looking at. He says that covert attention is the key to consciousness. Since you can attend to internal things, you need a model of that attention. This creates consciousness.

The Case Against Reality: Why Evolution Hid the Truth from Our Eyes (2019) by Donald Hoffman

Donald Hoffman argues that evolution endowed us with the ability to perceive the world to survive but not necessarily to know how the world actually is. He uses the analogy of an icon on your desktop. It takes you to the program, but it doesn’t represent what is happening underneath. He also uses apples as an example. When you hold an apple, there is no apple in your hand. The apple is in your brain. There is physical stuff out there, but we have no way of knowing what it is really. We only know that our perception of the apple is good enough to survive. This idea that we have representations to survive but not objective models of the world ties back to Mind in Life by Thompson and even back to Plato with his allegory of the cave.

A Thousand Brains: A New Theory of Intelligence (2021) by Jeff Hawkins

Like his other book, this book also provides a clear and understandable discussion of the brain. Jeff Hawkins views intelligence in terms of sequences of states and grids of states related together. For example, the handle of a coffee mug has a spatial relationship to the bottom that is consistent as the mug moves around.

Journey of the Mind (2022) by Ogi Ogas and Sai Gaddam (first half)

They put forth a compelling biological progression of how intelligence could have evolved. The book takes us step-by-step from life’s inception to smart animals. It describes the formation of the first effector and the first sensor. My favorite part is when Ogas and Gaddam describe how purpose came into being on our planet when, by chance, the first sensor element connected to the first effector. You can read just the first half; I didn’t enjoy the second half as much.

The World Behind the World: Consciousness, Free Will, and the Limits of Science (2023) by Erik Hoel

This book starts as an insider exposé on the limitations of neuroscience research results. It then pivots to philosophical themes such as the zombie hypothesis: whether it is possible to live as we do without any kind of conscious experience. But my favorite part is the last quarter where he talks about finding causal models at different levels of abstraction. Using the principles of error correction, Hoel shows that there are causal relationships that are easier to identify at higher levels of abstraction than at lower levels. By trading off with information loss, it is possible to find the best level of abstraction. Erik Hoel also has related work on dreams being low-fidelity abstractions. Because of this low fidelity, the movie Dream Scenario, where a guy becomes famous by appearing in everyone’s dreams, couldn’t happen. Dreams are too low fidelity to register a new face that we would recognize in real life.

A Brief History of Intelligence (2024) by Max Bennett

Simulating the world forward is likely the key to general intelligence. This book beautifully explains intelligence from its biological origins, and it describes how analogs of different AI algorithms might be running in the brain, ultimately leading to these simulations. The explanations are undoubtedly simplified, but an incomplete scaffolding is always a good place to start.

Conclusion

The next-word prediction method of LLMs will not get us to real artificial intelligence. It’s possible that we can design and train a new neural architecture in an end-to-end fashion that does get us there, but since we don’t have evolutionary time, energy, or computation, it is more likely that we will need to implement the ideas in these books.

The answer that emerges from the books in this list is that for an AI to understand the world and invent new things it must be able to take the current situation and put it on an internal mental stage. It should then simulate that situation forward to predict the future. That simulation should run at the level of abstraction appropriate to the question it wants to answer. To aquire the knowledge necessary to build the stage and run the simulation, it should be able to read the internet and convert it to its own knowledge format (not just to predict the next token) and be able to act in a real or simulated environment to obtain a grounded understanding and run experiments. When real AI emerges, it will likely take the form of a hybrid system that uses neural networks to make subtle decisions in perception and action and to dynamically build the appropriate symbolic representation for each task. I don’t know how long this AI will take to build, but hopefully it won’t be another 44 years.

Thanks to Noah Mugan for helpful comments and suggestions.

Cross-posted from https://deumbra.com/2024/08/the-top-30-books-to-expand-the-capabilities-of-ai-a-biased-reading-list/

120 Years of Reinforcement Learning

We first fly through 120 years of reinforcement learning work. If you want to get right into RLlib, fell free to skip to the next section.

Thorndike observed that some behaviors in animals arise from a gradual stamping in [Thorndike, 1898]. In psychology, this stamping in became the study of Behaviorism [Skinner, 1953] (see the Skinner box Figure 1). Behaviorism was the dominant paradigm for investigating human psychology and learning in the middle of the 20th century. It focused only on the responses that could be observed and studied the different conditions and training regimens that led to those responses. It was popular because it wasn’t’ fluffy — you could make testable predictions. The behaviorism framework proved too limiting to describe human psychology, but it showed itself to be a useful formulation for learning in artificial agents such as robots [Sutton and Barto, 1998], where it is called “reinforcement learning” (often abbreviated as RL).

At its simplest, in reinforcement learning an agent begins by randomly exploring until it reaches its goal, as depicted in Figure 2.

When it reaches the goal, credit is propagated back to its previous states, as depicted in Figure 3. The agent learns a function Q(s,a), which gives the cumulative expected discounted reward of being in state s and taking action a and acting according to the current policy thereafter. The values of Q(s,a) are depicted as the size of the arrows in Figure 3.

The agent starts out with a policy of just taking random actions. The policy is continually updated from experience, and eventually the agent learns the value of being in each state and taking each action and can therefore always do the best thing in each state. This behavior is then represented as a final policy, as depicted in Figure 4.

In the examples in the previous figures there was a discrete state and action space. This discrete state space is limiting because as the number of variables that makes up your state grows the size of the space grows exponentially. This means that you can’t visit all possible states, much less visit each one enough to learn Q(s,a). Because of this limitation, modern RL uses neural networks, as depicted in Figure 5.

RLlib automatically constructs neural network (deep learning) models behind the scenes based on your configuration. This tutorial focuses on RLlib.

RLlib is particularly good for learning with multi-agents, such as swarms, or learning across multiple machines. If you are just looking for straightforward RL with a single agent on a single machine, I recommend starting with StableBaselines3. If you want to learn more about reinforcement learning in general, see these great resources. On to RLlib.

Overview of RLlib

RLlib was built to solve the problem of distributed RL, as described in this paper. Parallel training in RL is hard because you must keep the policies in synch. RLlib solves this problem by using Ray. Ray is a system for parallel training, and RLlib is built on top of it. We see a depiction of parallel training in Figure 6.

Each copy of the environment can be running on a different computer. The environment is where the agent is actually acting in the world. Each worker uses a copy of the current policy to choose actions for the agent that are executed in the environment. The reward function then determines how good that resulting state is, and data samples in the form of (state, action, reward) tuples are sent to a central processor. The central processor then uses those samples to update the policy, which is broadcast out to the workers on the different machines. This policy update can be done with TensorFlow or PyTorch.

Parallel training is necessary for scaling, but for me the most exciting part of RLlib is the abstractions it provides for hierarchical multiagent reinforcement learning, and that is what we will focus on in the next couple of sections.

Hierarchical, Multi-Agent Example Environment

For this tutorial, we created a simple, custom, multi-agent, hierarchical environment to show you how to adapt RLlib to your problem. The environment and the code can be found here. Using a custom environment prevents us from glossing over any necessary details for using RLlib in your system. The environment is silly but entails the necessary complexity.

The environment of two robots in a chicken yard is shown in Figure 7. The robots want to capture (move to) the chickens that are most like them. In this scenario, each robot and each chicken has a personality based on the OCEAN model. The reward for the robots comes from getting close to a chicken that has a similar personality to the robot. The robot is also punished by life on each timestep (it receives a small negative reward per timestep), and its reward when it reaches the chicken is computed as the dot product of its personality with that of the chicken. For example, neurotic robots are more highly rewarded for finding equally neurotic chickens.

This environment is simple but it is multiagent and hierarchical. There are two robots, and each robot uses two policies:

• High-level policy: pick a chicken

• Low-level policy: move to chicken

The way the high-level policy sees the world is as a list of chickens, where each chicken’s personality is represented with a five-dimensional vector, and each chicken has an x and y location. The action space of the high-level policy is which chicken to choose.

The low-level policy is for when a chicken is already chosen, so it sees the world as the position of the robot x and y and the position of the chosen chicken, x and y. The action space of the low-level policy is to go in 8 directions: N, NE, E, SE, S, SW, W, and NW.

The episode ends when both robots get within a distance threshold of a chicken. The environment is shown in YourTargetSystem and YourAutonomousAgent in the repo. We will cover how the state and the action spaces for both policies are represented in the next section (Error! Reference source not found.).

RLlib Abstractions

We can think of RLlib in terms of its main abstractions shown in Figure 8. We will refer to this diagram as we go through the rest of the tutorial.

OpenAI Spaces

OpenAI created a the OpenAI Gym framework for reinforcement learning a few years back. Gym provides nice abstractions that are carried forward into both StableBaselines3 and RLlib.

The first significant piece of Gym is spaces, which make it easy to format your domain for neural networks. The observation spaces of the high-level and low-level policies are from your_openai_spaces.py are shown here.

robot_position_space = spaces.Box(
             shape=(2,),
             dtype=np.float,
             low=0.0,
             high=1.0)
 
 robot_ocean_space = spaces.Box(
             shape=(NUM_OCEAN_FEATURES,),
             dtype=np.float,
             low=0.0,
             high=1.0)
 
 high_level_obs_space = spaces.Dict({
         'chicken_oceans': chicken_ocean_space,
         'chicken_positions': chicken_position_space,
         'robot_ocean': robot_ocean_space,
         'robot_position': robot_position_space,
 })
 
 low_level_obs_space = spaces.Dict({
         'robot_position': robot_position_space,
         'chicken_positions': chicken_position_space,
 })

We see that we can represent these spaces in a semantically meaningful way. The beautiful thing about OpenAI spaces is that the learning agent can automatically translate this representation into input for a neural network.

OpenAI Gym Environment

The second significant piece of OpenAI Gym is the Env class. Env provides an uniform API to interact with your environment. We see in Figure 8 that the Env class has four main methods.

1. reset

2. step

3. render

4. close

Reset starts the episode over, render gives you an opportunity to visualize your environment, and close lets you close down.

The most important method is step. The input to the method is the action chosen by the policy. The output from the method is the observed state and the reward. The learning algorithm then uses this observation and reward to update the policy. The output for the method also includes whether the episode is done and any other important information you want to add through customization. The step method is run inside a worker in Figure 6.

I mentioned previously that I thought the most exciting part of RLlib is its abstractions for working with multiple and hierarchical agents. RLlib enables this functionality by extending OpenAI Gym Env class to work for multiple agents with the class MultiAgentEnv. In the step method of MultiAgentEnv the actions from the policies come in as a dictionary with one key for each agent name. Likewise, the information passed out is a dictionary with the units as keys (more on this later).

You control which agents are active by returning observations for only those agents, as you can see in the step function of your_rllib_environment. If an agent is not active, you don’t supply it an observation when returning from the step method. If there is no observation for an agent, then when the step function is called on the next timestep that agent won’t have an action. In this way, the RLlib MultiAgentEnv class enables you to dynamically say which of your agents are acting at which timesteps and you can specify what their observation and reward should be if they are active.

In fact, you can have multiple virtual agents per physical robot as a hierarchy of agents, as shown in Figure 9. You can also have a swarm of agents that share a policy so that anything learned by one agent in the swarm transfers to the other agents.

In particular, the setup for our environment is shown in Figure 10. This mapping is specified in our config file, your_rllib_config.py.

Testing Your Environment

Reinforcement learning is hard to debug. I don’t believe that life always finds a way, but RL agents will adapt to the world you give them, even if that world is not what you think it is. So a good way to start is with a random policy to test that your environment works as expected.

Initially, we factor RLlib out for testing so we can confirm that the environment works as expected, as shown in demo/demo_your_rllib_env.py and excerpted below

 env = YourEnvironment(config)
 env.reset()
 
 action_dict = {'robot_1_high': random.choice(range(NUM_CHICKENS)),
                 'robot_2_high': random.choice(range(NUM_CHICKENS))}
 
 obs, rew, done, info = env.step(action_dict)
 env.render()
 
 while not is_all_done(done):
     action_dict = {}
     assert 'robot_1_low' in obs or 'robot_2_low' in obs
     if 'robot_1_low' in obs and not done['robot_1_low']:
         action_dict['robot_1_low'] = \   
             random.choice(range(NUM_DIRECTIONS))
     if 'robot_2_low' in obs and not done['robot_2_low']:
         action_dict['robot_2_low'] = \
             random.choice(range(NUM_DIRECTIONS))
     obs, rew, done, info = env.step(action_dict)
     print("Reward: ", rew)
     time.sleep(.1)
     env.render()

The code first passes in random high-level actions, and then it loops over random low-level actions until both robots have reached a chicken. This is a useful way to see what is happening in the step function. Notice that we pass in actions only for agents (high or low) that we want to act.

Learning a Policy

To train your policy, you pick a trainer (algorithm) and tie your environment to it. RLlib comes with trainers for most of the standard algorithms, such as Proximal Policy Optimization (PPO) and Deep Q Networks (DQN). There are two ways to run your trainer as shown in your_rllib_train.py and excepted below

if RUN_WITH_TUNE:
 
     tune.registry.register_trainable("YourTrainer", YourTrainer)
     stop = {
             "training_iteration": NUM_ITERATIONS  
             # Each iter is some # of episodes
         }
     results=tune.run("YourTrainer",stop=stop,
                     config=config,checkpoint_freq=10)
 
     # You can just do PPO or DQN but we wanted to show 
     # how to customize
     #results = tune.run("PPO",stop=stop,config=config,
     #       verbose=1,checkpoint_freq=10)
 
 else:
     from your_rllib_environment import YourEnvironment
     trainer = YourTrainer(config, env=YourEnvironment)
 
     # You can just do PPO or DQN but we wanted to show 
     # how to customize
     #from ray.rllib.agents.ppo import PPOTrainer
     #trainer = PPOTrainer(config, env=YourEnvironment)
 
     trainer.train()

The first way is with Ray Tune. Ray Tune was originally developed for hyperparameter tuning, but works well for running RL algorithms. The second way is to run the trainer directly.

With Tune, the environment is specified in the config dictionary. When running it directly, you can specify the environment as a separate parameter. Regardless, configuration is a big part of the flexibility of RLlib, and it is worth taking a few minutes to go over it. The configuration is stored in a giant dictionary that comes from multiple levels of inheritance. The first level is at the general trainer level. Then, the type of trainer you use inherits it and adds to it, for example with PPO. Finally, you add to the config for your particular implementation; for example, to map agents to policies.

You can also create a custom trainer. The custom trainer is actually used in the tutorial code to demonstrate how that is done, but you can comment it out and uncomment the standard PPO trainer. A custom trainer allows you to do any operation you want through a custom execution plan. It can be a little trickly to get it all working together, but there are few limits to customization.

Looking at your training results

As shown in Figure 11, when you run your_rllib_train.py you get 83 graphs in TensorBoard.

To see these on your machine, Go to /Users/you/ray_results to find your run and type “tensorboard logdir=./”.

I let it run on my laptop for about an hour. That’s a long time for a small environment, but we didn’t add any smart features, and if you want, RLlib lets you put this directly on a cluster of machines by increasing the number of workers and turning off local mode ray.init(local_mode=False).

Figure 12 shows the mean policy rewards over time. The two robots share both a high-level policy and a low-level policy. The high-level policy learned at the beginning but there is a lot of randomness (evidently, finding the right chicken is hard). In Figure 12, the x-axis is timesteps, and the y-axis is reward.

Recall that the robots are punished a small amount on every timestep. This “reward” encourages them to find chickens increasingly quickly, as shown in Figure 13. In the figure, the x-axis is timesteps, the y-axis is the number of timesteps it takes for both robots to find a chicken.

Loading and using the policies

It’s no good to train if you can’t use the policy. After the policy is learned, we can load it up and get the best action for any state in demo/demo_after_training.py. In the excerpt below we see we see how to load a trainer with saved parameters.

from your_rllib_environment import YourEnvironment
trainer = ppo.PPOTrainer(config, env=YourEnvironment)
run ='YourTrainer_YourEnvironment_d9c79_00000_0_2022-08-18_21-37-21'
checkpoint = 'checkpoint_000500/checkpoint-500'
restore_point = os.path.join(YOUR_ROOT, run, checkpoint)
trainer.restore(restore_point)

And below we see how to call compute_single_action to get a best action from the learned policy for a current state.

robot_1_high_action = \
       trainer.compute_single_action(obs['robot_1_high'],   
       policy_id='high_level_policy', explore=False)
robot_2_high_action = \       
       trainer.compute_single_action(obs['robot_2_high'],  
       policy_id='high_level_policy', explore=False)
 
 action_dict = {'robot_1_high': robot_1_high_action,
                 'robot_2_high': robot_2_high_action}
 
 obs, rew, done, info = env.step(action_dict)
 
 env.render()
 
 def is_all_done(done: Dict) -> bool:
     for key, val in done.items():
         if not val:
             return False
     return True
 
 while not is_all_done(done):
     action_dict = {}
     assert 'robot_1_low' in obs or 'robot_2_low' in obs
     if 'robot_1_low' in obs and not done['robot_1_low']:
         action_dict['robot_1_low'] = \
             trainer.compute_single_action(obs['robot_1_low'], 
             policy_id='low_level_policy')
     if 'robot_2_low' in obs and not done['robot_2_low']:
         action_dict['robot_2_low'] = \
             trainer.compute_single_action(obs['robot_2_low'], 
             policy_id='low_level_policy')
     obs, rew, done, info = env.step(action_dict)
     print("Reward: ", rew)
     env.render()

You can also look at the policy itself, as we see in demo/demo_look_at_policies.py. You can look at the actual parameters and the size of the automatically created neural network.

from your_rllib_environment import YourEnvironment
trainer = ppo.PPOTrainer(config, env=YourEnvironment)
 
# Change these for your run
run = 'custom_execution_plan_2022-02-24'
checkpoint = 'checkpoint_000500/checkpoint-500'
 
restore_point = os.path.join(YOUR_ROOT, run, checkpoint)
trainer.restore(restore_point)
 
print("************** high level policy *************************")
# Note the output size of 10
policy: Policy = trainer.get_policy('high_level_policy')
#https://github.com/rayproject/ray/blob/releases/1.10.0/rllib/models#/torch/complex_input_net.py
model: ComplexInputNetwork = policy.model
for m in model.variables():
     print(m.shape)
 
print("**************** low level policy ***********************")
# Note the output size of 8
policy: Policy = trainer.get_policy('low_level_policy')
model: ComplexInputNetwork = policy.model
for m in model.variables():
    print(m.shape)

Conclusion

RLlib is complicated but powerful. It is particularly good for training swarms of agents and for providing flexibility to do just about whatever you want. After you go through this tutorial, you can look at the repo, especially the examples, and when you get stuck you can post something on the discussion.

Originally published at https://deumbra.com/2022/08/rllib-for-deep-hierarchical-multiagent-reinforcement-learning/

References

Thorndike, E. L. (1898). Animal intelligence: An experimental study of the associative processes in animals.

Skinner, B. F. (1953). Science and human behavior. Simon and Schuster.

Sutton, R. S., & Barto, A. G. (1998). Reinforcement Learning. MIT Press.

Credits

Icons from publicdomainvectors.org https://publicdomainvectors.org/

robot https://publicdomainvectors.org/en/free-clipart/Yellow-robot/81372.html

chicken https://publicdomainvectors.org/en/free-clipart/Vector-illustration-of-cartoon-chicken-confused/24675.htm

The ephemeral nature of thought is even more pronounced in our nightly dreams, where our thoughts feel like simulated training based on a curriculum derived from our failures, our unresolved conflicts, and our fears. You may dream of being back in high school or that your car is sinking into a lake, and you think to yourself, “Next time I should roll down the windows before the water reaches the roof.” But most dreams are forgotten, and all of that experience, seemingly as real as our waking life, is lost.

Humanity no longer has to accept this loss. By standing on the shoulders of biomedical and machine-learning giants, DeUmbra has partnered with the University of Buda to develop a technology that tracks and writes down your thoughts. We call it ThoughtLog. ThoughtLog uses an encoder-decoder (deep neural network) to translate brain waves to text. Encoder-decoder technology is what enables automatic language translation to finally work. Computers can translate English into French by training on identical documents written in both languages. The amazing thing is that these algorithms extend beyond text. Using the same method of training, they can generate captions for images. ThoughtLog takes this one step further: instead of translating images to text, it translates brain waves into text.

We have been training ThoughtLog for over 8 years with hundreds of volunteers. The process is that we ask the volunteers to tell us what they are thinking while the University of Buda brain wave detector converts their brain waves into vector embeddings (a mathematical representation). Using these embeddings along with the text representations of what the volunteers have reported as training data, the deep learning algorithm learns to match their brain waves to their descriptions of their thoughts. The result is that ThoughtLog can convert your thoughts into text, even while you are dreaming. It’s hard to imagine that such a thing is possible. A small company in Austin Texas has cleaved human history between what is lost and what is preserved.

Even more astonishing is that DeUmbra can provide ThoughtLog for free. Using ThoughtLog requires a minor surgery to insert the brain wave detector and processor, and although medical procedures can be expensive, we are proud to offer this service at no cost for our users. All expenses of ThoughtLog are paid by our sponsors. A few times a day, our sponsors will cause ThoughtLog to generate brain waves in our users that trigger thoughts about their products and services. This functionality is possible because neural network encoder-decoder models can work in both directions, and the University of Buda brain wave detector can also trigger neurons to generate waves.[1] There will be minimal “commercial breaks” during your waking hours because many of the sponsored thoughts about products and services can be sent to you while you are sleeping.[2]

No need to worry. ThoughtLog won’t bother you about products and services that you don’t like. Our partners will note your arousal responses to sponsored thoughts and will use that information to craft messages tailored to you. In fact, by thinking positive thoughts about the products and services of our sponsors, you can even earn ThoughtPoints! ThoughtPoints can be redeemed for discounted products and services, much like airline miles can.

You can rest assured while using ThoughtLog because all of your data will be stored securely in the cloud. Being stored in the cloud means that you can give access to your loved ones. Sharing your ThoughtLog stream is yet another way to gain trust and demonstrate affection. The data will also be made available to our partners in law enforcement, which means that your use of ThoughtLog will help to make our cities safe. When a crime is committed in an area, the authorities will be able to quickly home in on a list of potential suspects. ThoughtLog also enables workforces and schools to be more effective. If you are an employer or a school administrator, please reach out to our sales people about our enterprise edition.

At DeUmbra, we care deeply about creating a more just and moral civilization. To push society forward, our customers can also earn ThoughtPoints by thinking encouraged thoughts. To stay up-to-date on what thoughts should be encouraged and discouraged, DeUmbra continually applies deep learning algorithms to analyze tweets and reactions on Twitter. Thoughts in tweets that receive many likes and retweets and responses with positive sentiment are stored as “encouraged,” and thoughts in tweets that receive little attention or responses with negative sentiment are stored as “discouraged.” By using vector similarity, ThoughtLog can compare your thoughts to our lists of encouraged and discouraged thoughts in real time. Having thoughts that are similar to those on our encouraged list will earn you ThoughtPoints, and our customers will become better people because having discouraged thoughts will not only deduct ThoughtPoints, such thoughts will cause ThoughtLog to trigger brain waves that result in moderate, but physically harmless, pain.

Our customers will also earn ThoughtPoints for any useful intellectual property they create while using ThoughtLog. Customers won’t have the burden of owning the ideas they generate, but they will instead receive ThoughtPoints if their thoughts are useful to our sponsors. You could hear your thoughts later as a hit song on the radio or even see them in a patent! In addition to organic thoughts leading to ThoughtPoints, customers can also earn points by mentally answering survey questions posed by randomly timed prompted thoughts. Imagine earning points by giving your opinions on different packaging options for hot dogs while spending time with those you love.

ThoughtLog is currently in closed beta. Please reach out to us if you would like an invitation to be a pioneer.

[1] Generating brain waves is a harder problem than reading them, and the brain-wave generator technology is still imperfect. Most volunteers in our trials can distinguish between organic and sponsored thoughts.

[2] We find that the majority of volunteers are not woken up by sponsored thoughts and are able to seamlessly incorporate them into their dreams.

Originally posted at https://deumbra.com/2022/04/introducing-thoughtlog-the-free-automatic-diary/

Microsoft Flight Simulator (MSFS) 2020 doesn’t just feel real; it almost is real. You see the same objects and relationships when you look out the window that you would see out of a physical plane. Down there are your own cities and streets, and you can even fly into a live hurricane [1]. Because of its fidelity to the real world, MSFS represents an inflection point in virtual worlds. Almost all video games are based on fictional worlds, but MSFS is played within our own. By representing our planet so well, MSFS shows how having games set in our world creates value beyond entertainment.

Entertainment is weird. It hijacks our drive for information and puts us in a stupor, but entertainment’s beneficial side effect is that it gives us life experience on the cheap. The Shawshank Redemption taught us what prison in the 1940s might have been like. Of course, movies lie, but they provide initial models of new experiences that we can later refine with thoughts and further evidence. The experience gained through video games is even more potent because we take actions, and MSFS brings that experience with actions to our world.

I’m not an aviation guy (I don’t like the noise pollution private planes and helicopters make [2]), but I now have some sense of what it’s like to fly, and I can take my kids to see the Grand Canyon in the middle of the pandemic. I’ve always dreamed of video games that took place in my neighborhood. Imagine shooting zombies in your local grocery store or driving to work on your morning commute and thinking, “I always get shot down over this intersection.”

Video games set in a simulation of the real world will do more than enrich our experience: they will also teach our future robot helpers. To become intelligent, robots need real-world experience for two reasons. First, we could never explain the world in enough detail for them to understand it. You have to be there. Think of how hard it would be to describe a gas station to someone who had never been to one. No matter how hard you tried, they wouldn’t get the correct big picture, and you would almost certainly leave out small details, like all the questions you have to answer before they let you pump the gas: zip code, car wash, rewards program, and favorite food. It’s the same with a grocery store or even flying as a passenger in an airplane (did you remember to mention the little vent above your seat?)

The second reason robots need real-world-like environments is that they must create knowledge on their own by generating and testing hypotheses. Robots need causal models of the world to be able to handle novel situations, and the easiest way to build causal models is to run little experiments, “What happens if I flip this switch?” Experimentation, of course, is how children learn [3], which is why we keep them away from dangerous or fragile equipment until they are older. In a simulated world, the robots can break things all they want, and simulated experience is good for robots for the same reason it is good for us — it’s cheap. Cheap experience is even more critical for robots because they require a lot of maintenance, especially if we plan on letting them go around flipping random switches.

Besides having all of the familiar landmarks in place, another way to characterize a virtual world’s realness is a measure we can call “conceptual depth.” Conceptual depth is how well the dynamics of the game match the dynamics of the real world. For flying, MSFS is obviously good, that’s the stated point of the game, but we need a lot more if we want to get out of the airplane. Does paper tear? Does water spill if you pour a big cup of water into a small one? Does the spill make the counter slippery?

In most games, the ground is a 2D surface with some assets on top of it (apparently even in MSFS [4]). In the real world, when you walk outside on the grass, the ground is a maze of plants and dirt and organisms. It’s limitless. For humans or AIs to learn in virtual worlds, we need the freedom of action and consequences that match those of the real world. We’re getting there. In racing games from the 1980s, you were just a square that only moved left and right as objects came toward you, but now real-world details are slowly and consistently being added. To have common sense, a robot needs to “grow up” in a rich environment, and as the virtual worlds become more real and detailed, our AIs will get smarter.

I currently can’t kill zombies behind my local Arby’s, and my robot assistant can’t yet run amuck at the neighborhood grocery store. MSFS shows the correct roads, but it only renders generic buildings in areas outside of major cities or well-known attractions. We can do better. For each neighborhood, we could find the standard floorplans used when that neighborhood was built and generate from that. Using such realism brings up intellectual property issues. Should a company be able to scrape the real world to populate a virtual one? I hope so, because it serves a public good. Otherwise, our virtual worlds won’t be as fun. Trying to contact and pay every rights holder could be endless. One positive example is that Activision won a court case to depict Humvees without a license [5]. By contrast, according to [6], Sony removed the One World Trade Center (Freedom Tower) from its depiction of New York in its Spider-Man game, possibly because buildings can be trademarked. Another issue is privacy. If I own a Honda Accord, can a virtual world show that in my driveway? What about my poorly maintained lawn [7]? Although privacy could get messy as well, we can see that Google already shows my front yard with Street View.

I’ve always wanted a Google Street View with a better interface so I could explore new places, and MSFS moves us toward a Street View where you can fly. MSFS is an inflection point for virtual worlds because these aren’t just video games anymore; they are mirror worlds [8] where we can enrich our life experience, and robots can approximate what it is like to be human. There’s the old idea that the entirety of our world is merely a flight simulator built by a more advanced civilization. Maybe that idea subconsciously contributes to our drive to understand the universe, so we can outgrow our sandbox.

Originally posted at https://deumbra.com/2020/09/microsoft-flight-simulator-2020-is-an-inflection-point-for-virtual-worlds-and-our-own/

References and Comments

[1] https://www.theverge.com/2020/8/27/21403769/hurricane-laura-microsoft-flight-simulator

[2] They should have to pay me $20 every time they fly over while I’m trying to enjoy nature.

[3] Alison Gopnik, The Philosophical Baby: What Children’s Minds Tell Us About Truth, Love and the Meaning of Life, 2009.

[4] Video flying over Austin (with a better computer than my son’s) ending with sitting on the surface of water https://www.youtube.com/watch?time_continue=367&v=-XP5RymsqK0&feature=emb_logo&ab_channel=Squirrel

[5] https://www.theverge.com/2020/4/1/21203414/activision-wins-humvee-call-of-duty-trademark-infringement-lawsuit

[6] https://www.stevenbuccini.com/why-1wtc-isnt-in-spiderman with Hacker News discussion https://news.ycombinator.com/item?id=24502706

[7] When my parents came to visit once, my mom commented, “It looks like nobody lives here.” I tried to explain that my preferred aesthetic is “apocalypse chic.”

[8] David Gelernter, Mirror Worlds: Or: The Day Software Puts the Universe in a Shoebox…How It Will Happen and What It Will Mean, 1991.

The figure above depicts neural text generation. The world state is a representation, such as an image or written sentence. The encoder is a neural network that maps the representation into a meaning space. This meaning space is a space of vectors, like the popular word vectors. These vectors are long sequences of numbers, and the space is like the familiar two-dimensional space with x and y coordinates, with the exception that there are hundreds of coordinates instead of just two. The decoders are neural networks, and there can be multiple decoders, each capable of writing a version of the meaning in a particular language or style.

Neural text generation has many uses. It is the state-of-the-art method for machine translation, where the world state consists of a sentence in a source language, which is encoded into meaning space and then decoded into one or more target languages. Neural text generation can also be useful for chatbots, where a line of dialog is encoded into a meaning space and the decoder maps that point in meaning space into a reasonable response. The method is also used for image captioning, where an image is encoded into the meaning space and then decoded into a caption. The meaning space can also be decoded into various styles of writing. Imagine having your website be dynamically customized to match the preferences and language patterns of the user. Even better, imagine being able to have your boring HR manual translated into the style of Cormac McCarthy. Something like, “The coffee in the break room is available to all employees, like some primordial soup handed out to the huddled and wasted damned.”

Neural text generation, while useful, is currently limited by a lack of common sense. It is often imprecise; if you ask a personal assistant to make you a reservation on Tuesday, it may make the appointment on Wednesday because, you know, the word vectors are pretty close (also see [50]). Neural text generation also makes mistakes that no human would make. For example, in image captioning it can mistake a toothbrush for a baseball bat [31]. But the beautiful thing is that our neural networks are getting richer, and they can show flexibility and learn from large amounts of data. As the meaning space gets closer to being able to represent the concepts that a small child can understand, we will get closer to human-level literacy. We are taking baby steps toward real intelligence.

This blog post will present an overview of neural text generation. We will start with sequence-to-sequence models, which encode source sentences and convert them to target sentences, as is done for language translation. We will then discuss the space these sentences are mapped to and how we can ensure that it is well organized, and how we can move around within this meaning space. Then we will discuss neural text generation when we don’t have pairs of sentences to train from. This will take us into generative adversarial networks (GANs) where a text generator and a classifier train each other in a virtuous arms race. We will also discuss how reinforcement learning (RL), an idea stemming from behaviorism in psychology and the training of circus animals, can be used to teach a machine how to generate useful text. Finally, we will conclude with some suggestions on when it is good to use neural text generation compared with traditional template-based generation, and we will discuss some challenges when using neural text generation.

Sequence-to-Sequence Models: Learning from Pairs of Sentences

Seq2Seq Overview

A sequence-to-sequence (seq2seq) model consists of an encoder and a decoder, and it converts one sequence of words into another. The encoder converts the source sequence of words into a vector in meaning space, and the decoder converts that vector into the target sequence of words. Both the encoder and decoder are often recurrent neural networks (RNNs). The initial big application for seq2seq models was machine translation. For example, a model can be trained to convert source sentences in English into target sentences in Spanish.

We see a depiction of the encoding process in the figure below. As each word is examined, the encoder RNN integrates it into the hidden state vector h. The vector h4 is the final state vector and contains the information in the encoded sentence. The vector h4 represents a point in the meaning space of possible meanings.

Now consider the task of decoding the sentence in the figure above into Spanish, as is shown below. We start with state h4 from the encoded sentence. We use the decoder RNN to generate the first word “La” and then the decoder uses the previous state and the last generated word to update its state to h5. It keeps doing this until it generates a stop symbol. Importantly, the number of words in the target sentence doesn’t have to be the same as the number of words in the source sentence. It stops when it generates the stop symbol, which in this case is a period.

We now see the whole seq2seq process in the figure below, with the final vector h4 of the encoding in meaning space.

Encoders

Let’s take a closer look at the encoder and how it iteratively integrates the words in a sentence into the hidden state h. In seq2seq models, words are represented as vectors stored in a vocabulary table, which represents each word in the vocabulary by a vector of a fixed dimension, such as 300. In the simplest case, we have a fixed number of words in our vocabulary, such as 50,000. We therefore have a vocabulary table of size [50,000 x 300] as shown below.

The vectors in this vocabulary table are learned as part of model training, and it results in word vectors, similar to those learned with word2vec [33]. It is worth noting that instead of using words as tokens, we can also break words into segments and use those instead. For example, the word “abomination” might be represented as the three tokens, abom@@, in@@, and ation. Here, the “@@” indicates that this word is not finished yet. The advantage of using word segments instead of whole words is that you no longer have this problem of words not being in the vocabulary. If you cut your vocabulary off at 50,000, there will still be words not in there, like maybe “abomination,” and you would be forced to replace it with a special token indicating an unknown word, such as “UNK.” There is code [44] that automatically finds good ways to split up words so that the most common words are still represented as whole words and only uncommon ones are split up at their joints.

We’ve talked about how words are represented in the vocabulary table, now let’s take a closer look at how they can be integrated into the current state of the encoder. Each cell (yellow oval in the figures) is actually a neural network with a fixed set of weights (also called parameters), and each cell in the encoder is the same neural network with the same weights. The output of the cell feeds back into the cell, that’s why it’s called recurrent. In these figures, I’ve unrolled it for easy viewing. The simplest version of a RNN cell is shown below, with the last bullet point giving the equation to generate the cell state.

(Note that this is a simple cell to give you a grounded understanding. Most networks use more sophisticated LSTM or GRU cells, as discussed here [22].)

We can see that to update state vector h2, the vector for “ate” and the vector for h1 would be incorporated into the cell. And at h4, the output is the encoding of the sentence. The weights of the vocabulary table and RNN cell start off random and are updated through backpropagation, as discussed later.

Decoders

An RNN decoder unrolls a vector from meaning space into a target sentence. A simple decoder cell is shown below.

The decoder generates a probability distribution over words to write at each time step, in addition to generating the next state. In the simplest case, the output is generated by multiplying the state h by an output matrix of size [300 x 50000]. The product is a vector of size 50,000 that can be normalized into a probability distribution over words (or word segments) in the vocabulary. The network can then choose the word with the highest probability.

How can we use a corpus of paired sentences to train the encoder and decoder? The simplest way is to use a method called teacher forcing. Teacher forcing takes advantage of the fact that we know what the correct word should be because we have the corresponding pairs of sentences. For example, we have a huge list of sentences in English with their Spanish translations. For our current example, at time t=6 we know that the correct word is “comió”. The cost (error) then is the negative log of the probability of the network at step 6 generating the word “comió.” Even if the network erroneously assigns a higher probability to another word, “comió” is still fed to the cell at time step 7 (the “forcing” of teacher forcing.) Now that we have a way of computing the cost (error) for a network, we can do backpropagation to train it. The backpropagation algorithm uses calculus and dynamic programming to update all of the weights in our network. The weight updates go back through the decoder cell activations into the encoder cell activations and even into the vocabulary table. Implementing backpropagation can be tricky. Luckily, tools like TensorFlow and PyTorch can do that for us.

Once the network is trained, we want to use it to generate translations for sentences it has never seen before. This is sometimes called inference. When you are actually using the model to generate text, you don’t know what the right answer is. You could take the greedy approach and always take the most likely word at each time step, but this can lead to suboptimal results. Alternatively, you could generate all sequences of words up to a given length and take the one with the lowest overall cost, but there are too many possibilities to consider. A middle ground is to use a method called beam search. You start generating with the first word, and you keep the 10 most promising sequences of words (beams) around and generate on top of those. The sentences generated by beam search can sometimes lack diversity, and there have been proposals to address this, see [23] and [24].

Attention

Notice in the encoder and decoder that the size of the state vector h stays the same even as more words are processed in the sentence. This feels counterintuitive, and it seems like some information would have to be lost. This is true. One way to get around this limitation is to use attention [25], as shown in the figure below.

At time step 6, when the network is deciding what to write, attention allows it to not just look at state h5 and the last word written but also all of the previous encoded states. It takes a weighted average of their vectors, where the weights are determined by the neural network, and it uses that weighted-average vector as additional information. The way that attention fits into the general seq2seq model is shown below.

Attention works really well. In fact, there is a seq2seq model that uses nothing but attention, see [26] and the code [27].

Applications of Seq2seq Models

Seq2seq models work on all kinds of sentence pairs. For example, you can take a bunch of dialogs and use machine learning to predict what the next statement will be given the last statement. One can mine Twitter for these dialogs, or use movie and TV subtitles [28] or dialogs of people wanting technical support [29]. Another example is text summarization [30], for example, to learn to generate headlines for news articles by using the body of the article as the source and the headline as the target. Seq2seq models can also work for style transfer. In [1], they had Mechanical Turk workers write different styles of sentences for training, upon which they applied the model.

The methodology can even be generalized and applied to image captioning. If we replace the RNN encoder on a sequence of words with a convolutional neural network (CNN) encoder on an image, everything else works the same. This assumes you have a lot of pictures with captions. See [31] and [32] for examples, and see the figure below for a depiction.

The takeaway for seq2seq models is that they work well if you have a lot of parallel sentences and you aren’t too worried about exactly what the model writes. This technology is pretty well established, and there are many implementations, such as the one in TensorFlow [36]. We will next look at the meaning space to see how it can be controlled and enriched, and later we will talk about how to reduce the reliance on parallel sentences.

Meaning Space

The meaning space is a mapping of concepts and ideas that we may want to express to points in a continuous, high-dimensional grid.

The Meaning Space Should be Smooth

We want to be able to navigate this space to generate text with the decoder, so we want the meaning space to be smooth so that points near each other in space have similar meanings, like we see with the taco-related statements in the figure below.

If we don’t care about the representation and are only interested in a basic application such as machine translation, we can simply let the meaning space consist of the last state of the encoder RNN. However, if we want to use the space as a representation for meaning, such as by using it with multiple decoders for multiple languages or styles, we can get more continuity in the space if we use a bias to encourage the neural network to represent the space compactly [37].

We can achieve this bias by adding an organizational constraint in the form of a prior probability distribution. The prior wants to put all points at (0,0,…,0) unless there is a good reason not to, and that “good reason” is the meaning we want to capture. By forcing the system to maintain that balance, we encourage it to organize such that concepts that are near each other in space have similar meanings. In its most basic form, this means that instead of taking the last state as it comes from the RNN, we add two dense neural network layers to the end of the encoder to convert that last state to a mean vector and a covariance vector of a multivariate normal distribution. It’s neat how neural networks and probability theory can interface in this way, and we do this conversion so that it balances the prior and the desire to represent the meaning, as we will see now.

Variational Encoding Ensures Smoothness

This balance of the prior and the meaning is achieved using something called the variational method. In the movie Arrival, variational principles in physics allowed one to know the future. We won’t cover that in this blog post, but in our world, variational methods come from the calculus of variations, which is optimization over a space of functions. Variational inference is coming up with a simpler function that closely approximates the true function.

Recall that we add two dense network layers to convert the last state of the encoder into a mean vector and a variance vector of a multivariate normal distribution. This is the encoding part of what is called a variational encoder. Ideally, we want to encode text x into a meaning space representing the conditional probability distribution p(h|x), but it is hard to represent p(h|x) directly. A variational encoder approximates p(h|x) with a function q(h|x), where q(h|x) can be the multivariate normal distribution we are talking about. Since a multivariate normal distribution can be represented with a mean vector and a covariance vector (for the diagonal), all we have to do is what we said before, add these two layers on the end of our encoder that output the mean vector and covariance vector.

To maximize this approximation so that q(h|x) is as close as possible to the ideal p(h|x), it turns out that we maximize something called the evidence lower bound (ELBO), and we use something called the reparameterization trick to keep it differentiable [38]. Part of the ELBO (shown below) is the prior p(h)=N(0,I), which forces the algorithm to use the space efficiently, which forces it to put semantically similar latent representations together. That’s the KL part in the equation, which stands for Kullback–Leibler divergence.

Without this prior, the algorithm could put h values anywhere in the latent space that it wanted, willy-nilly. Our modified meaning space with the prior represented by the multivariate normal distribution N(0,I) is shown below.

We train the variational encoder as an autoencoder, depicted below. We encode x into the distribution q(h|x) and then sample h from q(h|x) and then pass that sample h to the decoder to regenerate x. Since we are regenerating x, it is called an autoencoder. The prior acts like regularization, so in training we are maximizing the ability of the autoencoder to reconstruct the input x, represented by p(x|h), and we are simultaneously minimizing the difference of the encoded distribution q(h|x) of x in meaning space from the 0 vector of the prior p(h), represented by the KL part. The equation is written using the expected value E.

This variational transformation of the space is not always used because it is hard to train so that the balance is right [37], but it is particularly useful if you want to measure the similarity of two blocks of text or interpolate between sentences [37].

Navigating the Meaning Space

To directly control the text we generate, we need to be able to find desired parts of the meaning space. In the vision domain, Google [41] has shown how you can take an image of a face and add glasses to it or make it older, depicted in the figure below.

In the text domain, [40] provides a way to rewrite sentences to change them according to some function, such as making the sentiment more positive or making a Shakespearean sentence sound more modern. For a sentence s you want to change, the algorithm encodes it using a variational autoencoder to get h. It then looks around the meaning space to find the nearby point h* that maximizes your evaluation function. It then decodes h* to create a modified version of sentence s.

Representing and manipulating ideas in meaning space is fascinating. You could imagine a moviegoer saying, “This movie is good, but make it a little more steampunk.” Hopefully, we will start to encode and decode just about anything. The meaning space could be drawings [53] or even a space of computer programs as shown in [42] and [43]. Search in meaning space could replace searches in much larger, unorganized spaces, and since the space is smooth, we might be able to hill climb to get just about any digital artifact we wanted.

Moving Beyond Pairs of Sentences

The seq2seq model works well when you have a lot of paired training data, such as data paired with itself like in the autoencoder, or English sentences paired with their Spanish translations, or one style paired with the same content in another style, or even images with captions. We now turn our attention to methods that remove this restriction of paired data. We will discuss methods that have the different domains share encoders and bootstrap learning by using autoencoders and round-trip translations. We will also look at methods that train the algorithm to generate desired text using classifiers, including adversarial approaches. And finally, we will discuss using reinforcement learning for training our text generation algorithms.

Bridging Meanings Through a Shared Space

Consider the case where you have pairs of languages or styles but not the pairs you want. For example, you may have a lot of translations from English to Spanish and from English to Tagalog, but you want to translate directly from Spanish to Tagalog. Google faced this scenario with language translation [8]. They had data for a lot of language pairs, but they didn’t have a lot of data for all possible pairs. Their solution is depicted in the figure below.

They use a single encoder and decoder for all languages. Instead of whole words, they use word segments so the encoder can use a single vocabulary table to encode all languages to the same space. They add the target language as the first symbol of the source text, so the decoder knows the language to use. After training many pairs of languages, they found they could translate pairs of languages even for which there was no paired data. The shared encoder and decoder had aligned the meaning space across languages.

What about when you have no language pairs at all? For example, imagine you have text written in two styles, but it is not tied together in pairs. With each language or style, you can learn a language model, as depicted below.

A language model computes the probability distribution over the next word given the previous words. For the sentence, “I went to the w” it computes a probability distribution over all 50,000 words (or word segments) so the probability that word w is “store” is higher than the probability that the word w is “junkyard,” which in turn is higher than “taco.” (Incidentally, language models are the technology behind those funny blog posts of the form: “I forced my computer to watch 10,000 hours of Hee Haw and this was the result.”) In a language model, the sequence of words has to start somewhere. How does it know the probability distribution of the first word? The decoder needs to be associated with a meaning space.

The way to associate text with a meaning space is to train it as an autoencoder. This is the same autoencoder that we encountered previously. The question then becomes how to align up the meaning spaces of the different languages or styles so that one can translate between them. One way to align the meaning spaces is to ensure that the autoencoders of the different languages or styles use the same encoder [7]. This results in the model depicted below.

With this model, we can create synthetic pairs for training. Creating synthetic pairs means that we create the aligned pairs and then train the model as a regular seq2seq model using those synthetic pairs as training data. For example, to map language A to B, we could encode sentences in B and then decode them with the decoder for A, and then we would have pairs of the form (A, B). You start with a sentence in B because we want to do our training in the other direction, starting from sentences in A. To create pairs of the form (B, A), we start with A. In general, this method of creating pairs is called back-translation [13].

Instead of explicitly creating synthetic pairs, we can use the cost of a round-trip translation as a training signal. For example, we can translate a sentence in A to a sentence in B and then translate that sentence in B back into language A, and we can use the cost of translating that sentence in B to language A for training both the shared encoder and the decoder of A. This was done by [6] for text attribute transfer, for example for converting a text from positive to negative sentiment. For instance, for a Yelp review, they automatically converted the positive sentiment phrase “love the southwestern burger” to the negative sentiment phrase “avoid the grease burger.” Notice that the word “burger” is in both, as it should be. Maintaining semantic meaning can sometimes be a challenge with these systems. In their work [6], they add an additional constraint that nouns should be maintained.

The language translation system from Microsoft [9] uses similar principles to achieve human-level performance on translating Chinese to English. They have pairs between these languages, but by using a method called dual learning, they can leverage additional monolingual training data. In dual learning [12], there is a model from language A to language B and a model from language B to language A that are trained together. Dual learning also entails training language models on A and B individually. Then, it seeks to maximize the communication reward, which is the fidelity of a round-trip translation of a sentence in A to language B and back to language A (and B to A and back to B), and also to maximize the language model reward, which for A to B is how likely the sentence in B is in the monolingual language model for B (and likewise for B to A). Instead of using teacher forcing, dual learning uses policy gradient, which will be discussed shortly.

Adversarial Text Generation

We have looked at aligning meaning spaces using autoencoders and round-trip training; another approach to training our algorithm to generate text without aligned pairs of sentences is to use supervised classification on the entire generated sentence. Supervised classification means that we train a learning algorithm to predict labels by feeding it a bunch of records where we already know the label. For example, we could train a classifier to predict who will repay a loan by feeding it records from the past, each labeled with whether the person repaid the loan. We have already used supervised classification at the word level with teacher forcing, and we now look at training a classifier to determine whether our generated text on the whole has a desired property. By doing this, our text generation algorithm can seek to generate text that will be classified as having that desired property.

The method [14] depicted in the figure below runs a classifier (also called a discriminator) on top of generated text. They encode text using a variational autoencoder and add an additional label (feature) in the encoding that is used by the decoder when decoding. Then, the classifier on the other side of the decoder determines whether the decoded text has the proper label. The dotted line indicates learning signal feeding backward from the classifier.

They showed how one could generate negative past-tense sentences such as “the acting was kind of hit or miss” and positive present-tense sentences such as “this is one of the better dance films” by specifying those labels (features) in meaning space. And they showed how one could convert the tense of a particular sentence by changing that label in the meaning space. By doing this, the sentence “this was one of the outstanding thrillers of the last decade” was converted to “this will be one of the great thrillers of the all time.”

We can also consider classifiers (often called discriminators) as part of a general framework called Generative Adversarial Networks (GANs) [11], as depicted below.

GANs consist of a generator and a discriminator (classifier). The generator takes a random seed and from that creates digital artifacts (synthetic data) such as images or sentences. The discriminator then tries to tell the difference between the synthetic data and real examples of that data. The results of the discriminator can serve as the error value for the generator, so that if the discriminator can easily tell the real data from the synthetic data, the error for the generator is high. In continuous domains like images, we can use this error for backpropagation to make the generator better. As the generator gets better, the discriminator has to get better at telling real from generated, so you have a virtuous cycle. In the vision domain, the results are astounding: a generator can create synthetic images of people that look just like real people, but they don’t exist.

In the text domain, we can’t as easily use this error from the discriminator because we have to pick discrete words instead of being able to change continuous pixel values. Picking discrete words in sequence means that the algorithm is not differentiable and so we can’t do backpropagation. There are three methods to deal with this lack of differentiability. The first method is to not pick specific words at each time step and instead to use the weighted average of the word vectors, where the weights come from the output word probability distribution of each cell of the decoder. This is what was done in the classifier method [14] we saw previously. The second method is to use reinforcement learning, which we will describe in the next section. The third method is to apply the discriminator to the continuous meaning space or to the continuous space of hidden states generated by the decoder.

One such method of applying a discriminator to the continuous hidden states of a decoder is called professor forcing [15]. In professor forcing, the discriminator of the GAN focuses on telling the difference between hidden states when the decoder is running in teacher forcing mode and running in inference mode. The original goal of this work was not to transfer style or language, but rather to make the decoder better at decoding when it gets off track and starts generating sequences that are significantly different from the training data. In [5], they use professor forcing for style transfer. They encode a sentence in a source style and they use a discriminator to ensure the hidden states of the decoding of that sentence are close to the hidden states of a normal sentence in the target style.

Learning Through Reinforcement Learning

Reinforcement learning (RL) is an alternative to teacher forcing for training our text generation algorithm. Reinforcement learning is a gradual stamping in of behavior that comes from receiving rewards and punishments (negative rewards). RL was studied in animals by Thorndike [47] as far back as 1898, and it later became the study of behaviorism in psychology [46]. It was then formulated into artificial intelligence; see the leading book by Sutton and Barto [16].

When using reinforcement learning for neural text generation, the actions are writing words and the states are the words the algorithm has already written. Choosing the best word to write is hard because there are as many actions as there are words in your vocabulary, usually around 50,000. A kind of reinforcement learning that works well in domains with large action spaces is called policy gradient (see [17] and [18]). A policy specifies what action to take (in our case what word to write) for each state. Policy gradient methods learn a policy directly on top of a neural network; basically, they learn which action to take as a classification problem [18]. This means that reinforcement learning starts to look more like supervised learning, where the labels are actions. Instead of doing gradient descent on the error function as in supervised learning, you are doing gradient ascent on the expected reward. Each action you take is the “correct” label for the current state, and the loss for that correct label is scaled by whether that action turned out to be good or bad, meaning, roughly, whether you got a positive or negative reward.

When using GANs, this reward will come from the discriminator, and so the discriminator determines the policy gradient. For example, this discriminator could try to tell the difference between dialog responses created by humans and ones created by the generator, and the generator would receive a positive reward when it can fool the discriminator, as was done in [3].

One difficulty with these high dimensional spaces is that it is hard to learn what action to take now (what word to write) when you don’t get a reward until much later when the discriminator is run and the sentence is scored. To address this problem, we can use either Monte Carlo search or the actor-critic method.

In Monte Carlo search, the algorithm does a search forward from the current state to the end to estimate how good different actions are at the current time step. For example, in the figure below, if we were to write “comió” and follow our current policy after that, how good would the final reward be?

We can use the estimate created by searching (simulating) forward until the end as the current reward now, as was done in [58]. That “how good” is the estimate we use to determine whether writing “comió” should be the action this network takes in the future when in a similar state. This approach is also used in game-playing AI, for example with the recent success in the game Go [20].

In actor-critic methods, instead of searching forward to determine how good each action might be in the current state, or how good the current state is, you have another neural network that learns to estimate the value of each state or state-and-action pair directly. This neural network is called the critic [17]. An example GAN for text that uses this approach is MaskGAN [21] (code available here [35]).

Currently, GANs have some drawbacks for text generation. The first is mode collapse: once the generator figures out how to fool the discriminator, it may keep generating the same thing over and over again. The second drawback is that GANs are hard to train because it is difficult to keep the generator and discriminator in balance. The third drawback is that training is slow. If your vocabulary size is 50,000, the algorithm has to try 50,000 random actions in many states. To reduce this search, researchers often pre-train text GANs with teacher forcing so that the generator chooses sequences of words during training that are at least reasonable. For a survey of reinforcement learning for generating text, see [2], and see [4] for an example of semi-recent work. Also, see the code implementations at [34].

The application of deep learning to reinforcement learning is still a work in progress [45]. Before deep learning, one significant challenge to employing reinforcement learning was abstracting the state. The world is complicated, and the only way to learn is to know when states are similar to states the algorithm has seen before. In 2013, [55] made a significant breakthrough using deep learning to play Atari with Deep Q-Networks (DQN). The input was the direct video from the game, and they used deep learning to solve the state abstraction problem. That algorithm couldn’t think very far ahead, but there has been significant recent progress in playing complicated video games, such as Dota 2 [57], capture the flag [56], and Doom [54]. Text generation has a larger set of possible actions than most video games because it requires choosing among which of 50,000 words to write at each time step, but these games are complex as well, and we can expect that many of these advancements will begin making their way into text generation.

Conclusion

When should you use neural text generation? If you can enumerate all of the patterns for the things you want the computer to understand and say, use context-free grammars for understanding [48] and templates for language generation [10]. You could, for example, build a chatbot where you use machine learning to map the input text to different possible topics where you have a context-free grammar for each topic. Another example is writing stories for Little League baseball games [51]. If you can come up with a couple dozen ways that a baseball game can go, pitchers duel, slugfest, come-from-behind victory, you can take the stats from the game and use it mad-libs style to automatically write a story.

If, on the other hand, you can’t enumerate all of the things you want the computer to understand and say, but you have a lot of data, neural text generation might be the way to go. Machine translation, of course, is an ideal example. Neural text generation is also useful if you want to convert between styles. What neural text generation lacks is exactness. Most of the time, we want our machines to say exactly what we mean them to say. We don’t want our website to tell people that all printing presses are 50% off when we intended the sale to be for inkjet printers.

Adding symbolic exactness back into the meaning space is a pressing problem. There has been some work in this direction, for example in generated summaries [52]. As we saw in [6], the scoring of style transfer can include whether nouns were preserved during translation of one style to another. Another option is pointer networks, which can choose to copy specific information from the source text during the decoding [49]. In addition to lacking exactness, neural text generation doesn’t yet work well on long text, but the attention-based method [26] seems promising in this regard.

Machines will be illiterate for a long time, but as algorithms get better at controlling and navigating the meaning space, neural text generation has the potential to be transformative. Few things are more enjoyable than reading a story by your favorite author, and if we could have all of our information given to us in a style tailored to our tastes, the world could be enchanting.

Originally published at http://www.deumbra.com/single-post/2018/07/22/Generating-Natural-Language-Text-with-Neural-Networks

References

[1] Seq2seq style transfer: Dear Sir or Madam, May I introduce the YAFC Corpus: Corpus, Benchmarks and Metrics for Formality Style Transfer, https://arxiv.org/pdf/1803.06535.pdf

[2] Neural Text Generation: Past, Present and Beyond, https://arxiv.org/pdf/1803.07133.pdf

[3] Adversarial Learning for Neural Dialogue Generation, https://arxiv.org/pdf/1701.06547.pdf

[4] Actor-Critic based Training Framework for Abstractive Summarization, https://arxiv.org/pdf/1803.11070.pdf

[5] Style Transfer from Non-Parallel Text by Cross-Alignment, https://arxiv.org/pdf/1705.09655.pdf

[6] Improved Neural Text Attribute Transfer with Non-parallel Data, https://arxiv.org/pdf/1711.09395.pdf

[7] Unsupervised Neural Machine Translation, https://arxiv.org/pdf/1710.11041.pdf

[8] Zero-Shot Translation with Google’s Multilingual Neural Machine Translation System, https://research.googleblog.com/2016/11/zero-shot-translation-with-googles.html

[9] Achieving Human Parity on Automatic Chinese to English News Translation, https://arxiv.org/pdf/1803.05567.pdf

[10] Summer School on Natural Language Generation, Summarisation, and Dialogue Systems, http://nlgsummer.github.io/index.html

[11] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio, Generative Adversarial Networks, https://arxiv.org/abs/1406.2661.

[12] Dual Learning for Machine Translation, https://papers.nips.cc/paper/6469-dual-learning-for-machine-translation.pdf

[13] Improving Neural Machine Translation Models with Monolingual Data, http://www.aclweb.org/anthology/P16-1009

[14] Toward Controllable Text Generation, https://arxiv.org/pdf/1703.00955.pdf

[15] Professor Forcing: A New Algorithm for Training Recurrent Networks, https://arxiv.org/pdf/1610.09038.pdf

[16] Reinforcement Learning: An Introduction. See new book online http://incompleteideas.net/book/bookdraft2018jan1.pdf

[17] D. Silver, http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching_files/pg.pdf

[18] Deep Reinforcement Learning: Pong from Pixels, http://karpathy.github.io/2016/05/31/rl/

[19] SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient, https://arxiv.org/pdf/1609.05473v3.pdf

[20] Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm, https://arxiv.org/pdf/1712.01815.pdf

[21] MaskGAN: Better Text Generation via Filling in the Blank, https://arxiv.org/pdf/1801.07736.pdf

[22] Understanding LSTM Networks, http://colah.github.io/posts/2015-08-Understanding-LSTMs/

[23] Mutual Information and Diverse Decoding Improve Neural Machine Translation, https://arxiv.org/pdf/1601.00372v2.pdf

[24] Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models, https://arxiv.org/pdf/1610.02424v1.pdf

[25] Neural Machine Translation by Jointly Learning to Align and Translate, https://arxiv.org/abs/1409.0473

[26] Attention Is All You Need, https://arxiv.org/abs/1706.03762

[27] Tensor2tensor code https://github.com/tensorflow/tensor2tensor

[28] OpenSubtitles, http://opus.lingfil.uu.se/OpenSubtitles.php

[29] The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems, https://arxiv.org/abs/1506.08909

[30] Deep Learning for Text Summarization, https://memkite.com/deeplearningkit/2016/04/23/deep-learning-for-text-summarization/

[31] Deep Visual-Semantic Alignments for Generating Image Descriptions, http://cs.stanford.edu/people/karpathy/deepimagesent/

[32] A picture is worth a thousand (coherent) words: building a natural description of images, http://googleresearch.blogspot.com/2014/11/a-picture-is-worth-thousand-coherent.html

[33] TensorFlow word2vec Tutorial https://www.tensorflow.org/tutorials/word2vec

[34] Texygen code https://github.com/geek-ai/Texygen

[35] MaskGAN code https://github.com/tensorflow/models/tree/master/research/maskgan

[36] Neural Machine Translation (seq2seq) Tutorial https://github.com/tensorflow/nmt

[37] Generating Sentences from a Continuous Space, https://arxiv.org/abs/1511.06349

[38] Auto-Encoding Variational Bayes, https://arxiv.org/pdf/1312.6114.pdf

[39] Variational Autoencoder in TensorFlow https://jmetzen.github.io/2015-11-27/vae.html

[40] Sequence to Better Sequence: Continuous Revision of Combinatorial Structures, http://www.mit.edu/~jonasm/info/Seq2betterSeq.pdf

[41] Latent Constraints: Learning to Generate Conditionally from Unconditional Generative Models, https://arxiv.org/pdf/1711.05772.pdf

[42] Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision, https://arxiv.org/pdf/1611.00020.pdf

[43] Learning to Organize Knowledge with N-Gram Machines, https://arxiv.org/pdf/1711.06744.pdf

[44] Subword Neural Machine Translation, https://github.com/rsennrich/subword-nmt

[45] Deep Reinforcement Learning Doesn’t Work Yet, https://www.alexirpan.com/2018/02/14/rl-hard.html

[46] Skinner, B. F. Science and human behavior, Simon and Schuster, 1953

[47] Thorndike, E., Animal intelligence: an experimental study of the associative processes in animals, Columbia University, 1898

[48] Chatbots: Theory and Practice, https://medium.com/intuitionmachine/chatbots-theory-and-practice-3274f7e6d648

[49] Pointing the Unknown Words, http://www.aclweb.org/anthology/P16-1014

[50] A Random Walk Through EMNLP 2017 http://approximatelycorrect.com/2017/09/26/a-random-walk-through-emnlp-2017/

[51] Parents Pay for Little League Sports Recaps Written by Computers, https://abcnews.go.com/Business/parents-pay-league-sports-recaps-written-computers/story?id=34743643

[52] Faithful to the Original: Fact Aware Neural Abstractive Summarization https://arxiv.org/pdf/1711.04434.pdf

[53] A Neural Representation of Sketch Drawings, https://arxiv.org/pdf/1704.03477.pdf

[54] World Models, https://worldmodels.github.io/

[55] Playing Atari with Deep Reinforcement Learning, https://arxiv.org/pdf/1312.5602.pdf

[56] Capture the Flag: the emergence of complex cooperative agents, https://deepmind.com/blog/capture-the-flag/

[57] OpenAI Five, https://blog.openai.com/openai-five/

[58] SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient, https://arxiv.org/pdf/1609.05473v3.pdf

More where this came from

This story is published in Noteworthy, where thousands come every day to learn about the people & ideas shaping the products we love.

Follow our publication to see more product & design stories featured by the Journal team.

I love when my old questions get answered. Twenty years ago, I lay awake wondering why life exists. Now, after reading books like [Schrödinger, 1944], [Tegmark, 2017], and [Carroll, 2016], I think I understand. Life is a non-magic, physical process. Life is like little dust devils; little whirlwinds of matter that get going but that by chance have a structure that allows them to keep going longer than they have a right to. Dust devils aren’t alive, of course, because they don’t reproduce, but they are a good reminder of the kind of process that life is.

Life exists because it exists. That sounds like a Zen koan, but what I mean is that once a process by chance gets started that has the property that it keeps going, it keeps going precisely because it has that property. And if that process by chance also has the property that it can reproduce itself, so that the property that kept it going is passed to its copies, there is even more of it. These copies aren’t perfect, of course. There are little imperfections in the copies, and some of these imperfections lead to variations that can keep going and reproduce even better. This leads to even more of those variations and more variations stemming from those, and the result is bacteria, trees, humans, spiders, and chickens.

What does this have to do with intelligence? Some variations of the dust devil happen to have the property that they move toward things that keep them going and avoid things that extinguish them; we call that intelligence. In some cases, more intelligence allows the dust devil to keep going for longer and in more diverse environments, so it can make more dust devils and variations of dust devils.

We human dust devils are now trying to build intelligence artificially. We are building it from the top down in an engineering effort to achieve human goals. By contrast, nature built intelligence from the bottom up, with no goal, as variation accumulated upon variation. This distinction is probably why artificial intelligence (AI) practitioners are generally less fearful of AI than other people. We see it as engineering, like building better toasters or dishwashers. People looking at AI from the outside don’t see it as engineering; they see artifacts acting like they are alive and doing things that humans do, like driving cars and identifying cats in pictures. The natural question to them is, “How are we going to control this life when it gets smarter than us?”

We build our AI from the top down because we need it to be useful now and because we don’t understand the fundamentals of natural intelligence. The result of this top-down approach is a superficial embodiment. The intelligences we build are not embodied, or even when they are embodied, we are trying to jump directly to building intelligences that can handle concepts relevant to human culture without the underlying plumbing. The result is that we get programs that grade student essays without being able to read, and we have robots that build cars without knowing what a road is.

By contrast, natural intelligence started in organisms with simple morphology, and these simple body shapes provided a limited perceptual input space and a limited action space, leading to simple decisions from simple inputs. Through evolution, the morphology and intelligence of organisms developed together, which led to a deeply embodied intelligence. For instance, we humans are so efficient at walking because we take advantage of the dynamics in our legs and treat them as pendulums [Clark, 2008]. And our sensory-motor routines enable us to do things like make sandwiches while minimizing memory requirements by allowing us to use glances to get the right information from the right part of the environment at the right time [Ballard, 2015]. We use our deep embodiment for both action and internal simulation. When we imagine kicking an object, we do that imagining with the part of our brain that actually does the kicking [Bergen, 2012]. Many different capabilities work together in an ad-hoc way, leading to a society of mind that governs our behavior [Minsky, 1988].

Because we want our AI to be immediately useful, we build robots with complex morphology that can pick up objects, identify cats in images, and classify movie reviews based on sentiment. We’ve had a lot of success lately with deep learning and even genetic algorithms, but when it comes to building in common sense, what we call artificial general intelligence (AGI), we’ve hit a wall. This lack of progress stems from the difficulty of building on top of lower-level abilities.

Human general intelligence is built on lower-level, pre-lingual concepts [Feldman, 2006], [Mandler, 2004]. To understand language and sentences like “Our startup has taken us on a long journey” we map them to physical experience, such as actually walking down a long path [Lakoff & Johnson, 1980]. To build real understanding, we may need to build abstract concepts on top of actual code that performs the underlying physical analog. We have self-driving cars that can perform navigation; maybe we need to build higher-level abstractions directly on top of that navigation code so that a robot can understand how starting a company could be a “journey.”

While we are purposefully trying to build AI from the top down, our economy and computer infrastructure may be allowing AI to emerge from the bottom up. Our computer programs themselves undergo a kind of evolution. Computer programs that work well are run a lot, like dust devils that keep going. And as they stay around, they accumulate changes from Jira tickets and variations from forks on GitHub. The code starts out designed top-down with beautiful abstractions, but over time, the mutations build up and it becomes less understandable but more fit for its environment.

We are also slowly merging our programs together so that they are approaching a society of mind. Tasks that used to be necessary for getting things done, such as sorting or running a CNN classifier, can now be simply called, letting the programmer focus on higher-level problems. We are slowly building more complex systems through makefiles, microservices, APIs, GitHub repos, and Apache Maven. These little societies of mind are now so complicated that code is often bundled once in a Docker container so that each user doesn’t have to fight through the version incompatibilities between the different variations of the components. And because these functionalities are built on top of simpler functionalities, we are getting a kind of deep embodiment within the cyber world. A Twitter bot is built on code that goes all the way down to sending packets over TCP/IP.

Will our intelligent machines ever be living entities like you and me, with a rich understanding of the world and even consciousness? We don’t have a clue on how to make a Docker container conscious, but our computer technology may be moving along a bottom-up path analogous to the one that led to our consciousness. Still, there are at least two mysteries. We have only seen one variation of dust devils develop rich language and the complexity of culture that it enables, and we have no idea how we did it or even how it all works. The second mystery of consciousness is even deeper. We don’t know if it is necessary, or a mirage, or based on mathematical principles. Once we solve those two mysteries, we’ll have a better idea of the likelihood of super-intelligent machines, unless they solve them first.

References

[Ballard, 2015] Ballard, D. Brain Computation as Hierarchical Abstraction.

[Bergen, 2012] Bergen, B. K. Louder than Words.

[Carroll, 2016] Carroll, S. The Big Picture.

[Clark, 2008] Clark, A. Supersizing the Mind.

[Feldman, 2006] Feldman, J. From Molecule to Metaphor.

[Johnson, 1987] Johnson, M. The Body in the Mind.

[Lakoff & Johnson, 1980] Lakoff, G. & Johnson, M. Metaphors We Live By.

[Mandler, 2004] Mandler, J. The Foundations of Mind.

[Minsky, 1988] Minsky, M. The Society of Mind.

[Schrödinger, 1944] Schrödinger, E. What is Life?

[Tegmark, 2017] Tegmark, M. Life 3.0.

Why is there life? And what does it have to do with AI? was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Theory: What is conversation?

Conversation begins with shared reference. We point to objects in the world so that we and our conversational partner know that we are talking about the same things.

Children use pointing to communicate desire for objects and shared interaction before they can speak. As we learn language, words then point to shared ideas in our minds (Gärdenfors, 2014).

What things words point to becomes a shared convention over time through language games, as described by Wittgenstein in Philosophical Investigations (1958). Language games are interactions that shed light on a particular aspect of language. Through cooperation in shared action, such as building a hut, we settle on labels for objects. When we ask for a “beam” we nod approvingly when someone finally brings us the correct item.

Of course, language is more than labels. Our brains map the concepts behind these community conventions to personal sensations and actions. When someone says “beam,” we associate that with our experience with beams, and since people generally have similar experiences, we are able to understand each other. When someone says they hurt their back carrying a beam, we understand because we too have carried heavy things. The meaning is grounded in that experience (see Benjamin Bergen, Steven Pinker, Mark Johnson, Jerome Feldman, and Murray Shanahan).

When our experiences and language games don’t sufficiently overlap, we must negotiate meaning in the course of a conversation. Consider the image of the discourse pyramid below. When everything is understood, we are at the bottom level and can just give and receive instructions for cooperation. When something is not understood, there is a break and we have to do a coordination of inner worlds, and if that isn’t understood we have to do a coordination of meaning. Once we have acknowledgment of understanding at a particular level, we can go back down to the previous level. For example, we ask a guy to get us some fish. He doesn’t understand how, so we have to coordinate our inner worlds and explain that he can catch fish with a fishing pole. When he doesn’t even understand what a fishing pole is, we have to coordinate meanings and explain that a fishing pole is a tool that consists of a stick, string, and a hook.

In addition to the meanings of words and sentences, conversation itself has its own rules. Consider Grice’s (1975, 1978) conversational maxims:

Maxim of Quantity: Say only what is not implied. Yes: “Bring me the block.” No: “Bring me the block by transporting it to my location.”

Maxim of Quality: Say only things that are true. Yes: “I hate carrying blocks.” No: “I love carrying blocks, especially when they are covered in fire ants.”

Maxim of Relevance: Say only things that matter. Yes: “Bring me the block.” No: “Bring me the block and birds sing.”

Maxim of Manner: Speak in a way that can be easily understood. Yes: “Bring me the block.” No: “Use personal physical force to levitate the block and transport it to me.”

Breaking these rules is a way to communicate more than the meaning of the words. Wikipedia has a nice summary here. When we break a maxim, it is assumed that it is for some purpose. For instance, when I say “I love carrying blocks, especially when they are covered in fire ants,” I am breaking the maxim of quality to use sarcasm to communicate that I don’t like carrying these blocks. As another example, if someone asks me how good Bob was as an employee and I respond that he had nice hair, I am communicating an idea by breaking the maxim of relevance.

Systems for converting speech-to-text have recently gotten pretty good, for example here and here, and we will largely ignore that aspect in this post, but sometimes meaning isn’t even in the text but in how the words are spoken (called prosody). Consider the following three examples taken from Voice User Interface Design by Giangola and Balogh (2004). The italicized words are emphasized. Based on the emphasis, each example means something different (stated afterwards).

1. You know. I don’t. [So don’t ask me.]
2. You know. I don’t. [As a matter of fact, I really don’t.]
3. You know I don’t. [You know that I don’t.]

Conversation is really hard. Yuval Noah Harari argues in his book Homo Deus that our ability to cooperate is what has allowed humans to take over the planet. Being able to converse with machines will allow us to further cooperate with this new kind of intelligence we have created. Let’s now turn our attention to the state of the art.

Practice: The current state of chatbots

We will look at the three classes of chatbots in turn: purposeless mimicry agents, intention-based agents, and conversational agents.

Purposeless mimicry agents

Purposeless mimicry agents give the appearance of conversation without understanding what is being said. We’ve all heard of ELIZA. ELIZA consisted of simple substitution rules to mimic a psychologist from the 1960s. This is the psychology that emerged after behaviorism and is often characterized as having the therapist repeat back the words of the patient. If the patient said, “My mother wants me to buy a bazooka.” The therapist might respond, “Tell me why your mother wants you to buy a bazooka.” ELIZA could have this type of conversation. You can find a Python implementation of ELIZA here.

Modern mimicry agents use deep learning to learn from example conversations. They train on a bunch of dialogs and learn to generate the next statement given the last statement. There are lots of dialog datasets out there. You can use examples of dialog from movie and TV subtitles, such as OpenSubtitles. One can also use the Ubuntu Dialog Corpus, which is dialogs of people wanting technical support. Or one can mine Twitter and look for replies to tweets using the API.

The deep learning method used for this parroting is usually sequence-to-sequence models. A sequence-to-sequence model consists of an encoder and a decoder. The encoder converts a sequence of tokens, such as a sentence consisting of word tokens, into a single vector. The decoder begins with this vector, and it keeps generating tokens until it generates a special stop symbol. Note that the lengths of the source and target sequences don’t need to be the same. Both the encoding and decoding are done using recurrent neural networks (RNNs).

The initial big application for sequence-to-sequence models was language translation (Cho et al., 2014). For example, one would have a set of source sentences in English and their corresponding translations into Spanish. The model would then learn to encode the source sentence in English into a vector and then to decode that vector into the corresponding sentence in Spanish. To generate conversations, sequence-to-sequence models treat a statement as the source language, such as “How are you?” and they treat the response as the target language, such as “I am great!” An example is shown below.

The problem with sequence-to-sequence models is that they are devoid of meaning. They can work for just about any kind of problem that can be cast as translating one sequence into another, but they don’t make use of any special machinery or knowledge for language understanding. Another problem with using sequence-to-sequence models for chatbots (and using deep learning in general for chatbots) is that they are insensitive to specifics. If I ask it to buy me 14 chickens, it doesn’t treat the number 14 as particularly special, and it might be just as likely to buy me 140.

The most recent sequence-to-sequence chatbots use generative adversarial networks (GANs). GANs have a discriminator that tries to tell if the generated response was from a real conversation or generated by the model. GANs are all the rage in image processing, but they are still a work in progress for language.

Intention-based agents

Intention-based agents understand language as commands, and they use that understanding to perform tasks in the world. Examples of intention-based agents include Amazon’s Alexa, Google Home, Apple’s Siri, and Microsoft’s Cortana. Understanding what we say as a command language requires solving two problems:

1. Identifying what the user wants the machine to do (the “intent”).
2. Figuring out the details of the intent so the machine can take action.

For example, if we ask our assistant to play Steve Winwood, the assistant first needs to understand that we want it to play music (the intent), and then it must understand that, in particular, we want to hear Steve Winwood (the details).

Consider the example below about determining intent. When the user asks about chickens, the machine has to figure out which of its four capabilities matches the intent of the person.

The assistant can determine the intent using either keywords or text-based classification. To use keywords, you simply associate words and phrases with intents. To do text-based classification, you label a bunch of statements with the correct intents and then train a classifier over them. You can train a classifier using a bag-of-words representation with the Python library scikit-learn, as described here. If you have lots of labeled data, another option for learning intents is to use deep learning with a convolutional neural network (CNN) in TensorFlow. An implementation can be found here.

Once the agent has determined the intent, in our example it is to order groceries, it needs to convert the statement details into a machine-readable form, such as a Python dictionary, as shown below.

Having the command in this kind of dictionary form is called a frame and slot semantics, where the frame is the topic and the slots are the individual features and values. Once we have the machine readable form, the computer can do with it what it wants, just like any other instruction. This is natural language understanding.

One way to do natural language understanding is to use context-free grammars (compositional semantics). Context-free grammars consist of production rules that have a single nonterminal on the left and a string on the right side that can consist of terminals and non-terminals. To each of these production rules, we add a semantic attachment. An example set of rules is shown below.

Nonterminals begin with ‘$’. The first rule says that an order consists of an intent to purchase and an item and amount of that item. The semantics of that rule are combined via a dictionary union of the semantics of $Purchase (s[0]) and the semantics of $ItemAmount (s[1]). The next rule says that an intent to purchase is determined by the function is_purchase, as shown below. (This, of course, is a compact way to describe a set of rules. I.e., one with “get me” on the right-hand side, and another with “buy”, and another with “grab me some.”)

The next rule says that $ItemAmount consists of an $Amount and an $Item, which are defined in the following two rules. Those two rules make use of the functions get_item and get_number, shown in example form below.

Each of those rules has an associated semantics at the end. We use parsing to apply those production rules to text to get semantics. Parsing is done using bottom-up dynamic programming. Dynamic programming consists of solving subproblems and then reusing those solutions over and over again. The parsing process builds up the parse by looking at pairs of words, and for each pair, it loops over all n possible words in between. Because the parsing algorithm deals with pairs of sub-parses, the production rules above each have either a single terminal or two non-terminals on the right-hand side. If your grammar is not like this (but still has a single nonterminal on the left-hand side of each production rule) this is not a problem; you can always convert a context-free grammar to this form (called Chomsky Normal Form).

The figure below shows the order the parsing algorithm follows as it considers pairs of words: from ‘Get’ to ‘Get’, then from ‘me’ to ‘me’, then from ‘Get’ to ‘me’, then from ‘fourteen’ to ‘fourteen’, and so on.

The next two figures show the production rules invoked and the semantics at each time point, respectively. For example, at box 8 we are looking at the words from ‘fourteen’ to ‘chickens.’ This corresponds to the rule with $ItemAmount on the left-hand side, which comes from box 4 (the parse on ‘fourteen’) and box 7 (the parse on ‘chickens’). Box 9 brings together box 8 and box 3 to invoke the rule with $Order on the left-hand side.

Of course, the parser also tries many other combinations that don’t fit any of the rules. In general, if there are n words in the sentence and R production rules in the grammar, it tries n*n boxes and looks at n boxes in between for each, trying to apply each of R rules. In total, the running time is on the order of n*n*n*R. Often, there can be many valid parses, and a scoring algorithm can be used to choose which one to use.

There is code available in Python for creating grammars and running the parser as part of the SippyCup library.

Conversational agents

Conversational agents expand on intention-based agents to have multi-turn conversations. To do this, they must keep track of the state of the conversation and know when the person wants to talk about something else.

Conversational agents aren’t yet common but imagine an agent like the one I described here. The agent would stay with a person for their whole life. It would start out as a companion when the person was a child, where it would live as a cell-phone app on the parent’s device. It would have a cartoon face, and it would learn about the child and teach her about the world. For example, if the agent knew that the child loved giraffes, it could use that to teach her math. “If you have 7 giraffes and bought 3 more, how many would you have?” It could also help with cognitive biases and over-discounting the future. “If you eat your chocolate bear today, how are you going to feel tomorrow when you no longer have it?”

The developers of the app would be working furiously behind the scenes, and the agent would become more sophisticated as the child grew older. When the child became an adult, the agent would become her operating system. It could provide turn-by-turn directions in life, such as guiding her through how to fix a sprinkler system. The agent could then serve as her assistant when she got old. Imagine that she is standing in the kitchen and can’t remember how to make coffee. The app could use cameras in her home and could guide her through the process, allowing her to live independently longer.

Conversational agents such as this one need a dialog manager to handle long conversations. A dialog manager must take the human through all of the things the chatbot wants to talk about. For example, it could first want to teach the child math skills, and then it could want to talk about the solar system. The dialog manager also needs to recognize when she wants to talk about something else and queue the current topics for later.

RavenClaw from CMU is probably the best-known dialog manager (see Bohus and Rudnicky, 2009). RavenClaw consists of dialog agents, which are little programs organized hierarchically that correspond to different bits of conversation. For example, there could be a general dialog agent about cooking that wants to discuss multiple things related to preparing food.

RavenClaw has a dialog stack and an expectation agenda. The dialog stack is a stack of dialog agents to keep track of all the things the chatbot wants to talk about. The expectation agenda is a data structure to keep track of what the chatbot expects to hear. For example, if the child says “yes” she probably isn’t answering a question from thirty minutes ago like Rain Man; the answer probably corresponds to the last question asked. But if she says, “I remember! Saturn is the one with rings!” she may be talking about a previous topic, and the expectation manager needs to match the utterance with all of the expectations to find the right one.

During dialog execution, RavenClaw alternates between an execution phase and an input phase. During the execution phase, it invokes the top dialog agent on the dialog stack and lets it talk. It also sets up what the machine expects to hear in the expectation agenda. During the input phase, RavenClaw processes what the person said and updates its knowledge, which it uses to determine which dialog agent can respond during the execution phase.

Consider the example below with a dialog stack and an expectation agenda.

1. Chatbot just finished asking, “What is 4 + 5?”
2. Child says: “Do you like Mr. Fluffles?”
3. Chatbot responds: “Very nice. Is he your favorite?”

The chatbot is expecting to hear an answer to the math problem, which is why “9” is at the top of the expectation agenda. The child instead says something completely different, so the algorithm searches the expectation agenda for what she might be talking about. It finds that she is talking about toys. The chatbot then could move talking about toys to the top of the dialog stack, so it could generate an appropriate response, such as “Very nice. Is he your favorite?” The chatbot would also modify the expectation agenda to match that it expects to hear an answer to this new question.

RavenClaw isn’t based on machine learning, but we can alternatively use a learning-based approach to build conversational agents. We can treat creating a conversational agent as a reinforcement learning (RL) problem (Scheffler and Young, 2002). Reinforcement learning problems take place within a Markov decision process (MDP) consisting of a set of states, actions, and a reward function that provides a reward for being in a state s and taking an action a.

In the context of chatbots, states are things such as what the bot knows (questions it has answered), the last thing the bot said, and the last thing the user said. Actions are making particular statements, and reward comes from meeting a goal state, such as the child giving the correct answer to a math problem or, in a different domain, successfully completing a travel reservation.

We can use reinforcement learning to learn a policy that gives the best action a for being in state s. As one can imagine, learning a policy requires a lot of training, and it is difficult to pay people to sit and talk with the agent to generate enough experience, so researchers train their agents with simulated experience (Scheffler and Young, 2002).

There is an additional problem with using learning for chatbots; it is hard to know exactly what state the agent is in because of errors in speech-to-text or errors in understanding. For example, does the child want to talk about her toys, or is she telling the agent that it should be fluffier? This uncertainty calls for a Partially Observable Markov Decision Process (POMDP) (Young et al., 2013). In a POMDP the agent doesn’t know what state it is in and instead works with a distribution over the states it could be in. These types of agents are still experimental.

Conclusion: How well does practice match theory?

Let’s compare the current state of chatbots with our theory. We started with shared reference, which is making sure we are talking about the same thing. Shared reference is pretty easy to implement as long as we limit the discussion to a set of known things. Likewise, the shared conventions for word meanings can be programmed into intention-based agents or conversational agents. We then hit a wall.

The meanings that we code in are not mapped to a grounded sense of sensation or action in the agent. When we refer to an object, the agent has never held the object or used it like we would in the real world. This means that its understanding will be limited. I mentioned previously the idea of someone hurting their back carrying a heavy beam. A computer’s possible understanding of this scenario is limited to logical inference, which leads to two disadvantages. The first is that logical inference is surprisingly hard. Inference itself is well understood, but it is hard to set up the rules to make all relevant inferences. The second disadvantage is that the agent’s understanding will be at a low resolution. A computer can only understand variables populated with values — it can’t feel the pain. I wrote a post about how we could tackle this problem by having agents simulate experience in our world, but we have a long way to in this area.

More bad news is that the meanings that chatbots have are largely fixed, and so we currently can’t negotiate meanings with them in the discourse pyramid. There has been research into pragmatics, such as Grice’s maxims, but I don’t know of any work ready to be implemented on a general-purpose agent. The same is true for prosody. We have a long way to go.

In summary, the current state of the art is that we can build knowledge into chatbots that they can use to cooperate with us on tasks requiring minimal understanding. Building better chatbots will necessitate knowledge engineering and research into how agents can better follow and adapt to the subtleties of meaning and conversation.

If you prefer video, I cover much of the same content here.

Chatbots: Theory and Practice was originally published in Intuition Machine on Medium, where people are continuing the conversation by highlighting and responding to this story.

In my last blog post, I discussed how computers could understand natural language by simulating our world. Such an understanding would enable us to build a bot that could be our lifelong teacher, guide, and advocate. Simulating our world is a huge task, so I propose that we start by building the bot for young children, since their experience is a small but complete sliver of our own. The proposed bot could interact with children using voice and vision. The bot would initially live in the cell phone of the child’s parents, much like Siri does.

The bot would get to know the child individually, and they would both develop together. This bot would stay with the child for her whole life. As the child grew in sophistication, the bot developers would keep up, making the bot increasingly sophisticated. When the child was young, the bot would teach her about the world. As she grew older, the bot would live inside all of her devices and become her operating system and user interface to the world. If she experienced cognitive decline in old age, this same bot would guide her through daily living and help her stay independent longer.

A Teacher For The Young

The initial purpose of the bot would be to entertain and teach children. It would integrate learning into daily conversation, and it would have an animated cartoon face and would see using a phone’s camera, so the child could show it her world. The bot would learn about the child and get to know her favorite toys.

The bot would be an effective teacher because it would know which concepts the child had already mastered, allowing it to explain new concepts to her in terms that she already understood. The bot would also know what topics the child was interested in, so it could teach new concepts in an engaging way. Instead of just asking the child to figure out the answer to 7+3, the bot could ask how many giraffes she would have if she had seven in her backyard and bought three more.

In addition to systematically introducing concepts, the bot could teach organically and expand the child’s curiosity through conversations [1]. It could say things like, “Yeah, fire trucks are red and shiny. Did you know that they go by burning dead dinosaurs?” The bot could teach estimation and the meaning of large numbers with conversations such as, “I like ducks too. How many ducks do you think there are in the world?” Through these interactions, the bot would also gently expand the vocabulary of the child.

The bot would teach critical thinking using fun brain teasers, such as this one described in Kahneman [2], “A bat and ball together cost $1.10. The bat costs $1.00 more than the ball. How much does the ball cost?” (It’s not $0.10.) The bot could also point out biases in thinking and logical errors. It could nudge and teach the child about opportunity costs and delayed gratification. “If you eat your chocolate bear today, how are you going to feel tomorrow when you no longer have it?” The bot could also teach the child to see things from another’s point of view. For instance, a conversation fragment might be, “You told me you only had one chocolate bear, how could I know that you also had a chocolate turtle?”

The value of this kind of teaching would be even greater for children who lack access to good educational opportunities. The bot could be provided at a subsidized cost to these children because although building the bot would be expensive, once it was built the marginal cost of giving it to another child would be small.

The bot would initially live on the parent’s smartphone, but the intelligence and memory of the bot could live in the cloud, so the child could take it with her to other domains and devices and even into virtual reality.

A Guide For The Adult

As the child grew older, the bot would become her operating system and interface to the world. In addition to handling normal things like scheduling and playing music, the bot would provide a seamless voice interface into internet-connected devices. The bot would also continue to teach us, even as adults, although the learning would be more directed by the user. We’ve probably all had the experience of looking up a concept outside of our area of expertise on Wikipedia and not understanding the first sentence. The bot would know what we know, and so it could explain new concepts in a way that we could understand.

The bot could also migrate to a Google Glass-like wearable device and help us do things we didn’t know how to do, like fix a washing machine or restore an old car. For me, it would know that my mechanical knowledge was limited to loosening and tightening bolts using the mnemonic “lefty loosey righty tighty,” and so its instructions would describe actions at that level.

Having a bot that knows us would also alleviate one of the most dreaded tasks of life — filling out forms. I hate filling out forms. If the bot had our permission, it could fill in the fields automatically with less hassle than current existing browser extensions. The concept of filling out forms extends to complicated domains such as medical diagnosis and law. It’s difficult to find what we need online in these domains because one must wade through a sea of irrelevant information. For this reason, any current automated algorithm either gets bogged down asking irrelevant questions or gives generic answers based on prior probabilities. A bot that knew us could guide us through.

An Advocate For The Old

After the bot had taught us as children and guided us as adults, it could help us with daily living if we faced cognitive decline in old age. It could advocate for us as it guided us through the government and healthcare bureaucracies. The bot could even help us do things we once did with ease. Imagine that your bot could watch you with cameras in your house. If one morning it saw you standing in the kitchen looking confused, it could know that you make coffee at that time and could talk you through the steps. The bot would help us live independently longer.

Technical Challenges

Building this bot entails technical, business, and social challenges. The most obvious technical challenge is natural language understanding. My last blog post discussed a method for understanding natural language using 3D physics simulations, but this set of ideas needs to be implemented, and challenges will surely creep up. For example, pragmatics will have to be accounted for because what people say and what people mean are often two different things. These difficulties will be compounded by errors converting speech to text.

The next looming technical challenge is the large amount of world knowledge that needs to be built in to create effective simulations. This difficulty is the primary motivation for beginning with young children, who live in worlds that are less complicated than ours.

The bot also needs to know how to teach. There exist automated teaching tools that tailor lessons to individual progress, and these ideas have to be integrated into the bot along with the different concepts it will teach. The bot also must be able to represent the concepts that the user knows so that it can teach new concepts using those known ones.

In addition to teaching, the bot needs to keep track of the lifelong conversation. It needs to understand when a user doesn’t respond to the question asked, but instead wants to talk about something else. And the bot needs to be able to understand what that something else is. This burden of holding a logical conversation is also partially alleviated by initially starting with young children. Children are more forgiving of weird transitions between topics.

The challenge for the bot developers will be to build sophistication into the bot as fast as their initial customers progress through childhood. Once the children reach adulthood, it gets harder because the bot will need to know complicated stuff, like how to actually restore an old car. The bot will at least need to be able to find appropriate instructions on the internet and translate them into the owner’s level of competence. And, of course, the bot will know everything about an individual, so all of this will need to be done while preserving personal information. It will be hard. Whole areas of research are dedicated to addressing many of these issues, so the bot developers will need to make pragmatic tradeoffs.

Business Challenges

The business challenges relate to funding a project of such technical complexity. This project is a good fit for a startup because it starts small, focusing on children, but appealing to children presents its own challenges. Parents can be reluctant to pay for apps, and it doesn’t seem feasible or desirable to monetize via advertising. In general, the bot needs to always have the best interest of the owner as its primary goal, and advertising can be antithetical to that. This bot would undoubtedly create a lot of value. The business challenge is how to capture enough value along the way so that the full bot can be built.

Social Challenges

The social challenges stem from the fact that the human brain was wired through evolution to have relationships with living beings. Social processing makes up a disproportionally large part of our intelligence relative to other animals, and a lifelong relationship with a bot would be a new kind of experience in a significant aspect of our lives, which could have unpredictable consequences. For example, would children still create imaginary friends? Or, what would happen if the bot maker went out of business and the bot went away? The social challenge is to design the bot to minimize the probability of unforeseen adverse consequences.

Given the social challenges, one could ask whether such a bot should be built. Creating something new that reaches so deeply into our psyche seems risky, but we would miss out on the benefits if we didn’t build it. When it comes to potential dangers of technology, I lean toward cautious advancement. Fire is dangerous, but we certainly wouldn’t go back and tell our ancestors not to master it. This endeavor is of course more modest, but as the virtual world becomes increasingly integrated with our physical one, such an interface, teacher, guide, and advocate could bring substantial benefit.

Conclusion

Imagine how cool it would be to have Bing Bong from your childhood serve as your OS. As a child, the bot would be like an engaging and extra-personalized Khan Academy. In adulthood, the bot would give us turn-by-turn directions through many of the tedious and complicated parts of our lives, and in late adulthood, the bot could help us maintain our independence. All the while, the bot would provide an external memory of your life with the fidelity of a computer, because it would be there with you the whole way.

I’m speaking at O’Reilly Artificial Intelligence in New York on June 29, so come check it out!

Notes

1. See The Curiosity Cycle: Preparing Your Child for the Ongoing Technological Explosion (written by me) for how discussions can instill curiosity.
2. D. Kahneman. Thinking, Fast and Slow. Farrar, Straus and Giroux, 2011.

You and Your Bot: A New Kind of Lifelong Relationship was originally published in Chatbots Magazine on Medium, where people are continuing the conversation by highlighting and responding to this story.