The End of the Language Model: The Day AI Learned to See
How We Got Here: Episode 21 of ~26. We are walking the AI timeline one contribution at a time.
I took a screenshot of a stack trace, the list of calls a program leaves behind when something fails, and dragged it into the chat. Didn’t type a word of the error. It found the line that mattered.
I can’t remember the last time I typed an error message out by hand, letter by letter. With a text-only assistant, you had to copy those lines into the chat, cleaning them up if needed. Now a picture of the error can be enough to start debugging it together.
A text-only language model takes in text and generates more text. The words in a screenshot are still language; getting at them through pixels is the extra capability. We call a model multimodal when it handles several kinds of input, such as text, images, and sound. You can show the assistant something instead of translating it into a typed description first.
Where we left the language thread
Episode 20 walked through reasoning models, the shift from a model that blurts out an answer to one that thinks first. Systems like OpenAI’s o1 and DeepSeek’s R1 learned to spend extra effort at the moment you ask, working through a problem step by step before committing to a reply. That was a new place to spend compute, not just at training time but at the moment of the question.
The language thread of this series has mostly stayed with text. A transformer is a neural network architecture that works out relationships between pieces of a sequence. We met it reading words and writing words. GPT predicted the next chunk of text, over and over, at scale. Retrieval-augmented generation, or RAG, fetched more text for it to read before answering. And reasoning models, the ones we just talked about, worked through a problem before giving you their reply.
The vision story comes back
One line was vision: machines that see. Episode 4 was LeCun’s convolutional network reading handwritten ZIP codes off real mail, the idea that you could bake a truth about images (a pattern means the same thing wherever it sits) directly into the architecture of a network. Episode 7 was AlexNet, the year that idea broke the dam and machines could suddenly recognize what was in a photograph.
The other line was language: machines that handle sequences of words. That’s Episodes 9 and 10, attention and then the transformer, the 2017 architecture that let every word in an input sentence draw information from the others. Everything since (GPT, ChatGPT, Claude, RAG, agents, reasoning) flowed down that second line.
The tasks rewarded different things. An image-recognition system could identify what was in a photograph and have little to offer when you wanted to ask a question about it. To get from the picture to a conversation, visual information had to become something a language system could use.
CLIP approached that problem by asking which words belong with which pictures.
Teaching pictures and words to share a map
OpenAI published CLIP, short for Contrastive Language-Image Pre-training, in 2021. The lead author is Alec Radford, the same Radford who led GPT-1 and GPT-2 back in Episode 11. Here, the training problem was matching pictures with text.
CLIP uses embeddings, the numerical representations we met in Episode 16.
An embedding represents something as a list of numbers. You can think of those numbers as coordinates on a map, with related things placed near each other. “King” and “queen” might sit close together, while “bicycle” sits farther away. That gives a model a way to compare things numerically, even when they aren’t written with the same words.
CLIP’s question was simple to ask and hard to pull off. What if you built a space like that, but let both pictures and words live in it? One shared map, where an image and the sentence that describes it land in the same neighborhood.
Training it meant collecting around 400 million image-and-text pairs from the internet. Imagine a photo of a golden retriever paired with “a photo of a dog,” or a picture of the Eiffel Tower paired with “the Eiffel Tower at night.” Those are examples of the relationship the training is trying to learn: which text belongs with which picture.
CLIP has two encoders, components that turn an input into an embedding. One reads images; the other reads text. During training, the model compares the images and text in a batch. It learns to bring matching pairs closer together and separate mismatches. The dog photo and its description should score as a better match than the dog photo and the Eiffel Tower caption.
Once trained, it can compare a new photo with text descriptions, or use a phrase to find related images. You put a photograph on one side of the comparison and a sentence on the other, and the system has a numerical way to judge whether they belong together.
What a matching model can do
Suppose we made a tiny training batch with three pairs: the dog photo and its caption, the Eiffel Tower and its caption, and a terminal screenshot paired with a description of an error. This is an illustration of the matching task, not a claim about three records in the actual dataset.
There are three images and three captions. Compare every image with every caption and you get nine possible pairings. Three are the pairs we supplied. The other six combine an image with a caption from somewhere else in the batch. Training rewards the model for distinguishing the supplied pairs from those alternatives.
The dog image has to score better against its own description than against the terminal description. From the other direction, the words about the terminal have to pick out the screenshot rather than the tower. Both encoders have work to do. A useful image representation has to preserve something the text can refer to, and the text representation has to express something the image can support.
After training, we can supply descriptions the model wasn’t explicitly trained to choose between for this particular photograph. Perhaps we want to compare “a dog,” “a cat,” and “a bicycle.” The system encodes those phrases and compares them with the image. The highest-scoring candidate gives us a classification without training a new classifier for that exact list of labels. This is called zero-shot classification: applying the trained model to a new classification task without examples labeled for that task.
Notice who’s supplying the possible answers here. We are. A matching score can help select a description or retrieve a picture; it hasn’t written an explanation of what caused a program to fail. That is why the shared-map idea matters without being the whole assistant. It gives us a relationship between two kinds of input that later systems can build on.
It also makes the choice of descriptions part of the question. If every candidate describes the wrong thing, the best-scoring one is still only the best of those candidates. “Closest” doesn’t mean “verified.”
How the transformer ate pixels too
In Episode 4, I called the central idea structure is knowledge. LeCun’s convolutional network was built around useful assumptions about images: nearby pixels belong together, and a pattern worth finding in one place is worth finding elsewhere. The architecture did some of the work before training even began. Convolutional networks later became the dominant approach to image recognition.
Then, in 2020, Alexey Dosovitskiy and a team at Google published a paper with a title that tells you the whole story: “An Image is Worth 16x16 Words.” The model is called the Vision Transformer (ViT). What they did is almost cheeky in its simplicity.
The chunks a transformer processes are called tokens. For text, they’re often words or pieces of words. Self-attention lets each chunk draw information from the others to work out what matters. The architecture came from a translation paper, but nothing in that description requires the chunks to be words.
The Vision Transformer team chopped an image into a grid of square patches, 16 by 16 pixels each in the version named in the title. An image 224 pixels wide and 224 pixels tall gives you 14 patches along each side, or 196 patches altogether. Each patch becomes a numerical representation in a sequence. Self-attention then lets the model relate information from one patch to information in the others.
What the patches still have to learn
Cutting up a picture creates another problem. The contents of a patch and its location are different pieces of information. A patch containing the head of an arrow means something different depending on where it sits relative to the rest of the diagram.
ViT adds position embeddings, learned numerical information about where each patch belongs in the sequence. The model receives information about the patch and its position together. A sequence can therefore retain information about an image’s arrangement even though the input is no longer presented as one two-dimensional grid.
A diagram with two boxes and an arrow contains more than a list of objects. Which box the arrow starts from matters. A caption that lists the visible objects could leave that out, while a representation that retains their arrangement gives the model something to work with when you ask about the direction of the connection.
That is the reason for spending time on the patches. We haven’t translated the image into a sentence saying what is in it and thrown the picture away. We’ve turned visual material into numerical inputs that can retain information beyond a short description.
I like the patches-as-words analogy because it makes that jump easier to follow. A patch isn’t literally a word. It’s a piece of a picture put into a form the transformer can work with.
Get Cordero Core’s stories in your inbox
Join Medium for free to get updates from this writer.
It worked. Given enough data, the Vision Transformer matched and then beat the convolutional networks that had owned image recognition since AlexNet. The thing built for language turned out to be general enough to handle pixels, once you were willing to hand it the picture in pieces.
The “enough data” part belongs in that sentence. In the paper, convolutional networks’ built-in image assumptions helped when training data was more limited. ViT’s results improved with larger-scale pretraining. Some knowledge that a convolutional architecture starts with has to be learned from examples instead. The result was a change in how that knowledge could be acquired, with consequences for the amount of training required.
Once the inputs are prepared and the model is trained for the task, attention can work with other kinds of sequences too, including image patches, audio, and video.
From pixels to a written answer
A text-generating language model works with numerical representations of tokens and predicts what comes next. For images to inform that answer, the system needs numerical representations of visual information too. The patch-and-encode idea gives you a way to prepare that information for a transformer.
For the error image in the opening, both the image and the question, “why is this throwing?”, have to be represented. A multimodal system can use information from both when producing its answer. You might ask about the failing call, or ask it to explain the error more generally. The picture stays the same; your question changes what you want from it.
How the image and text components work together varies between systems; audio has its own encoding too.
One published design makes the connection concrete. In the 2023 paper Visual Instruction Tuning, Haotian Liu and colleagues introduced LLaVA, the Large Language and Vision Assistant. It connects a CLIP vision encoder to Vicuna, a language model, through a learned projection: a transformation that puts the visual features into a form the language model can use.
The training has two stages. First, the vision encoder and language model stay fixed while the projection is trained to align them. Then the projection and language model are tuned with visual instructions and answers, while the vision encoder remains fixed. Matching image features to language and learning to answer requests about images are separate parts of the work.
In that design, the visual representations and the question participate in generating the answer. There is a specific connection to train, rather than a requirement that every component be identical from the start. LLaVA is one documented architecture, useful here because we can inspect how its pieces meet.
OpenAI made GPT-4V, GPT-4 with vision, available in 2023. Google introduced Gemini that year as multimodal from the ground up. Anthropic’s Claude 3 family followed with image-reading capabilities in 2024.
One screenshot, several questions
A terminal screenshot can support several requests: copying the error message, identifying the call that failed, or explaining what might have caused it. Those are different requests about the same image, and they require different things from the answer.
Copying the message is a reading task. The output should reproduce the visible characters. Identifying the failing call also requires interpreting the arrangement of the stack trace: which lines name calls, which line reports the error, and how those pieces relate. Explaining the cause adds knowledge about the programming language and whatever context the screenshot provides.
An error about a missing value makes the distinction easier to see. Recognizing the error correctly doesn’t establish why the value was absent. The code that supplied it could be in another file. The relevant input could have arrived long before the lines shown on screen. The image may contain no evidence about either.
I would want the answer to keep those boundaries visible. “The screenshot says this” and “this may be why” give me different things to check. An assistant that rolls both into one confident explanation makes the next step harder, even when its reading of the image was good.
You can change the question without changing the screenshot. Ask for the exact text first, then ask which part supports the diagnosis. If the transcription is wrong, the proposed fix is already resting on a bad reading. If the transcription is right but the explanation goes beyond the visible evidence, you know to bring in the code or inputs it hasn’t seen.
The work before the question
With a text-only assistant, a question about a whiteboard sketch or a chart started with your description. You had to explain the arrows, axes, and trend, deciding which details to carry over.
Now you can give it the diagram and ask about an arrow. You can leave the chart in the paper and ask about a trend. That doesn’t guarantee the model will read either correctly, but you can at least start with the material you’re looking at, rather than your attempt to turn all of it into prose.
That matters because your world was always multimodal. A question about a diagram arrives with relationships that are already visible on the page. You used to have to choose which of them to describe before an assistant could help you.
Now the world gets to stay the way it already was.
Try the same distinction with a chart from a paper. Suppose two lines move in opposite directions and you want to know whether the result supports the claim in the caption. The question includes more than recognizing that one line goes up. You need the axis labels, units, legend, and the relationship between the plotted quantity and what the paper says it measured.
An assistant might describe the broad trend correctly while reading the wrong series from the legend. It could read the series correctly and miss that the vertical axis is logarithmic, so equal distances on the page represent multiplication rather than equal-sized increases.
This doesn’t make the model’s account a substitute for the paper. A figure may show an association while the caption or methods section explains the conditions under which it was measured. Supplying that surrounding text gives the assistant a different evidence base from supplying the picture alone. You can now ask about the relation between the two, including whether the explanation actually fits the plotted result.
Where the seeing breaks down
A model that describes an image fluently can still be wrong about it. The fluency makes the error harder to catch, especially when the description gets most of the picture right.
OpenAI’s 2023 GPT-4V system card records a specific example in Figure 4. In a scientific diagram, two nearby pieces of text were combined: a label for a blood-forming stem cell and wording about the cell’s division. The report describes the model merging them into unrelated terms. The words were present, but their relationship on the page was read incorrectly.
That is a different failure from simply not knowing the topic. The assistant can produce an explanation using a term assembled from the wrong parts of the image. Asking a follow-up about that term could carry the reading error into the next answer. The first thing to check would be whether the combined label exists in the figure at all.
The wider limitations include counting objects, reading tiny text, and locating an exact connection between lines or labels. A model might get the broad account of a chart right while misreading a specific number on its axis.
This is the hallucination problem from earlier in the series: a model produces something plausible that turns out to be false. Grounding the answer means making sure its words answer to the pixels in the image you actually supplied.
In the stack-trace example, reading most of the error correctly would be little comfort if the model invented the line you were supposed to fix. I’d still want to compare that line with the terminal.
Image preparation can also affect what the system has available to read. OpenAI’s vision guide lists small text, graphs, precise spatial localization, incorrect descriptions, and approximate counts among its limitations. It also notes that images may be resized before analysis. A readable image on your own screen doesn’t establish what detail survives the system’s processing.
For a dense figure, I would keep the full image for context and provide an enlarged view of the part the question depends on. A crop can make a small label easier to inspect, but it can also leave out the legend or the neighboring line that explains it. The point is to preserve the relevant relationship while making the detail readable, rather than assume that a larger-looking upload guarantees a better answer.
What’s actually different now
Language remains a large part of what these tools do, but it no longer describes everything you can hand them.
We spent weeks building up to the transformer and next-word prediction, and those ideas still matter. Then you drag a screenshot into the same chat box and keep working. There’s no obvious point in that interaction where you’re asked to change what you call the tool.
I don’t think that name survives the decade, though I don’t have a replacement I like yet. For now, I notice the mismatch when I write it down. Less often when I’m trying to fix a bug.
Next Tuesday: DeepSeek and the efficiency turn. We’ve spent much of this series watching capability grow with bigger training runs. Next week we look at what DeepSeek’s results did to the cost assumptions, and what those comparisons actually include.
If you’ve used an assistant to read a screenshot or diagram, where did it help, and what did you still have to check yourself? I’d be interested in the detail it missed as much as the one it caught.
You can follow along on Medium or subscribe to the Substack. Both run the full series, and the Wednesday and Thursday pieces go deeper on whatever I’m building or watching that week.
Cordero is a senior research software engineer at the Scientific Software Engineering Center at the University of Washington’s eScience Institute. If this landed, forward it to the engineer who pasted a stack-trace screenshot this week and never stopped to ask how the “language” model read it.
