Last fall, Fei-Fei Li published a piece called “From Words to Worlds: Spatial Intelligence is AI’s Next Frontier.” Her core argument: large language models are “wordsmiths in the dark — eloquent but inexperienced, knowledgeable but ungrounded.” They’re incredibly good at manipulating language. They just don’t understand the world they’re talking about.

I’ve been building infrastructure for AI agents, and this line of thinking kept pulling me in a specific direction. If LLMs don’t understand the physical world, what happens when you ask them to understand how people experience the physical world? Not what people would say about an experience — but what actually happens in their heads during one?

That question led me down a rabbit hole over a weekend. Here’s what I found.

The rabbit hole

Meta released a model called TRIBE v2 on March 26, 2026. It’s a brain encoding model — you feed it a video, audio, or text, and it predicts how the average human brain would respond to that stimulus, second by second, across ~20,000 points on the cortical surface. It was trained on fMRI data from 720+ people watching movies and listening to stories.

📓 Full experiment notebook

This is not a chatbot. You can’t ask it questions. It’s a one-way function: stimulus goes in, predicted brain activation comes out. It combines V-JEPA2 (video), Wav2Vec-BERT (audio), and LLaMA 3.2 (text) into a single Transformer architecture that maps multimodal representations onto the brain.

I had a simple question: does this model actually find the “right” moments in a video?

If you show it a famous ad — one where critics, audiences, and neuroscience researchers all agree on which second is the emotional climax — does the brain model’s peak activation line up with that second?

The setup

I picked three iconic ads. Each has a well-documented “peak moment” — a specific second that critics, directors, and researchers identified as the climax before I ran any experiments. I committed to these ground truth moments based on published sources, then ran TRIBE v2 to see whether its predictions aligned.

This ordering matters. I didn’t run the model and then go looking for sources that matched the peaks. I found the documented moments first, wrote them down, and then checked whether the model agreed.

The other critical detail: TRIBE v2 never saw these ads during training. I checked the source code. It was trained on Friends episodes, feature films (Bourne Supremacy, Wolf of Wall Street, Hidden Figures), short naturalistic clips, and spoken narratives. Zero commercials. Every prediction here is out-of-distribution — the model is generalizing from movies and sitcoms to advertisements it has never encountered.

I also ran each ad through Claude Sonnet — 100 independent API calls at temperature 1.0, each a fresh stateless request with no memory of prior answers. Like asking 100 different people the same question. Each trial received extracted video frames (sampled every 3 seconds) plus a text description of what happens in the ad. The LLM had to pick which second the average viewer would find most exciting.

Experiment 1: Cadbury Gorilla

The ad: A gorilla sits behind a drum kit for 60 seconds while Phil Collins’ “In the Air Tonight” plays. The gorilla just sits there, eyes closed, breathing. Then the drum fill hits and it erupts into the most cathartic drumming performance in advertising history. No chocolate is shown.

The known peak moment: ~second 60, when the drum fill begins.

This isn’t my opinion. It’s one of the most thoroughly documented moments in advertising history:

• Director Juan Cabral told The Guardian (2016) that he conceived the entire ad around the drum fill, saying: “There’s 60 seconds where nothing happens — just a gorilla and Phil Collins.” He told Marketing Week that the idea came from saying “he’s animalistic, he’s like a gorilla drumming” and the image stuck.
• Neuro-Insight tested the ad with SST brain imaging and ranked it in the top 1% of all ads ever tested for long-term memory encoding, with the peak response at the drum fill.
• When the Cannes Lions jury screened the ad at the 2008 press conference, jury president Craig Davis noted the room broke into air drumming (Ad Age, 2008).
• The IPA Effectiveness case study documented an ROI of £4.88 per £1 spent, and Millward Brown reportedly had to recalibrate its engagement scale for this ad.

📓 Brain activation timeline — Gorilla

TRIBE v2 found it. The top three peak activation seconds were 82, 83, and 59. Second 59 — one second before the documented drum fill — is the #3 peak. The higher activations at 82–83 correspond to the drumming crescendo near the end. The overall shape tells a clean story: seconds 0–15 are flat (gorilla sitting still, quiet vocals), activation spikes at 18–27 (music building, drum kit reveal), drops through 35–50 (quieter verse), then climbs sharply from 53 to peak at 59 as the drum fill arrives.

Claude Sonnet (100 trials): 100/100 recognized the ad. The mode was second 63, chosen by 68% of trials. Average peak: 66.5 seconds. Top 3 picks: second 63 (68%), second 75 (23%), second 72 (8%).

📓 LLM trials — Gorilla

This is where it gets interesting. The brain model peaks at second 59. The LLM peaks at second 63. A 4-second gap. What’s happening?

The frames were sampled every 3 seconds. So the LLM saw frames at seconds 0, 3, 6, … 57, 60, 63, 66… The frame at second 63 shows the gorilla in the middle of active drumming — arms up, sticks hitting drums, clearly the visual climax. That’s what the LLM sees and correctly identifies as exciting.

But the sound of drumming actually begins earlier — as early as second 59. The brain model caught the audio onset of the drum fill. The LLM caught the visual evidence of drumming. The brain hears the drums before you see the gorilla in full swing, and that 4-second gap is the difference between a system that can hear and one that can’t.

Yes, 100/100 Claude instances recognized the ad — it’s almost certainly seen commentary about the Cadbury Gorilla in its training data. It “knows” the drum fill is the peak. But even with that knowledge, it’s anchored to the frame where drumming is visually obvious (63), not the second when the audio actually hits (59). It’s picking the right general area for the right reasons, but its precision is limited by what it can see.

Now mute the ad. What you see is a gorilla sitting motionless for a minute, then moving its arms. Without the music — without 60 seconds of vocal buildup, without the anticipation of the most famous drum fill in pop music — it’s barely an ad. The entire emotional architecture is constructed by audio.

Experiment 2: Apple 1984

The ad: A dystopian world. Rows of bald people in uniforms sit before a giant screen where Big Brother preaches conformity. A woman in bright athletic clothing runs in carrying a sledgehammer, chased by guards. She hurls it at the screen. It explodes. “On January 24th, Apple Computer will introduce Macintosh.”

The known peak moment: ~second 41–46, when the woman throws a hammer at the screen.

Again, comprehensively documented:

• Wikipedia’s entry on the ad describes the moment precisely: “The runner hurls the hammer towards it, right at the moment Big Brother announces ‘we shall prevail!’ In a burst of light and smoke, the screen is destroyed.”
• When the ad aired during Super Bowl XVIII, CBS announcers Pat Summerall and John Madden responded on air with “Wow, what was that?” (Britannica).
• The Smithsonian’s Lemelson Center (2014) called it the moment that “cemented the Super Bowl as each year’s blockbuster moment for advertisers.”
• CNN (2024) described it as: “A young woman throws a hammer at the screen, exploding it into a white light.”

📓 Brain activation timeline — Apple 1984

TRIBE v2 nailed it. Peak #1 at second 43, #2 at second 42, #3 at second 41. All three top peaks land in a single 3-second window — precisely on the hammer throw. Activation builds steadily through the 30s as the runner is chased, peaks sharply at 41–45, then drops off a cliff to near-zero by second 46 (quiet aftermath, scrolling text).

Claude Sonnet (100 trials): 100/100 recognized the ad. The mode was second 42, chosen by 98% of trials. Average peak: 41.8 seconds.

📓 LLM trials — Apple 1984

Both the brain model and the LLM converge on the same second. 98% agreement from the LLM is remarkable — essentially unanimous.

This is the control experiment. The hammer throw is a massively visual event — a bright explosion, a screen shattering into white light. You can identify this as the peak moment from a still frame alone. When the peak is visually obvious, the LLM gets it right. Both systems agree. That’s the expected result and it confirms the LLM works when visuals carry the signal.

The gap between brain model and LLM only appears when audio carries the emotion. Which brings us to experiment 3.

Experiment 3: Apple AirPods “Quiet the Noise”

The ad: A woman walks through a chaotic city wearing AirPods. Everything around her — people, cars, construction equipment — floats upward away from her, a visual metaphor for noise cancellation. Around second 35, she taps the AirPods stem and everything crashes back to earth. Later, around second 51, she puts the AirPods back in, noise cancellation re-engages, and the objects float back up.

The known peak moment: ~second 35, the crash-down.

This is the moment every piece of trade coverage identifies as the climax:

• AppleInsider (2023): “Active Noise Cancellation is disabled and all of the previously floating objects crash back down to earth.”
• 9to5Mac (2023): “she takes out one of the AirPods and sounds of the city come rushing back.”
• DesignRush: “The ad climaxes when the protagonist taps her AirPods to switch from Active Noise Cancellation to Transparency mode, causing the floating objects to crash back to the ground.”

📓 Brain activation timeline — AirPods

TRIBE v2 result: Top three peaks at seconds 23, 21, and 24. The crash-down at ~35 shows as a visible secondary bump — elevated above surrounding seconds, but not the highest peak. The brain model’s top pick is the moment of maximum visual spectacle, when the floating effect is at its most dramatic.

Claude Sonnet (100 trials): 100/100 recognized the ad. Primary peak: second 15, chosen by 91% of trials. Secondary peak (required to be 5+ seconds away from the first): second 51, chosen by 88% of trials.

📓 LLM trials — AirPods

This is the most interesting experiment, for three reasons.

First: The LLM’s primary pick — second 15, at 91% — is the moment when floating objects are visually spectacular. It’s a pretty frame. Cars and people suspended in midair. But this isn’t the moment people talk about when they discuss this ad. It didn’t generate press coverage. It didn’t stick. It’s visually striking but narratively meaningless. The brain model also picks up on this visual intensity (its top peaks are at 21–24, the same neighborhood). Both systems are drawn to raw visual spectacle.

Second: The LLM’s runner-up pick — second 51, at 88% — is genuinely puzzling. Second 51 is when items float back up (noise cancellation re-engages). But here’s the thing: some of the LLM instances that picked second 51 described it as the moment “everything crashes back down to earth.” That’s not what happens at second 51 — that’s what happens at second 35. This raises a real possibility that the LLM is pulling from its training data knowledge of the ad rather than reasoning purely from the frames. It “knows” the crash-down is supposed to be the peak, and it’s attributing that description to a visually ambiguous frame.

Third: Neither the brain model nor the LLM puts the documented crash-down at second 35 as their top pick. The brain model shows it as a secondary bump. The LLM apparently mapped it to the wrong second. The moment that stuck with people — that got written up by AppleInsider, 9to5Mac, and DesignRush — isn’t the moment of maximum visual complexity. It’s a narrative beat. It’s the moment the ad’s meaning clicks. It sticks not because it’s the prettiest frame, but because the sharp change in audio pulls you in. That audio transition — from serene calm to full urban cacophony in a split second — is what makes the crash-down immersive and memorable. And it’s exactly the kind of thing neither a brain model looking at visual peak activation nor an LLM looking at still frames can fully capture.

What the brain model gets right on its own

Before getting into implications, I want to step back and acknowledge what’s already here — without even comparing to LLMs.

This model was trained on people watching Friends and feature films. It has never seen a commercial. It has never encountered any of these specific videos. And yet:

• It finds the Cadbury Gorilla drum fill within 1 second of the moment the director built the entire ad around
• It finds the Apple 1984 hammer throw at the exact 3-second window documented by Wikipedia, Britannica, CNN, and the Smithsonian
• It detects the AirPods crash-down as a visible peak, while correctly identifying that the moment of maximum visual complexity happens earlier in the ad

The peaks aren’t random. They’re responding to real perceptual features — a sudden onset of loud drumming after 60 seconds of quiet vocals, an explosion of light after a slow build of tension, a burst of color and floating objects at maximum density. These are the things that would genuinely drive brain activation in a real viewer.

If you squint at this practically, there’s something useful. Advertisers are constantly making decisions about where to place a logo reveal, when to show the product, how to time a brand message against the emotional arc of a spot. If you could computationally pinpoint the seconds of maximum perceptual engagement — without recruiting participants for an fMRI study — that’s genuinely valuable. From what I can see, TRIBE v2 does this for videos it has never seen before, out of distribution, in a few minutes on a Colab GPU.

What the comparison shows

Putting the three experiments side by side:

When the peak moment is a visual event (Apple 1984 — screen exploding), both systems agree and both get it right. 98% LLM consensus on the exact same second the brain model peaks on.

When the peak moment is an audio event (Cadbury Gorilla — drum fill), the brain model hears the audio onset 4 seconds before the LLM can see the visual evidence. The LLM gets the general area right (likely because it recognizes the ad from training data), but its precision is limited to what’s visible in a still frame.

When the peak moment is a narrative event tied to an audio transition (AirPods — crash-down), neither system puts it as #1. Both are drawn to moments of higher raw visual complexity instead. The moments that people remember and talk about — the moments that stick — aren’t always the moments of maximum sensory intensity. They’re the moments that mean something, and meaning is hard to measure in brain activation or frame analysis alone.

Why this matters

There’s a growing industry of synthetic audience tools — AI systems that simulate how people react to ads. Every one of them is built on LLMs. Every one of them processes video as extracted frames. They’re analyzing muted television.

Mute the Cadbury Gorilla ad. What you see is a gorilla sitting motionless for a minute, then moving its arms. Without the music — without 60 seconds of vocal buildup, without the anticipation of the most famous drum fill in pop music — it’s barely an ad. The entire emotional architecture is constructed by audio.

Brain encoding models like TRIBE v2 can hear. They process audio through Wav2Vec-BERT the same way they process video through V-JEPA2. And they appear to use it — the Gorilla’s peak at second 59 is driven by the onset of the drum fill, something no amount of visual analysis could find.

But the AirPods experiment also shows the limits. The moments that become culturally significant — that generate press and conversation — aren’t always the moments of maximum perceptual activation. The crash-down at second 35 is memorable because of an abrupt audio shift, a narrative twist, and the surprise of context breaking. Neither the brain model nor the LLM fully captured why that specific moment is the one people talk about. Perception and meaning are different things.

I don’t think brain models are going to replace focus groups anytime soon. But I think the people building synthetic audience tools should probably make sure their AI can hear the music. And I think the people building brain models should think about why the moments we remember aren’t always the moments our brains work hardest.

The caveats

Recognition contamination. Claude recognized all three ads — 100/100 on every experiment. It’s almost certainly seen these ads discussed in its training data. This means its peak picks may be influenced by what people have written about these ads, not just what it sees in the frames. The 100-trial design with temperature 1.0 partially controls for this, but it’s a real confound. The Gorilla result is still meaningful because even with recognition, the LLM is anchored to the visually obvious frame (63) rather than the audio onset (59).

Sample size of three. This is a weekend project, not a published study. Three ads is enough to demonstrate the concept, not enough to make statistical claims.

Frame sampling granularity. I sampled video frames every 3 seconds for the LLM trials, meaning the available seconds were 0, 3, 6, … 57, 60, 63, 66, and so on. I started with this resolution to see if anything interesting emerged before committing to finer granularity. As it turns out, the 3-second spacing actually strengthens the Gorilla finding rather than weakening it: the LLM had the frame at second 60 available — the exact second the drum fill begins — and chose second 63 instead. It looked at the moment the drums start and decided the frame where the gorilla is already in full swing was more exciting. That’s a real signal about what the LLM is optimizing for: peak visual intensity of an action already in progress, not the onset of an audio event. Finer sampling at one frame per second would allow more precise comparison, and is a natural next step — but the current resolution was enough to reveal the gap.

Perceptual peaks aren’t “best moments.” As the AirPods experiment showed, the moment the brain works hardest isn’t necessarily the moment the ad means the most. TRIBE v2 measures perceptual intensity, not narrative significance. Both matter. They’re different.

Try it yourself

The full experiment notebook is public: 📓 Google Colab Notebook

Everything runs on Google Colab with an A100 GPU. You need TRIBE v2 (pip installable from Meta’s GitHub), an Anthropic API key, and a HuggingFace account with LLaMA 3.2 access.

TRIBE v2 has been public for 3 days. This stuff is brand new. If you’re curious about world models, multimodal perception, or what AI actually “understands” when it watches a video — it’s worth your afternoon.

I’m a grad student doing research in ML systems and got curious about where world models fit in. This started as a weekend rabbit hole and turned into something I thought was worth writing up. If you want to run a different ad through the same pipeline, the notebook is linked above — I’d love to hear what you find. Feel free to say hi at ifesionubogu@gmail.com.

Thanks to Ivan Bogatyy for reviewing a draft of this post.