The People the Model Didn’t Expect
Learning from synthetic data without losing sight of the people we need to understand
This essay was developed through original research and analysis with AI assistance in the research, drafting and editorial process.
Some people struggle to fall asleep every day. Others almost never do. In a Pew Research Center survey, 36% of participants said it happened to them on some days. Pew asked an AI model to answer as if it were each of those people. Among the synthetic responses, the figure was 77%. The answers were much more concentrated in a single category. [1]
An occasional bad night and spending most nights staring at the ceiling are very different experiences. If our representation blurs that difference, it may also make it harder to see whose experience we need to understand better.
Pew published the experiment on September 30, 2026. It built synthetic versions of real participants in its American Trends Panel, giving them demographic information and previous answers about their political views. It then asked them questions from three surveys conducted in 2026. [2]
With Claude Opus 4.6, the main model used in the experiment, the estimates had a mean absolute error of 12.4 percentage points. That figure summarises how far the synthetic response percentages differed from those recorded in the human surveys. [2]
The findings raise a question for those of us working in Consumer Insights: how do we check that these tools preserve the differences we need to understand?
Experimenting with AI and insisting on validation are part of the same professional standard. We are learning how to use these tools. We try things, make progress, encounter limits, take a step back and try again. What fails today may improve. We also need to be willing to reconsider what excites us today. Our conclusions should change as the evidence changes.
What we are testing
A synthetic data provider could reasonably object that Pew did not evaluate its product. It may use different data, models adapted to a particular category or specific calibration methods. It may also be solving a different problem. Choosing between proposals and estimating how many people hold an opinion require different capabilities.
The term synthetic data covers several different approaches. Take a survey that has already been conducted. We can generate additional records answering the same questions, aiming to preserve the patterns in the original data. This is often called sample boosting. We can also build a digital twin of a participant and ask it something we never asked that person. Here, we are trying to predict a new response. A method that works for expanding an existing dataset has not necessarily shown that it can answer new questions accurately.
An August 2026 report by KS&R makes this distinction explicit. Its ECHO Index assesses how well synthetic sample expansion preserves response proportions and relationships between variables in the original data. Extending the index to digital twins is described in that report as future work. [4]
The results include a successful case. For a dataset of 518 participants and 11 questions about trust in technology, all five providers tested met its fidelity criterion. For another dataset, with 2,058 participants and more than 500 questions across different topics, all four providers tested fell outside it. These are results from KS&R’s own framework, with the providers anonymised. [4]
There is also promising evidence on simulations of individuals. Joon Sung Park and his colleagues built agents using information from 1,052 people. In the June 2026 version of their study, agents based on two-hour interviews answered General Social Survey questions that had not been used to build them. Their accuracy was equivalent to 83% of the accuracy achieved by the people themselves when answering again two weeks later. That benchmark acknowledges that people do not always give the same answers either. It is a different measure from the gap between response percentages calculated by Pew. [5]
These findings invite us to investigate what information and design help a model represent someone more accurately. A promising result deserves attention, as does a failure. Both need to remain tied to the task that was actually tested.
What counts as getting it right also matters. KS&R distinguishes between preserving the statistical characteristics of human data and obtaining the same results when analysing it. It calls the first fidelity and the second utility. [4]
Imagine two equally sized groups rating their interest in an idea on a scale from one to five. One group gives it a two; the other gives it a four. The average is three. A simulation that gives everyone a three gets the average right but erases the difference in interest between the groups. That difference could be decisive if we want to know whom to develop a product for.
How much each difference matters depends on the question. A tool can be useful for a specific decision without accurately representing every aspect of a population. We need to test the capabilities required for the use we have in mind.
Room for uncertainty
In Pew’s study, 47% of questions had at least one answer option that no synthetic participant selected. That did not happen for any question in the three human surveys. [1] Some differences present among people disappeared from those simulations.
For opinion questions that explicitly offered a “not sure” option, human participants selected it 16% of the time. Synthetic participants did so 4% of the time, even though their instructions allowed them to be unfamiliar with a topic or make mistakes. [3]
Get Roger Federico Rybak’s stories in your inbox
Join Medium for free to get updates from this writer.
An “I don’t know” can be the beginning of a better question. Perhaps the topic is unfamiliar. Perhaps someone is still forming an opinion, or the options fail to express what they think. The answer alone does not tell us which explanation applies. But it lets us recognise that there is something we still need to understand.
This makes me think of a temptation in category and brand research: assuming that people pay them as much attention as we do. A team may spend weeks debating a sentence on a pack. The person buying it may be choosing it while trying to remember what else they need for dinner.
My hypothesis is that a representation too ready to answer could reinforce that assumption. It could make a category seem more familiar or more present in everyday life than it actually is. We would need to test that. Pew studied US surveys on topics of public interest, rather than brand research.
What we choose to investigate
Validation is already part of the design of some commercial solutions. YouGov Parallax combines exploration using twins of real participants with research involving people. It distinguishes between verifying the method, by comparing a twin’s responses with those of its human counterpart, and validating the findings of a study. For study validation, it offers the possibility of checking all results, selected results or new questions. This is the provider’s description; it does not by itself establish how the approach performs in each application. [6]
The ability to choose what to check raises another question. Imagine ten product ideas. We test them first with synthetic consumers and select three to research with real people. The human research can tell us whether those three appeal to people. But it cannot tell us whether there was a good idea among the seven we discarded that the simulation judged poorly.
How do we check whether AI is discarding ideas that would be worth exploring with people?
Human researchers also select and discard. AI could help us explore many more alternatives with the same budget while missing some that another method would have picked up. We need to compare what we discover and what we miss when researching with and without that help, taking the available resources into account.
We could research some of the discarded ideas with people, or compare what we find when we start with people instead. These are ways to examine that risk. Pew did not study teams selecting concepts or demonstrate that they are losing opportunities. That is a question I am proposing we investigate.
The KS&R report adds another observation. Its process includes adapting hypotheses to what a synthetic model can address. When reviewing the results, it recommends checking whether the study’s core objectives were lost. [4]
Narrowing a question can be a sound methodological choice. But if we wanted to understand a need and ended up studying only the part the model handles well, we should return to the original question. What we left out may still need an answer through another method.
Rare responses are not automatically insights either. They may reflect errors or differences that are irrelevant to a decision. Their value requires interpretation. What we need to preserve is the opportunity to examine them before deciding they do not matter.
Keep learning
Human surveys have their own limits. The sample and the questions shape what we can learn; our interpretation may simplify it further. We therefore need to compare each tool with the alternatives available for the task. Sometimes the alternative to exploring with AI will be deciding on intuition or leaving a question unresearched.
If combining simulation with human research helps us discover more, preserves relevant differences and improves decisions with the resources available, we will have reasons to expand its use. When a test reveals a limit, we will have reasons to adjust the model, narrow its scope or choose another method.
I want to keep exploring what AI can offer us. I am interested in whether it helps us ask questions we could not address before, and recognise what we still do not understand. Including the people the model didn’t expect.
Sources
[1] Pew Research Center — How well synthetic polls capture the diversity of public opinion.
[2] Pew Research Center — Can AI Stand In for Human Survey-Takers? Not Really.
[3] Pew Research Center — How synthetic respondents express certainty and factual knowledge.
[4] KS&R — State of Synthetic Data Governance + The ECHO Index. Ben Cortese. August 2026. Pages 5–7, 13, 15–18 and 26–27.
[5] Park and colleagues — LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals. Version 3, June 2026.
[6] YouGov — Parallax and frequently asked questions.
