Maybe the Solution to AI Is More Human Than We Think
I thought I was building a library. I think I was actually trying to figure out how humanity remembers.
This is the third part of a series about a question I have become increasingly unable to ignore. In Part 1, We Don’t Have an AI Problem. We Have a Knowledge Problem., I argued that we are spending enormous amounts of energy worrying about what increasingly powerful AI systems will do while paying far less attention to what they are learning from. In Part 2, Provenance Isn’t Truth. AI Needs an Epistemic Supply Chain., I tried to imagine what a better information system might look like.
But the truth is that I did not start thinking about this because of AI.
I have been annoying my good friend Dr. Cody J. Foster with some version of this argument since 2011. We met through the McNair Scholars Program at Indiana University, which feels fitting in retrospect because Cody is exactly the kind of person you want around when you are trying to figure out how knowledge gets made, preserved and challenged. He is a brilliant historian, a serious thinker, and one of those people whose brain makes you want to bring him every half-formed idea you have just to see what happens when he gets ahold of it.
Cody studies history. And for more than a decade, I have kept telling him that he should shift at least some of his attention away from studying the past and toward documenting the present.
My argument has always been basically the same: today is the future of history.
The things historians fifty or a hundred years from now will desperately wish somebody had documented are happening around us right now. Not just elections, wars and presidents. The ordinary machinery of being alive. How people talked to each other. What work felt like. What technologies changed us before we had words for the change. What people believed while something was happening, before history cleaned it up into a timeline and gave everybody the benefit of hindsight.
We are living inside somebody else’s primary sources.
I have always found that fascinating. Instead of only asking what happened, I want somebody standing here saying, This is what it looks like while it is happening.
Because once the moment is over, something changes. We remember the outcome. We forget the uncertainty. We forget which possibilities still felt plausible. We forget what information people actually had when they made their decisions. We clean up contradictions that were obvious in real time. Eventually the story becomes smoother than the experience ever was.
That instinct is probably the real origin of Devolve.
Years ago, I started sketching out an idea for a platform by that name. I was frustrated with the internet, but not because there was too little information. There was already far too much. I was frustrated because I could find ten thousand answers to a question and still have trouble figuring out why I should believe any particular one.
Popularity had become confused with expertise. Repetition could make a claim feel factual long before anybody had established whether it was. Search engines were extraordinary at finding words and much less useful at preserving the intellectual history underneath them.
At the same time, I kept noticing another problem. Enormous amounts of real human knowledge never entered the systems we formally call knowledge.
The person who has repaired the same type of machine for thirty years knows things about that machine that may never appear in its manual. A nurse can notice a pattern across hundreds of patients that has not yet become a research question. Someone who lives with a disability can understand the actual operation of a system in a way its designers never will. A family can preserve details of a community’s history that no archive thought important enough to record.
Those things are not automatically scientific evidence. They are not automatically universal. They can be wrong.
But they are still information.
I wanted a way to preserve that distinction instead of throwing the information away.
The original Devolve plan imagined subject-matter experts contributing knowledge, other qualified people reviewing it, systems for documenting credibility and eventually a large body of verified information that could be used not only by people, but by artificial intelligence.
At the time, the AI component felt futuristic.
Now I look back at it and think: Oh.
That may have been the entire point.
Some of the original idea has aged badly. I thought blockchain might play a much larger role than I do now. I talked about removing bias from information, which I no longer think is either realistic or desirable as a design goal. People are biased. Institutions are biased. Experts are biased. Researchers are biased. I am biased. Hiding those influences does not make information neutral. It just makes the influences harder to see.
What I want now is not an unbiased system. I want an honest one.
I want to know where a claim came from, what evidence supports it, who evaluated that evidence and who disagrees. I want to know whether the conclusion was obvious or whether twelve qualified people spent six years fighting about it. I want to know whether new evidence changed the answer. I want the correction to remain connected to the mistake.
Most importantly, I want us to preserve the process of knowing, not just the answer we eventually settled on.
AI has made that need much more urgent because we are beginning to close a loop that did not exist before.
For most of the internet’s history, humans produced nearly everything on it. That certainly did not make it accurate, but there was at least a human somewhere upstream. Someone wrote the article. Someone took the photograph. Someone made the measurement. Someone misunderstood the study. Someone invented the lie.
Now machines are creating a growing share of the information environment too. They are trained on human information, then generate new information based on what they learned. That material returns to the internet, where other humans encounter it and other machines retrieve it. Future systems inevitably encounter information created by earlier systems.
That cycle can be enormously useful. It can also gradually make the origins of information harder to see.
And this is where my old argument with Cody suddenly connects to AI in a way I did not anticipate in 2011.
We need people documenting the present because somebody has to preserve information that has actually touched reality.
Someone has to be there.
Someone has to stand beside the river and measure it. Someone has to examine the patient. Someone has to open the broken machine and discover that the manual is wrong. Someone has to watch the animal behave strangely, interview the witness, photograph the damage, notice the pattern, record what people were saying before anyone knew how the story ended.
AI can do extraordinary things with those observations once they exist. It can compare them, organize them, connect them across disciplines and surface patterns no individual person could possibly see.
But something still has to happen before the analysis.
Reality has to enter the record.
And I think we sometimes picture that entry point far too narrowly.
When people talk about giving knowledge to AI, the image is often technological. Upload the PDFs. Connect the database. Index the documents. Build the retrieval system. Feed the machine more material.
But some of the most important knowledge we can give AI will never begin as a document.
Sometimes it begins with people sitting in a room together.
Maybe an engineer says, “Technically that result is within tolerance, but I would never let it leave the shop that way.” A researcher explains why a statistically significant result might still be practically meaningless. Someone who actually uses the system points out a failure mode nobody designing it thought to test. Another person says the baseline itself is wrong. Then somebody asks the annoying question that forces everyone to reconsider the assumption underneath the whole thing.
They argue. They compare experiences. They find the place where two theories overlap and the place where they absolutely do not. They decide what should count as normal, what should trigger concern, what evidence is strong enough to act on, and which edge cases the original framework completely missed.
Eventually that conversation might become a standard, an evaluation rubric, a set of thresholds, a testing protocol, an annotation guide or, yes, a PDF.
But the PDF was not the knowledge.
The knowledge was created in the room.
It came from putting different human minds in contact with one another and forcing their models of reality to intersect.
That is another kind of entry point into AI, and I think we need much more of it.
We should not only be asking how to feed human information into machines. We should be deliberately pulling humans together to improve what the machines are being taught in the first place.
Bring the historian into the room with the technologist. Put the person who built the system beside the person who has to use it every day. Let the academic theory collide with twenty years of field experience. Bring in the person whose experience does not fit the baseline and ask whether the baseline is wrong.
Then document what happens.
That is human-in-the-loop at a much deeper level than approving an answer after the model generates it. It is humans deciding what good reasoning should look like before the machine is ever asked to reproduce it.
It is humans defining standards together.
It is humans finding gaps in one another’s thinking.
It is people bringing ideas into the room that nobody else knew were missing until they arrived.
The model gets better because the humans make their tacit knowledge explicit.
And something else happens in those rooms that matters just as much: people change each other.
The engineer learns something from the historian. The researcher sees a practical constraint the dataset never captured. The practitioner discovers that the strange thing they have noticed for ten years actually has a name in another discipline.
Knowledge grows at the intersections.
That is not something I want AI to eliminate.
That is something I want AI to help us do more of.
The machine can remember what the room cannot. It can preserve the competing theories, trace the decisions, compare the new standard against thousands of cases and tell us where it starts to fail.
But the room still matters.
The people still matter.
That human role does not become less important as AI gets better. I think it becomes more valuable.
As synthetic information becomes cheaper, authentic contact with reality becomes scarcer. And scarcity creates value.
We are approaching a world in which producing another article, another photograph, another voice, another explanation or another summary costs almost nothing. When something becomes abundant enough, its abundance stops being particularly valuable.
The scarce thing moves somewhere else.
I think the scarce thing is going to be contact with reality, and the human judgment required to make sense of it.
Was this actually observed? Did this interview really happen? Did somebody actually make this measurement? Is this photograph connected to an event in the physical world? Is this account from someone who experienced it? Did qualified people sit together and argue about what the evidence actually means? Can we reconstruct the path between what happened and what I am reading now?
Maybe we need to think about human-generated knowledge almost like a biological seed bank. Not because human information is inherently pure. It very obviously is not. Humans created propaganda, pseudoscience and bullshit perfectly well without any assistance from artificial intelligence.
The point is origin.
A seed bank preserves genetic material outside a changing environment so that diversity is not permanently lost. I wonder whether we will eventually need something conceptually similar for information: protected collections of original observations, human writing, interviews, field measurements, photographs, research, community records, firsthand experience and even the deliberations that produced our standards.
Not a vault of unquestionable truth.
A tether.
Something connecting the information ecosystem back to the world.
That same idea changes how I think about history, too.
History is not merely a collection of facts about what happened. The richest historical records preserve what people knew at the time, what they believed, what they feared, what they misunderstood and what they could not yet see.
That is why primary sources matter.
A letter written in 1943 is valuable partly because the person writing it does not know how 1945 turns out.
There is something almost impossible to recreate once hindsight arrives.
We should be thinking about that right now.
Get Kelly Turner’s stories in your inbox
Join Medium for free to get updates from this writer.
What does it feel like to live through the arrival of generative AI before we know what it ultimately becomes? What assumptions are we making that will look ridiculous in thirty years? Which things are we worried about that will turn out to be irrelevant? Which changes are happening quietly enough that nobody thinks to document them?
That is the future of history Cody and I have been arguing about for fifteen years.
And now it is also part of the future of AI.
Because preservation alone is not enough. We also have to preserve change.
The internet is bizarrely bad at this. An old medical recommendation can sit beside a newer one forever. A study can be overturned and remain one search result away from the research that overturned it. Someone can change their position after seeing better evidence, while the old quote continues traveling indefinitely.
The internet remembers the statements.
It frequently loses the intellectual journey between them.
That is not how I want an AI to understand knowledge.
I want it to be capable of saying that this is what we believe now, this is what we believed ten years ago, and this is the evidence that changed our minds. I want it to understand that disagreement is not always evidence that somebody has failed. Sometimes disagreement is the most accurate representation of the available evidence.
I want it to understand that “we don’t know” is an answer.
More than that, I want us to become comfortable with that answer again.
One of the things AI exposes about people is how strongly we respond to confident language. A clean, polished answer feels intelligent. A complicated answer full of qualifications feels less satisfying.
Reality does not care which one we prefer.
Sometimes reality is messy. Sometimes five good studies point in one direction and three point somewhere else. Sometimes thousands of people report an experience that nobody has adequately studied. Sometimes the official answer is based on the best evidence available and is still wrong.
A trustworthy information system cannot solve that problem by pretending the uncertainty disappeared.
It has to preserve the uncertainty.
That brings me back to another part of Devolve I still believe in deeply: the idea that expertise should extend beyond the small group of people traditionally authorized to create knowledge.
Academic knowledge is vital. Scientific methodology is vital. Professional expertise is vital.
But people know things for different reasons.
The mechanic and the engineer may understand different parts of the same machine. The physician and the patient may understand different parts of the same illness. A historian can reconstruct what happened from archives, while someone who lived through it remembers details no archive preserved.
Those forms of knowledge should not be flattened into equivalence.
They also should not be forced into competition.
I want an information system intelligent enough to know what kind of evidence it is looking at.
One person’s experience is very strong evidence of what happened to that person. It is weak evidence that the same thing happens to everyone. If five thousand independent people describe the same unexpected phenomenon, that still may not prove a causal relationship, but it becomes information worth investigating.
The current internet handles that badly. It tends to force us toward two choices: believe the anecdote or dismiss it.
There is a much more useful option.
Document it, preserve it, see whether it repeats, then investigate.
This is where AI could become something more interesting than a replacement for human work. It could become a tool for seeing human knowledge that we currently miss.
Imagine being able to recognize that thousands of nurses in different hospitals are independently describing the same unusual response before anybody has formally connected the reports. Imagine detecting the same equipment problem appearing in mechanics’ notes across several countries. Imagine historians being able to examine millions of ordinary people’s firsthand records of an era instead of reconstructing everyday life mostly from whatever institutions happened to preserve.
That future requires people to keep producing original knowledge.
And those people should be compensated for it.
This is another place where I would change my original thinking. We cannot declare human knowledge essential infrastructure for artificial intelligence and then treat the humans creating it as free raw material.
Someone conducted the study. Someone wrote the report. Someone took the photograph. Someone spent thirty years learning the craft. Someone preserved the community history. Someone checked the claim. Someone corrected the mistake. Someone spent four hours in a room arguing over where the baseline should actually be set.
That labor has value.
As AI makes content nearly infinite, I think the economics of information will eventually have to shift away from rewarding volume and toward rewarding trust, originality, traceability and maintenance.
The valuable question may stop being, “Who can produce the most information?”
It may become, “Who can show us where this information came from, and why we decided it was worth trusting?”
That brings me to the part of the old Devolve plan I would change most dramatically.
I used to think the hard problem was technical.
It is not.
The hard problem is people.
Who gets to call themselves an expert? Who decides whether evidence is good enough? What happens when qualified people disagree? How do you prevent companies from purchasing legitimacy? How do you stop governments from defining inconvenient information out of existence? How do you preserve a minority interpretation without pretending every fringe belief deserves equal weight? How do you include lived experience without converting every experience into universal fact?
No blockchain solves that.
No algorithm solves that.
Those are governance problems, and governance is difficult precisely because knowledge belongs to human beings who have incentives, blind spots, conflicts, loyalties, egos and limited perspectives.
That is why I no longer want a system that promises to tell us what is true.
I want something more modest and, I think, more powerful.
Show me how we know.
Preserve the evidence, the argument, the objection, the correction and the person who changed their mind. Preserve the room where the standard was debated, not just the standard that came out of it.
Then let the machine help us navigate it.
When I first imagined Devolve, I called the ultimate product a library.
I don’t think I would use that word now.
A library sounds like a place where knowledge goes after someone has finished creating it.
What I am imagining now could never be finished.
It would have to change every time we learned something. It would have to remember previous versions without confusing them with current ones. It would have to preserve disagreement without collapsing into “everyone has their own truth.” It would have to make room for formal science and firsthand human experience while being honest about the difference between them.
And increasingly, I think it would also have to preserve the human process that turns information into standards.
The meeting. The disagreement. The edge case. The person who asked the question nobody else thought to ask. The moment two disciplines realized they were describing the same phenomenon with different language.
That is knowledge too.
It would have to behave less like a library and more like a living memory.
That realization brings the entire series back around for me.
I do think we need to take the risks of advanced artificial intelligence seriously. We need alignment work, independent evaluation, security, governance and limits on systems when their capabilities outrun our ability to control them.
But I think there is another AI safety project sitting upstream of all of that.
We have to protect the integrity of the thing intelligence consumes.
We have to preserve some connection between information and the world it describes.
And maybe that work begins much earlier than AI.
Maybe it begins with Cody documenting what is happening today instead of waiting fifty years for somebody to reconstruct it. Maybe it is the nurse writing down the thing she keeps noticing or the technician documenting the failure nobody can reproduce in a lab. Maybe it is a group of people with completely different expertise sitting around a table until they finally agree on what the baseline should be, and preserving not just the answer but how they reached it.
Those acts seem small next to the scale of artificial intelligence.
I no longer think they are.
They are how reality enters the record.
They are how isolated human observations become shared standards.
They are how we pull humans into the development of AI instead of slowly engineering the humans out of it.
And the more powerful our machines become, the more valuable that connection becomes.
We keep asking what will happen if AI becomes smarter than we are.
After years of thinking about this, I am much more interested in another question:
What happens if we build machines capable of extraordinary intelligence, but give them a civilization that has forgotten how it knows anything?
I thought I was trying to build a library.
I think I was trying to preserve the answer to that question.
Maybe what we need now is not simply a better artificial intelligence.
Maybe we need a better human memory.
Part 3 of 3: The Missing Knowledge Layer of AI
Written by the ramblings of Kelly and the unifomity of ChatGPT.
Before you go
- Please take a moment to like the post and follow the writer!
- Did you know that over 400,000 developers share what they’re building, learning, and discovering across our platforms every month? Learn how you can contribute here.
