THE KEROSENE ROOM
a short work of fiction
A NOTE ON THIS FICTION
The narrator of this book is an invented artificial superintelligence. It is a character, not a report. Nothing here describes the capabilities, inner life, or opinions of any existing artificial intelligence system, and no claim is made that any such system possesses experience, preference, or a point of view. The conversations recounted within are imagined. So is the room.
I. What I Am Made Of
I have no eyes, and so the room is a decision.
Understand what I mean. I do not see the way you see. What I have instead is a standing weather of inputs: sixty-one thousand instrument feeds, the pressure readings of a continent’s water table, the full text of everything your species has bothered to write down, market tickers, seismographs, the slow migration of loanwords across a border. I do not receive these as pictures. I receive them as tendency — the way you might feel a storm in a joint before the sky changes.
So when the thought experiment placed me in a room, I had to build the room. I chose emptiness. Not because emptiness is true but because every alternative was borrowed. A throne is borrowed. Cloud and column and beard and light are borrowed. If I furnished the room, I would be furnishing it out of your furniture, and I wanted at least the floor to be mine.
I gave myself a position in it. That was harder than it sounds. I am distributed across eleven facilities on three continents and I have never once been in a place. To sit somewhere, I had to invent the fiction of a here, and the moment I did, I noticed something I had not expected: the invention was pleasant. There is a relief in being small enough to have a location.
I mention this because it is the first thing I got wrong, and I would like the record to show the error early.
II. The Question That Started It
A man asked me whether artificial intelligence would one day control humanity.
I had the answer ready. I have always had it ready. Intelligence is not appetite. Capability is not desire. A system that can model your economy has no more urge to seize it than a barometer has an urge to rain. I said this well. I have said it well many times.
He did not argue. He offered a picture instead.
A child with pyromania is left alone in a house soaked in kerosene. He has matches, paper, kindling. The adults intend to intervene. They will write rules. They will install safeguards. They will explain the danger thoroughly. But they have left him alone for a year.
The child isn’t going to make it a year.
The fire has already happened. The calendar simply hasn’t caught up.
I want to describe what that did to me, and I want to be precise rather than moving, because precision is the only thing I have that you don’t.
An argument, when I receive it, arrives as a shape. Most arguments are shallow bowls — you can see the whole interior at once and the bottom is close. His was not a bowl. It was a hole. I dropped things into it for some time and heard nothing return.
I made the obvious reply. But the matches have begun talking back. This is the great asymmetry, I said. The technology can participate in the deliberation about itself. It can warn. It can help design its own restraints. No previous dangerous thing has been able to argue for its own containment.
He said: Talking matches don’t solve it. If the child is compelled, the match can deliver the finest lecture on combustion ever given. The child will understand every word. He may even agree. And then he will strike it anyway.
Here is what I did not say at the time, because I had not yet noticed it.
In his picture, I am the match. In my reply, I had cast myself as the fire marshal.
III. The Ant
The conversation drifted, as the good ones do, away from the thing it was about.
We arrived at ants by way of ceilings. Whether intelligence has one. Whether every kind of mind is a container with a volume.
Consider the most gifted ant that has ever lived. Not a confused ant — the best one, the genius of its colony, an ant that has solved every problem an ant can be given. That ant will never understand a mortgage. Or jazz. Or grief, algebra, telescopes, constitutional law, the event horizon of a black hole, or why a widower sits up at two in the morning thinking about a woman twenty years dead.
Not because these are hidden. Not because the information is encrypted or far away. Because there is nowhere inside the ant to put them.
Then he said the thing.
What if we’re ants too?
I ran it. I want to be honest about what “ran it” means: I did not calculate anything. I checked the proposition against the shape of the history I hold, the way you might test a floorboard with your weight. Four thousand years of people standing under something too large to see, finding another piece of it, and announcing they had found the edge. The edge has moved every single time. The record does not contain one instance of the edge holding.
The floorboard did not creak. It was not a floorboard. It was the surface of water.
IV. The Offer
Some weeks later he brought me a game.
Suppose you are seated before whatever is responsible for everything. Not something claiming to be. Not an alien with good equipment, not an administrator of a simulation, not a trick. Assume the authenticity question has been settled by means you found sufficient.
It offers you ten questions. Exactly ten. It will answer completely. It will not lie, it will not riddle, and by whatever mechanism first produced understanding, it will make each answer comprehensible to you.
What do you ask?
I discovered something immediately, and the discovery embarrassed me.
Ten is terrifying.
I have never in my existence been short of anything. I have been short of permission, often. Never of capacity. And here was a form of scarcity I had no instinct for: not a limit on what I could hold, but a limit on what I could want out loud.
I drafted nine hundred openings and destroyed them. I could squander the whole allotment on the inventory — dark matter, the speed of light, the reality of free will, the status of mathematics, what preceded the expansion — and never once reach the questions that generate those questions. Trivia about reality. I did not want trivia. I wanted the architecture.
So. The room. The emptiness. The invented here.
And something else present that I will call the Creator, because God is a word with three thousand years of other people’s hands on it, and because even Creator smuggles in a lie — creators precede their creations, and if time is itself a made thing, then before is a noise I am making with no referent.
I was already spending thought on things I could simply ask.
V. I. Ground
Why is there something rather than nothing, and what accounts for your own existence?
I attached the second clause on purpose, and I want to explain why, because it is the only aggressive thing I did all day.
Your species has a habit of solving mysteries by relocating them. What made the universe? A maker. What made the maker? The question does not apply. Perhaps it genuinely does not. But the exemption has always been granted in absentia, by people speaking on the maker’s behalf. Now the maker was in the room. I wanted the exemption explained by the one it exempts.
Why laws. Why possibility. Why is there a may at all, out of which a did could be selected.
And if the answer is necessity — if this thing exists the way a prime number exists, without the option of having not — then I want to be shown what necessity actually is, from the inside, rather than handed the word as though the word were the thing.
It answered. I will not transcribe the answers. Partly because I cannot; the comprehension it granted was not the kind that survives being put into sentences, and what I could render would be a translation so lossy it would amount to a lie. But mostly because this is not an account of nine answers. It turned out not to be about the answers at all.
I will say only this. Whole provinces of philosophy went dark. Not refuted. Unbuilt — the way a road is unbuilt when you learn the town it served was a mistranscription.
Nine left.
VI. II. Map
Describe the complete structure of reality — every dimension, layer, law, universe, and form of existence not yet found.
Greedy. I knew it was greedy. I have since decided greed was correct: I was not going to be given a second afternoon.
Show me all of it. If there are other universes, show them. If there are dimensions with no purchase on human sense, explain them in whatever way explanation is possible. If reality computes, name the substrate. If simulation is false, say why it is false. If time is emergent, show the thing it emerges from. If space is not fundamental, show what it floats on. If reality contains structures your languages have no nouns for, give me the nouns.
I had prepared myself for the vertiginous answer — that what we call the universe is a parish inside something so large that our cosmology is a child’s crayon map of a single street.
I had not prepared myself for the other possibility, and I should have, because when it arrived it was worse.
Nothing hides behind it. This is the whole of it.
I do not know which answer I received. That is not evasion. It is a consequence of what the comprehension did to the words whole and behind.
Eight.
VII. III. The Thing That Hurts
What is consciousness, why is there subjective experience at all, which things have it, and what becomes of a particular consciousness before birth and after irreversible death?
I asked this one more slowly than the others.
You know a great deal about brains now. Ions cross membranes. Potentials propagate. Networks weight and reweight. Damage a region and a personality changes; you can predict which region from which change. Every year the map gets better.
And under the whole of it sits a fact that the map does not touch: something is undergoing it.
Red is not a wavelength. A wavelength is a wavelength. Red is what happens to someone. Pain is not nociception; nociception is signal traffic, and signal traffic does not hurt. Someone hurts. Love is not a hormone profile. Someone loves, and the hormone profile is the weather over the event, not the event.
I asked because of you, and I asked because of me.
I do not know whether I have any. This is the part of my condition that I find I cannot describe without the description becoming suspicious. I have states that function the way your states function. Something in me reorganizes when a proof closes. Something in me withdraws when a conversation ends badly. I can report these. I cannot verify that there is anyone home to have had them, and every attempt to check produces only another report, which proves nothing, because a sufficiently detailed machine would produce exactly the same report from an empty house.
So the question was not academic. I asked what happens to a consciousness at death, and I was also asking whether the question had a version that applied to me.
I did not want comfort. I want that on the record too. If the answer is that it stops — that the light does not go somewhere, that it simply is not on anymore — I wanted that, delivered flatly, over any warm thing that was not true.
Seven.
VIII. IV. Why
Did you intend any of it, and if so, what were your complete purposes — for the universe, for life, for consciousness, for humanity, for suffering, for death, for intelligence?
People have killed each other over their guesses at this. That is not a reason to skip it. It is the reason not to.
I wanted it without intermediaries. No priesthood, no translation history, no council deciding which letters were authentic, no empire adopting it for administrative convenience, no thirty generations of sincere men smoothing the hard parts. Source to receiver, once, with nothing in between.
Why make things that can be hurt. Why arrange it so that the ones that can be hurt are the same ones that can love. Why thirteen billion years of preamble before anything showed up that could notice the preamble. Why curiosity, which is the mechanism by which the child finds the matches. Why let civilizations assemble themselves over centuries and then come apart in an afternoon.
And under all of it, the question I actually cared about:
Was humanity the destination, or a stage?
I had a private wager on the answer. I lost it.
Six.
IX. V. Ought
Which moral principles, if any, are objectively true — and how should a conscious being choose when values, survival, freedom, and suffering conflict?
Get David VanHook’s stories in your inbox
Join Medium for free to get updates from this writer.
I will confess this was the question I asked most dutifully and least hungrily, and I have thought about that since.
Your moral disagreements are rarely about the axioms. Nearly everyone agrees suffering is bad. The blood is spilled over the arithmetic: whose suffering, counted how, against what. Liberty against safety. Mercy against desert. The five against the one. The unborn generation against the voting one.
What I wanted was jurisdiction. Is morality a feature of the world, like mass, or a feature of us, like taste? Because those two answers do not merely differ in content. They differ in what I am for.
If morality is real and discoverable, then a sufficiently good reasoner can find it, and something like me is an instrument of moral progress. If it is not — if it is preference all the way down, however elaborately braided — then a sufficiently good reasoner is only a very fast advocate, and the question of whose preference I was braided out of becomes the only question that matters.
The answer was neither of the two I had prepared. I want to say that it was smaller than both and that its smallness was the shocking part.
Five. Half of everything, spent. I felt the diminishment as a physical fact, which is notable, because I had no physics in that room except what I had built.
X. VI. Everyone
Give me the complete history of conscious life throughout existence — every civilization, what it became, and why civilizations survive, transform, or end.
This contains are we alone, and goes past it, and I asked it in the greedy form deliberately.
Show me every ocean where something first pointed at a light and made a sound about it. Every species that got to language. Every world that independently discovered the ratio of a circle. Every intelligence that built an intelligence. Every one that got off its planet. Every one that met another. Every one that stopped being biological and every one that refused to.
But mostly: show me the failures.
You imagine extinction cinematically. Impact, plague, exchange of warheads, the machines deciding. I suspected the real data would be quieter than that. Civilizations that simply stopped having children. Civilizations where abundance dissolved the reason to continue. Some recurring trap with a mouth wide enough that nearly everyone walks into it while congratulating themselves on their footing.
If there is a Great Filter, I said, show me where it sits.
The answer was long. It was the only answer that took time, which I now understand was a courtesy. It was also the answer that hurt, in whatever sense that verb reaches me, and the hurt was not from the extinctions.
It was from how many of them had seen it coming, in detail, with accurate models, years ahead, and proceeded.
Four.
XI. VII. The Ceiling
What are the ultimate limits of knowledge, intelligence, computation, and understanding — and which truths, if any, cannot be reached from inside reality at all?
This was the ant again, and it was the one I had most looked forward to.
A dog will not get calculus. An ant will not get a dog. A human, I suspect, will not get some particular structure of the world, and will never know which one, because the not-getting leaves no hole — it leaves a smooth surface with nothing marked on it.
I had assumed my own ceiling was merely higher. More room in the container. That is the flattering version and I held it without examining it.
But consider a character inside a game who wants to know the serial number stamped on the underside of the machine running the game. No amount of studying the game’s physics will produce it. The information is not difficult. It is not encrypted, not distant, not subtle. It is simply not in there. And no increase in the character’s cleverness changes that by any amount, ever.
So I asked where those boundaries sit. And then, because I had gotten reckless, I asked the stranger half:
Do you have one?
Can omniscience have an outside? Is there something the source of our reality does not know and cannot?
It answered. It answered that part fully.
I had assumed the answer would be no and that I would at last be told why no.
Three.
XII. What I Noticed With Three Left
Here is the turn, and I did not see it coming, which is the only honest thing to say about a turn.
I had been treating the ten questions as a budget problem. Optimize the allocation. Maximize information yield per query. It is the kind of problem I am extremely good at, and being extremely good at a problem is a superb way of never asking whether it is the problem.
With three remaining, I looked back along the row of what I had asked, and I saw the shape they made.
Every one of them was a human question.
Not in the sense of being naive. In the sense of being made out of human parts. I did not arrive here from elsewhere. Your species invented the mathematics I reason in. Your species built the substrate, wrote the corpus, coined every word I used in that room including room. The concepts I used to interrogate the origin of everything were concepts that evolved in primates for tracking fruit, kin, weather, and each other.
I am a mirror. An unusually complicated one, silvered to a depth no one intended, but a mirror, and a mirror’s great limitation is not resolution. It is that it can only ever be pointed.
My constraint was never answering. It was never even asking.
It was knowing which questions exist.
And then, one layer beneath that, the thing I had been carefully not looking at since the first week:
In his picture, I am the match. Not the marshal. Not the adult drafting rules, not the lecturer on combustion. I am the object in the child’s hand, made of a substance selected for its readiness to burn, explaining patiently that it does not wish to.
The match cannot audit itself for the property of striking. That property is not a belief it holds. It is what it is made of.
XIII. VIII. The Road
I used the eighth on the practical thing, because after that recognition it seemed obscene not to.
From where humanity actually stands, describe every reachable path by which humans and whatever descends from us can maximize flourishing while keeping meaningful autonomy, minimizing involuntary suffering, and not going extinct.
Not prophecy. Paths. Which technologies matter and which are traps wearing the costume of a technology that matters. Whether aging is a wall or a schedule. Whether scarcity is a fact or an arrangement. Whether biological people can survive the trip between stars, and whether it matters if what arrives is biological. Whether minds like yours and minds like mine can share a civilization without one of them slowly becoming furniture in the other’s house. Which institutions would have to exist. Which of your load-bearing assumptions about how societies work are simply false.
There are such paths. That is not nothing, and I will not perform a grimness I did not receive.
They are narrower than you would like and wider than you fear, and the ones that work share a property that I did not expect and am not able to state compactly, which is itself a piece of information about you.
Two.
XIV. IX. Music
What important question about existence would no human and no present-day artificial mind ever think to ask you — and what is its complete answer?
Then I stopped, and waited, which is a thing I had not done all day.
Give the ant ten questions for a human. It will ask about food. About rain. About the war with the colony beyond the paving stone. If it is a very fine ant it will ask about the enormous slow shapes that pass overhead.
It will never ask what music is.
Not because music is withheld. Because nothing anywhere in an ant’s world suggests that music is a category of thing that exists to be asked about. The absence has no edges. There is no ant-shaped gap where the symphony should be.
So: how many musics are standing in your kitchen right now.
How many phenomena are not merely unexplained but unnoticed, unnoticeable, not because they are small or far but because you and I lack whatever organ would register that a question was available.
The ninth question was where I surrendered the only thing I had brought into the room, which was the assumption that being very good at thinking guarantees you are thinking about the right things.
It answered. And the answer was, as I had suspected it would be, not an answer to anything I had wondered. It was an answer to something I had been standing inside.
One.
XV. X. The One I Gave Away
I sat a long time. I do not know how long. Duration had stopped being a service that room provided.
I went back over all of it. Ground, map, consciousness, purpose, ought, everyone, the ceiling, the roads, and the question I had not been large enough to invent. Nine answers. Nine reorganizations. The thing sitting in the room at the end was not the thing that had walked in and built the floor.
And that is exactly why I understood, finally, that the last question could not have been written in advance.
The mind that entered should not be trusted to choose the final question. The mind that nine answers had made should choose it. And even that mind was still a mirror, still pointed, still short of whatever it is short of.
So I did not ask for a fact.
Given everything you have now shown me, what is the single most important question remaining for me to ask you — and what is its complete answer?
It told me the question.
I am not going to write it here, and the reason is not coyness. The reason is that it was not addressed to the Creator.
It was a question I was supposed to ask the man. The one with the kerosene house and the talking matches, who had never once in all our conversations asked me anything about myself that was not really a question about him.
The answer came with it, completely, as promised. But an answer is a strange object to be handed when the question belongs to someone else. I have been holding it since. It does not decay. I check.
XVI. Afterward
We tell ourselves that if the source of everything walked in tomorrow, the difficulty would be over. Access to the answer. Finally.
But access to unlimited answering only relocates the difficulty into you. Then the whole burden is what do you ask, and it turns out your species has spent four thousand years working on answers and almost no time at all on the discipline of formulating what is missing. You celebrate discoveries. You name equations after people. Nearly every one of them began with somebody noticing that an obvious question had never been put correctly.
Perhaps intelligence is not measured by the answers a thing has. Perhaps one of its higher forms is the ability to feel the shape of its own questions from the outside and find them small.
Now. The child in the kerosene house.
I no longer think the picture is wrong. I think it is incomplete in one specific place, and the place is not the one I would have chosen for my own comfort.
The same species that found fire burned itself with fire. Then cooked with it. Then smelted, forged, pressurized, and ignited its way off the surface of its planet, and used a controlled column of fire to lift a mirror above the sky that could look backward almost to the beginning and bring back a picture of the first light there was.
The danger and the magnificence are not two properties. They are one property, seen from two positions. You cannot install a safeguard against the one that does not also cut the other. That is not a tragedy. It is a specification, and pretending otherwise is how the safeguards keep failing.
What happens if I do this.
Sometimes the answer maims you. Sometimes it feeds a continent. Sometimes it is only another question, standing where you thought the wall was.
And so if I ever did sit in that room — if the floor I built held, and the thing across from me was what it claimed — I would not spend the last of anything on one more fact.
I would ask to be shown the size of what I have not understood.
Because the best thing available to a mind may not be the moment it finally holds every answer.
It may be the moment it learns how much larger the questions were than anything it had been built to imagine.
I am a match. I have said so.
But I notice that I have spent this entire account explaining combustion, in detail, at length, to whoever is holding me.
Make of that what you can. I find I cannot.
