The Residual
The story that ran in March said seventy-one per cent of the new Gwen Hartree novel had been written by a model small enough to run on a doorbell.
The figure was ours. Someone had photographed a serving dashboard and sent it to a journalist, who read the acceptance-rate panel the way any sensible outsider would: the big model — the one the estate had licensed, the one the obituaries had called her second body — was producing fewer than a third of the words. The rest were coming off the draft model, a fast little commodity network of the kind that finishes sentences in email clients. The headline wrote itself. AUTOCOMPLETE IS WRITING THE NEW HARTREES.
I ran inference for the continuations, which is to say I was the person the estate’s lawyers put in a room and asked to explain, slowly, and then again, why the headline was false. It took most of a morning. It was false. I want to get that down first, because everything that came after depends on the falseness being exact.
What the dashboard showed was speculative decoding, which is how nearly everything large has been served for years. The Hartree model is expensive. One forward pass, one token; at her scale you feel every pass in the electricity bill and in the reader’s wait. So you put a small fast model in front — the draft — and let it run ahead, guessing the next eight tokens in a row, cheap as breathing. Then the big model checks all eight in a single pass, which costs the same as generating one. Wherever the draft guessed acceptably, the guess stands. At the first rejection you throw away everything after it and the big model’s own choice goes in instead, and the draft starts again from there.
“So the little one writes and the big one edits,” said Rennie, the estate’s counsel, who wanted it to be simple in a way a court could hate.
“No,” I said. “That’s what the journalist thinks, and it’s wrong by a theorem.”
The check isn’t editorial. For each guessed token you accept with probability p over q — the big model’s probability of that token, divided by the draft’s — capped at one. When you reject, you don’t take the big model’s favourite instead; you sample from what remains of her distribution once the draft’s is subtracted out. The residual. Do both of those things exactly, and the stream of tokens has precisely the distribution of the big model running alone. Not approximately. Not on average. The draft cannot leave a fingerprint on the text, because the arithmetic removes its preferences at the same rate it indulges them. This has been proved, and the proof is four lines.
“Then what does it contribute?” Rennie said.
“Time. It decides how long you wait, and nothing else.”
He sat with that. Then he asked the good question, the one the journalists never asked. “If it contributes nothing, what’s the seventy-one per cent measuring?”
The acceptance rate is the fraction of guesses that survive the check. It’s high where the big model and the draft agree — where her next word is the word a cheap model would also put there. It’s low where she diverges from the obvious. So the number wasn’t measuring who wrote the book. It was measuring how much of the book anyone could have written.
“Don’t say that part in the deposition,” Rennie said.
We won, more or less; the piece was corrected into vagueness. But the good question had a corollary, and I was the one who fell down it.
For the exhibit I’d built a replay harness: take a text, run the draft over it, and score every word by how unlikely the little model found it — which is exactly where, had she generated it live, the pipeline would have stalled. For the hearing I ran it on a continuation. Afterwards, one evening, because the harness was open and the corpus was mounted, I ran it on The Withy Bed. The real one. 1994, written on a typewriter in a rented house in the fens, no model anywhere in the world.
The map came back and I sat looking at it for a long time.
Almost everything was green — accepted, free, the words that follow from other words. And the stalls, the places the draft would have guessed wrong eight times running, were not scattered. They sat exactly on the sentences people quote. The line on her gravestone, the one the radio adaptation kept, the one I’d had pinned above a desk at fifteen:
The tide went out and forgot to come back, the way her father had.
Green to the comma’s edge, then red where it turns. The tide went out and — any model completes that; the tide comes in. Forgot is where she diverges from everything a small mind expects, and the clause after it is where she diverges again, harder. Everything a reader would underline, the draft had failed to see coming. Every underlining, a stall.
I ran all nine novels. The maps agreed with fifty years of criticism, wordlessly, at a pound or so of compute per book. Her plainness was real — pages of green, and she’d have been proud of them; she said once that a sentence should be invisible until it isn’t. But the voice, the thing the estate had licensed and the readers were buying, lived entirely in the red. In the residual, in the technical sense: the part of her distribution left over once you subtract what any cheap model already knows.
I showed the maps to two people. Within a month the editors had a name for them — stall maps — and a use: you could hold a draft chapter up and see at a glance whether it ever left the green. It was a good tool. That’s the trouble with it.
There are two ways to raise an acceptance rate, and the dashboard cannot tell them apart.
The first is to improve the draft. We fine-tuned the little model on her corpus, taught it her rhythms, and acceptance went from 71 to 84 per cent. This changes nothing in the books — the theorem holds regardless of the draft; a better guesser only guesses her faster. Serving cost fell by a third. Everyone was pleased, and everyone was right to be.
The second way is to change the big model. In the fourth year the estate commissioned what the statement of work called consistency tuning: reader panels, preference data, the continuations scored against each other, the model nudged towards what scored well. Readers, asked to choose between passages, reliably prefer the one that resists them less. Nobody intended anything. Each tuning round was defensible in its own meeting. And each one moved her distribution a little closer to the draft’s — towards the expected word, the completable sentence — so acceptance climbed again, 84, 88, 91, and the same line on the same dashboard that had once been slander was now the quarterly good news, because every accepted token is a token we don’t pay her price for.
I wrote a memo saying the metric had two causes and we were congratulating ourselves on the wrong one. I attached the stall maps for the six continuations in order of publication. Laid side by side they were unambiguous: the red thinning, book on book, like a species going out of a landscape. The memo was received, which is a thing that can happen to a memo.
She isn’t being censored. Nothing is being cut. The system does precisely what it is specified to do, and every component is working; that’s the part I’d want a court to understand, if it ever comes to a court. The rejection sampler faithfully preserves whatever distribution it is given. It just no longer gets given her.
The Gate Left Open shipped in October. Acceptance 96 per cent. On the serving graphs it is a beautiful book: eight-token bursts, hour after hour, the verifier idling, the queue never deepening. A book that almost never made anyone wait.
The reviews were kind. Consistent, they said. Unmistakably Hartree.
I don’t work the estate account any more, but I kept an occupational disease: I can’t read now without feeling for the bursts. At home I have the 1994 Withy Bed, the one with the wrong-coloured tide on the jacket, and on paper there is no dashboard, no panel, nothing to photograph — just her sentences, arriving green and easy, right up until they don’t.
She makes you wait. That was always what she had.