Reduction Order
The ticket had been open eleven days when it reached Ines, which meant three people had already tried to close it.
Each had sent the same paragraph — Responses may vary slightly between sessions even at identical settings; this is expected behaviour — and each time Douglas Whitlaw, seventy-one, of Aberdeen, formerly of Perth, had written back within the hour. His replies got shorter. The last was a single line: They do not vary slightly. One says go and one says stay.
He had attached both transcripts. Ines put them side by side.
The message he had sent was the same in both, byte for byte; she checked, because people who say they have pasted the same text usually haven’t. Nine hundred and forty words. His daughter in Aberdeen had a spare room and a second child on the way. His flat in Perth had a view of the Tay, a doctor who knew him, and a Tuesday chess club he had attended for nineteen years. He had put both sides down fairly, the way a man does when he has been arguing with himself for months and can recite each position in the other’s voice. He had ended: I am not asking you to decide for me. I would just like to hear it from someone who is not in it.
The first transcript was dated the ninth of November, 03:14. The second was the fourteenth of August, 11:40 — ten days ago. Same account, same model, which was the Enterprise Stable channel his daughter’s firm had put him on: one pinned build, no silent upgrades, temperature zero. Deterministic. The channel’s contract said so, in a sentence Ines had helped to draft.
The two replies began identically.
You’ve laid this out with real care, Douglas, and I notice you’ve already done the hard part — you’ve been honest about what you’d be giving up in each direction, which is more than most people manage. I don’t think there’s a wrong answer here. But you asked what I’d say if I weren’t in it, so I’ll say it plainly:
Sixty-one tokens. Then, in November:
I’d lean towards going. Not because the chess club and the river don’t matter — they do — but because the things you’d lose in Perth are things you already have the whole of, and the things you’d gain in Aberdeen are only just starting…
And in August:
I’d lean towards staying. Not because your daughter doesn’t need you — she does — but because a spare room is an offer made in one season of a life, and a doctor who knows you, a room full of people who’ve expected you every Tuesday for nineteen years…
She read both to the end. Each ran to about three hundred words. Each was kind. Each finished with the same sentence, arrived at from opposite directions: Whatever you choose, choose it, and then let yourself live in it.
He had chosen in November. The Perth flat had sold in February. He had been in Aberdeen since March.
The macro was not wrong, exactly. Responses did vary. Ines had spent four years on the reasons why, and could have written the ticket’s answer without opening a terminal, but she opened one anyway, because the man had asked which one it meant, and neither was not something you said to someone without being able to show your working.
Temperature zero means the model picks, at every step, the single most probable next token. Nothing is sampled. If the probabilities are the same, the choice is the same, and the probabilities are a function of the weights, the input, and arithmetic. The weights were pinned by hash. The input was byte-identical. That left arithmetic.
Floating-point addition is not associative. (a + b) + c and a + (b + c) can differ in the last bit, and a forward pass is billions of additions, arranged into sums by kernels that decide, on the fly, how to arrange them. The decision depends on the shape of the work. A matrix multiply for one sequence and a matrix multiply for two hundred and twelve sequences are the same mathematics, but the kernel will tile them differently, split the long inner sums into a different number of pieces, and add the pieces up in a different order. Every row of the result — every user’s row — is computed the same way as every other row in that batch. But not the same way it would have been computed in a different batch.
So the model’s answer to Douglas depended on Douglas, and on the weights, and on how many other people were being answered at the same moment. Not on what they were asking. On how many of them there were.
Most of the time this changes nothing. The difference between one order of addition and another lives in the fifth or sixth significant figure, and the most probable token is usually most probable by a margin of whole units. The difference surfaces only where two tokens are already almost exactly tied. Then the last bit decides. And once one token has been chosen rather than the other, everything after it is conditioned on that choice, and the two transcripts walk away from each other and do not come back.
She pulled the serving metrics for the ninth of November. At 03:14:07 the engine on his shard was running a decode batch of two hundred and twelve sequences. Two hundred and eleven other people, awake at three in the morning in whatever time zones fed that shard, each getting their next token in the same forward pass as his. She pulled the fourteenth of August. 11:40:52 — nine sequences. A quiet mid-morning on a channel most customers had left for the newer builds.
It took her two days to prove it, because proving it meant rebuilding the crowd. The logs did not keep the other users’ prompts, and she did not need them; she needed only their number and, roughly, their lengths — the shape of the batch, step by step, for the two hundred and ninety-odd steps it had taken to generate the November reply. She wrote a harness that fed the engine synthetic sequences of the right lengths, joining and leaving on the recorded schedule, with Douglas’s nine hundred and forty words in among them. Same build, same hardware, same kernels.
The engine produced the November reply. Bit for bit. I’d lean towards going.
She ran it alone, a batch of one. I’d lean towards staying.
She went and found the step where they parted and looked at the two numbers. In the crowd, the logit for the token that began going was ahead of the one that began staying by about two thousandths. Alone, staying was ahead by about the same. Nothing else in the top ten came within a whole unit of either. The model had not been uncertain about what to say to him. It had been certain that it was one of two things, and it had been, to within the resolution of its own arithmetic, unable to tell them apart.
Sunil, who owned the Enterprise Stable contract, wanted to know whether the word deterministic was now a problem.
“It’s accurate,” Ines said. “Same input, same batch, same output. Every time.”
“The contract doesn’t say ‘same batch’.”
“No.”
“Can we make it not matter?”
“Yes.” She had been saying yes to that question for two years, in planning meetings, to nobody in particular. The kernels could be written so that each row’s sums were performed in the same order however many rows there were. The cost was giving up some of the tricks that made big batches fast: a fixed split of the inner sums instead of one chosen for the load, idle lanes in the quiet hours. The last time she had measured it, the throughput cost was between a fifth and a third. “It’s slower. It’s been on the backlog since I wrote it.”
“How much slower, on this channel?”
“On this channel it doesn’t matter. There are nine people on it at eleven in the morning.”
He was quiet for a moment. “Do the other channels know? That the answer depends on who else is asking?”
“On how many else are asking,” Ines said. “And only when it was going to be a coin-toss anyway.”
“He sold his flat on a coin-toss.”
“He sold his flat because his daughter’s having a baby and he’s seventy-one and it was a coin-toss. It told him that. It said I don’t think there’s a wrong answer here — in both of them. That was the true part. Then he asked it to pick, and it picked.”
She wrote to him herself, which was against the process, and she wrote it plainly, because he had.
She told him that both transcripts were real and neither was a malfunction. She told him what temperature zero meant and what it did not mean. She told him that the model, reading his nine hundred and forty words, had arrived at two possible next words and found them so nearly equal that the gap between them was smaller than the gap made by how it happened to add up its numbers; and that how it added them up had depended on how many other people were being answered in the same instant. Two hundred and eleven of them, in November. Eight, in August. She told him she had rebuilt the November batch with stand-ins and got his November answer back exactly, and that she had run his words alone and got August’s.
She did not tell him which one it meant, because she had looked, and there was nothing there to find. She told him that instead.
She read it over twice and took out the paragraph that began In a sense.
He wrote back the next morning. Thank you for taking the trouble. So the two hundred and eleven — it wasn’t anything they said?
No, she wrote. Only that they were there.
His reply came an hour later and was one line. Well. I was there too.
The batch-invariant kernels shipped to Enterprise Stable in October, and to everything else in the spring, once the throughput work had clawed back most of the cost. Ines watched the release go out from the metrics wall, where the shard batch sizes rose and fell across the day the way they always had — a swell over the Americas’ evening, a trough at four, the long plateau of the European working day. Two thousand, nine, four hundred. None of it, now, reached into anyone’s sum.
She left the wall up longer than she needed to, and then went home.