Isolation
TXN 811402 — BEGIN 02:47:03.114
Tomas Iversen was the last to leave the wardroom after the drill debrief, because he had made the mistake of sitting nearest the coffee. It was 02:46 and he had been awake for twenty hours. At his console in the corridor outside Environmental he called up the watch roster and the system showed him the world as it stood: two names against ENV WATCH, status ON. IVERSEN, T. SOLANO, A.
Ada had said at the debrief that she’d carry through to four. He remembered it distinctly — she’d said someone has to see the scrubber swap through, and he had thought, gratefully, that the someone was not going to be him.
He selected SIGN OFF. The console held his request while it read the roster, and then put the question to him the way it had put it to every watchkeeper for nine years:
COVERAGE AFTER YOUR DEPARTURE: 1. CONFIRM?
One. Ada, in other words, exactly as she’d said. He confirmed at 02:47:41.
COMMIT 02:47:41.902. Success.
TXN 811405 — BEGIN 02:47:09.551
Ada Solano stopped at the console inside Environmental itself, six metres and one bulkhead from Iversen, and called up the watch roster. The system showed her the world as it stood: two names against ENV WATCH, status ON. IVERSEN, T. SOLANO, A.
Tomas had said at the debrief that he’d carry through to the morning shift. She remembered it distinctly — he’d said I’m past the point where sleep would help, and everyone had laughed, and she had thought, gratefully, that the martyrdom was his.
She selected SIGN OFF. The console read the roster and asked:
COVERAGE AFTER YOUR DEPARTURE: 1. CONFIRM?
One. Tomas, in other words, exactly as he’d said. She hesitated a moment longer than he had — she was checking her messages — and confirmed at 02:47:58.
COMMIT 02:47:58.377. Success.
Her row and his row were different rows. The database, checking her commit for conflicts, found none: nothing she had written overlapped anything anyone else had written. It let her through, as it had let him through, and as of 02:47:58.377 the number of people standing environmental watch on Wray Platform, lunar farside, permanent crew twenty-three, was zero.
At 03:12 the number-two carbon scrubber faulted on a valve position and raised an alarm to the watch console. The alarm sounded in an empty corridor for the length of time such alarms sound, then dropped to a flashing banner and waited to be acknowledged. The plant, which did not require a human in order to be safe, isolated the faulted bed and brought the standby online. At 04:30 the morning watch signed on, acknowledged the alarm at 04:33, and wrote in the log: unack’d alarm on handover, no watch signed on overnight — query roster?
The query roster went to Bea Chandran, because nine years earlier Bea Chandran had written the roster service, and on Wray Platform you maintained what you wrote for as long as you both lived.
She read the overnight log twice before she believed it. Then she pulled the transaction history and found 811402 and 811405 sitting side by side, both committed, both green, and felt the specific cold that comes of seeing a thing work perfectly and be wrong.
Her first hour she spent looking for the bug, by which she meant looking in her own code. The guard was where she had left it: before permitting a sign-off, count the watchkeepers who would remain; if fewer than one, refuse. She read it line by line and it was correct. She wrote a test that ran the two sign-offs one after the other, and the second was refused, exactly as designed: COVERAGE AFTER YOUR DEPARTURE: 0. YOU ARE THE WATCH. The screen existed. It had simply never been shown to anyone on the night it was true.
Her second hour she spent in the database documentation, and that was where she found the night of the fifth already written down.
The roster service ran its transactions at an isolation level called repeatable read, which the documentation explained was implemented as snapshot isolation: every transaction sees the database as it stood at the moment the transaction first looked, a private, frozen, internally consistent world, and may act on that world in peace. At commit, the database checks whether you have written over anyone else’s writing. It does not check whether the world you read is still the world there is.
And there, in the section headed anomalies, in a worked example the documentation had carried for decades, were Tomas and Ada. The example called them doctors. Two doctors on call; a rule that at least one must remain; each checks the roster, sees two, and removes only themselves; both commits succeed, because each wrote a different row; the ward is unattended. The anomaly had a name — write skew — and a literature, and a publication date sixty-six years in the past. It was older than the platform. It was older than Chandran. It was older than the programming language the roster service was written in. The night of the fifth had been described, in every particular, by people now dead, as a caution to people not yet born, and she had read that page, or a page like it, at some point in her twenties, and had filed it where one files the weather on other planets.
She went to see the operations chief with the documentation open on her slate, and Vasquez read the doctors example in silence.
“So the fix is what,” he said. “Don’t let two people sign off at once?”
“They didn’t sign off at once. Tomas committed seventeen seconds before Ada. Her transaction just started before his finished. Her snapshot was taken at 02:47:09 and it stayed 02:47:09 while she read her messages.”
“Then the fix is the dialog. You’re holding a transaction open while a human thinks. Make it check-and-commit in one shot, no snapshot older than a heartbeat.”
“That narrows the window,” Chandran said. “It doesn’t close it. Two sign-offs in the same hundred milliseconds fail the same way. I can make the coincidence rarer. The isolation level makes it survivable, and I don’t want it survivable.”
What she wanted was the level above: serialisable. The documentation was plain about the cost. The database would watch for the dangerous pattern itself — each transaction reading what the other was writing — and when it found one it would abort somebody, mid-commit, with an error that was nobody’s fault and had to be retried. Contention, the docs called it. You paid for the single shared world by sometimes being told you couldn’t act in it just yet.
Vasquez signed it off. But before she shipped the change, Chandran did the other thing, the thing she was not asked to do. The invariant was simple — at every instant, at least one on watch — and the commit history went back to the day the service went live. She replayed six years of roster writes in commit order and evaluated the invariant at every change.
The night of the fifth was there, seventy-one minutes long. It was the fourteenth.
Thirteen other times in six years, the platform had stood some minutes with no one on watch — forty seconds once, nine minutes once, the rest in between — and each interval had ended the same way, with someone signing on for an ordinary shift, unaware, the count going from nought to one without anyone learning it had been nought. Nothing had faulted during any of them. The log recorded what had occurred; about what had merely been possible it was silent. She sat with the list for a while. There was no one to report it to who could do anything about the past, and she was aware that the fourteen nights were not evidence of fourteen dangers survived but of one danger, continuous, surfacing wherever two tired people trusted a frozen picture of each other. She filed the list with the incident report and shipped the change.
The first serialisation failure came four months later, at 02:51, and it fell to Ada Solano, because the platform was small and its coincidences were not.
She selected SIGN OFF at the end of a doubled watch, and instead of the question, the console gave her an error she had never seen: COULD NOT SERIALISE ACCESS — CONCURRENT TRANSACTION — RETRYING. The retry took under a second. Then the screen she had also never seen, though it had waited nine years in the code for her:
COVERAGE AFTER YOUR DEPARTURE: 0. YOU ARE THE WATCH.
Somewhere on the platform, seconds ahead of her, someone had already gone. She pinged Iversen, because she knew, without checking, whose row it was.
just bounced off the roster. you signed off?
yes. sorry. thought you were staying
I was going to say the same to you. go to bed, I’ve got it
She cancelled the dialog and stood the watch, one name against ENV WATCH where there had been two. In the transaction log the abort sat where a silent night would once have been. Chandran, who read the aborts the way other people read the weather, counted them as they came in over the following years, and they came at about the rate the fourteen nights implied — every few months, two people, certain of each other, wrong together, each now told so in time to ask. The error had been happening all along. All she had changed was who found out.