Clean Merge

The question I was instructed to answer ran to eleven words: on what date did the defect first exist in the Software? The capital S was doing contractual work. The Software meant the firmware of the Petrel P3 insulin pump as it stood at Closing — 19 March 2021, the day Meridian Devices plc completed its purchase of Petrel Medical Ltd of Cambridge, and eleven million pounds went into escrow against the founders’ warranty that the code they were selling contained no defect capable of causing harm.

Margaret Ashe, 58, of Whitstable, died in her sleep on the night of 14 January 2024. Her pump had delivered a little under twice her overnight basal insulin. Meridian’s engineers reproduced the fault within a month — roughly one night in every few thousand, under a particular overnight profile, the pump delivered a scheduled segment twice — and made it stop in firmware 7.3 by the oldest method there is: they took the concurrency out. One large lock around the scheduler. The symptom ended. The mechanism was never established. Then the family’s claim settled, quickly and quietly, and the real fight began: Meridian against the founders, over a date.

I was appointed single joint expert under Part 35, which means both sides pay me and neither is pleased.

Two facts were agreed before I started. Firmware 1.0, the clinical-trial build of 2014, did not contain the defect; it had only one thread, and we ran it for two hundred thousand simulated nights without an event. Firmware 7.2, the build in Mrs Ashe’s pump, did contain it. Between those two commits lay 38,214 others, eleven years of work by a company that no longer existed. Binary search does not care what it searches. Test the middle of the interval; whichever half still contains the change in behaviour, keep it; discard the other; repeat. Thirty-eight thousand commits is at most sixteen questions. Anyone who has played twenty questions knows the shape. What they may not know is that the method has a name in the tool itself — bisect — and that at the end it does not give you a culprit. It gives you the first commit at which the behaviour is present. The distinction took me most of a summer to learn to say precisely, and it is the only thing of value in this account.

The test itself lived in an evidence facility in Slough: sixty pump heads on a bench, drawing saline from bags and delivering it into weighed vials, each running a firmware harness that compressed a night into ninety seconds. Compressing time changes a race, so before I trusted the rig I checked its event rate against Meridian’s fleet telemetry. The fleet saw one event in roughly 3,600 profile-nights; the rig, at compression, saw the same within error. That is as much agreement as the world offers.

38,214

The midpoint of history fell in November 2017. Merely building it took nine days: the 2016 toolchain wanted a licence server that had been switched off when Petrel’s office was cleared, and four commits in that region would not compile at all — they depended on a header that existed on someone’s machine in 2016 and nowhere since. Those four were marked skip, with a small prayer that the answer did not live inside them.

Then the rig ran twenty-five thousand nights in a weekend and recorded nothing. Zero is not proof. Against a rate of one in 3,600, a defective build will pass twenty-five thousand clean nights about once in a thousand tries; I recorded the figure, accepted the odds, and marked the commit good. With that one word, half of the history became innocent. Four years of work by fourteen people, eliminated as suspects between Friday and Monday. The thing to understand about halving is how fast it is. The first answer removes an era. The tenth removes an afternoon.

9,553

Most of the engagement was waiting for saline, and while I waited I read the log. A repository is a diary that nobody meant to keep. Fix typo. Revert “fix typo”. Do not touch this, it is load-bearing. A calibration commit at 03:12 on Christmas morning 2018, message: motor cal, again. I came to know the handful of names that recurred, their hours, their tempers, who apologised in commit messages and who never did. None of them knew, typing those lines, that a stranger would one day read them under instruction from a court.

At the eighth question the search converged on an absurdity — a change to the README. A README cannot double a dose, which meant that somewhere upstream a good had lied: a defective build had passed its twenty-five thousand nights clean, as one in a thousand will. I re-ran the last three questions at a hundred thousand nights each and found the liar. It cost nine days, and taught the solicitors more about statistics than they wished to know.

1,194

By late June the interval was 1,194 commits wide and had narrowed onto a single spring: February to June 2021. Closing sat inside it like a blade, and both solicitors now attended in person.

The shape of that spring was this. On 1 March, with due diligence beginning, Petrel froze its integration queue: no branch would be merged to the mainline until the sale completed. The work did not stop — feature branches accumulated, each tested green in its own copy of the world — but the mainline held still for eleven weeks, the way a house is tidied for viewing. On 26 May, sixty-eight days after Closing, Meridian’s release engineer thawed the queue, and a program called autointegrate merged thirty-one waiting branches into the mainline in a little under four hours, in queue order, exactly as designed.

74

Seventy-four commits: in effect, the evening of 26 May. Every one of them post-dated Closing. Mr Osei, for the founders, stopped attending; the ninth halving appeared to have carried his clients’ liability away with it. Ms Pryce, for Meridian, grew correspondingly grim. The questions came quickly now — the builds were minutes apart in history and the toolchain never changed — and the interval fell from seventy-four to thirty-seven to eighteen to nine to four to two.

1

The first bad commit is e4c19a7. Its timestamp is 23:04:51 BST, 26 May 2021. Its author is autointegrate. It is the seventeenth of the thirty-one thaw merges, and when you ask the repository to show it — git show e4c19a7 — it prints the header, the two parent hashes, and nothing else. A merge that resolves without conflict contributes no lines of its own. There was nothing to show.

I verified what the search implied. The first parent — the mainline as it stood that evening — ran one hundred thousand nights clean. The second parent — a branch called np/overnight-profiles, frozen since February — ran one hundred thousand nights clean. The merge of the two produced eleven events in forty thousand nights, the fleet’s own rate. The defect exists in neither parent. It exists in their combination.

Read side by side, the two histories explain themselves. In February 2021, before the freeze, an engineer named Okafor had moved motor control onto its own thread — the new pump head was quieter but needed tighter timing — and put a lock around the pending-delivery total. He updated every place in the firmware that touched the total. There were seven. He missed none, because the eighth did not exist yet: it was on Whelan’s branch. Whelan had forked the mainline in January, when the firmware had one thread, and her overnight-profiles module did what was then correct — at each profile boundary it read the pending total, recomputed the remaining segment, and wrote it back, with no lock, because in the world she forked there was nothing to lock against. Her branch passed every test it was frozen with. His refactor passed every test it was merged with. Both of them were right. They were right about different repositories.

After the merge, both worlds ran at once. Twice a night, at a profile boundary, Whelan’s code read the total and wrote it back some microseconds later; every few minutes, on the other thread, a completed delivery was subtracted from it. When the subtraction landed inside those microseconds it was overwritten — the insulin already in Mrs Ashe’s body restored to the ledger of insulin still owed, and delivered again. Nobody’s line is wrong. Shown either diff on its own, I would have approved it.

Okafor left three weeks after Closing. Whelan stayed on at Meridian. They had sat four desks apart.

0

My report found as fact that the defect first existed at 23:04:51 BST on 26 May 2021, sixty-eight days after Closing; that no natural person authored it; that each of its two constituent changes was authored before Closing; and that each, in the repository in which it was written, was correct. The warranty asked whether a defect existed in the Software as of the Closing Date. The defect did not. Every line of it did. Both parties informed me, through separate letters, that the report supported their position, and five weeks before trial they settled, so no court has ever ruled on what that sentence in the warranty means. The escrow was divided on terms I was not told.

There is a tool in the repository called blame. It is not a metaphor; it is the command’s name. Run over the scheduler as it stood the night Margaret Ashe died, it names an author for every line — Okafor, Whelan, fourteen others, back to 2013 — and every line it names is correct. I put a sentence to that effect at the end of my report, outside the questions I had been asked. It was the only sentence to which both parties objected.