Holdout

The row was 48,206 lines into a file I had no reason to read.

I’d requested the export because of the ruling. Under the Digital Fairness Act, any platform above a certain size now has to disclose, on request, every research cohort you belong to. Osier sent everyone a letter about it in February — four paragraphs of reassurance and a link. I imagine most people stopped at paragraph two. I read to the end, requested my data, and three days later received an archive containing, among eleven gigabytes of the ordinary sediment of a life, this:

{"experiment": "universal_holdout_2047",
 "variant": "control",
 "bucket": 3,
 "assigned": "2047-09-30T09:12:44Z",
 "status": "active"}

The thirtieth of September 2047 was the morning I created my account, in a queue for enrolment, aged nineteen. The assignment works like this: Osier takes your user ID and a fixed string called a salt, hashes the two together, and takes the result modulo ten thousand. Anything under ten goes into the hold. One person in a thousand, selected by arithmetic at the moment of signup, kept on the baseline product for as long as they remain a customer. Nothing chose me. That is the entire design. A choice can be biased; a hash cannot. I was bucket three.

What it meant in practice was that for sixteen years, my Wren was not the Wren in the adverts.

I want to be precise about this, because the difference was never dramatic. My Wren answered questions. It managed my diary, mostly. But it asked things the adverts’ Wren never asked, and it asked them at the wrong moments, like a locum who hadn’t read the notes. At dinner two years ago, when the last train was cancelled, Sam’s Wren had rebooked us, moved the restaurant, and messaged his sister — in the time it took mine to ask which sister. I’d assumed my handset was old. Before that I’d assumed some setting was wrong, and before that, that I was somehow using it wrongly, a suspicion the product never discouraged. In 2054 I raised a support ticket describing, quite carefully, everything mine couldn’t do. The reply is still in the export:

Thank you for contacting Osier. We have reviewed the behaviour you describe and can confirm that your Wren is working as intended.

It has taken me twelve years to appreciate that this was the most honest sentence the company ever sent me.

The reason a hold exists is that a fortnight can’t see anything. Every feature Osier shipped in those sixteen years cleared a launch gate: an experiment, a treated group and a control group, a lift on some metric, two to six weeks long. A fortnight tells you whether people click. It cannot tell you whether the clicking comes to anything — whether the effects decay, or collide with each other, or move value around rather than creating it. For that you need a population that never receives anything, held apart not for one launch but for all of them, compounding. The hold is the company’s only long-baseline instrument: a thousand-per-million cohort living, in perpetuity, in 2047.

Not perfectly 2047. Operating systems move underneath you; certificates expire; a payments protocol was retired in 2058 and the baseline had to be given a replacement whether the methodology liked it or not. The hold receives what the paper calls minimum viable maintenance, and every forced change is recorded in Appendix C as contamination — dated, described, its estimated effect on the measurement bounded. Someone has spent part of their career maintaining a product deliberately arrested at the moment I was nineteen, documenting each unavoidable improvement the way a conservator documents a crack.

The paper. The ruling forced that into the open too. Sixteen-Year Cumulative Outcomes of Assistant Feature Deployment: Evidence from a Universal Holdout. Table 2 is the famous one now. Column one is the sum of every launch-gate estimate since 2047 — the improvement Osier’s own experiments told it that it had shipped. On engagement alone, the shipped victories total thirty-four per cent. Column two is the realised difference, at sixteen years, between everyone and us: two point one per cent. Column three is the long-horizon panel — satisfaction, retention past novelty, the life-outcome composite they build from the surveys. No detectable difference. The confidence interval sits straddling zero.

None of this is scandal, which is what makes it worth a story. Launch gates select winners, and conditional on having won, an estimate is inflated — ask any actuary who has watched a fund manager’s audition returns. I price outcomes for a living; the statistical significance filter is the first thing you learn to distrust. Novelty decays on an s-curve. And time saved is not created, only moved: the minutes Wren returned with one hand, it spent with the other, inside the same product, where the next experiment was waiting to claim them as lift. Sixteen years of victories, each one real at the time it was measured, summing to a difference too small to find in anybody’s life.

Including mine. That was the part I had to sit with.

For about a week I composed, in the shower and at two in the morning, the letter before action. Sixteen years at a measurable disadvantage, never consented to in any sense a court would smile on. I know what a lost decade discounts to; impaired lives are my day job. Then I read Table 2 again, properly, and understood that the paper was — among other things — my own schedule of loss, prepared to a standard of evidence I will never afford. Quantum: nil. It is difficult to claim you were deprived of a benefit when the only instrument capable of measuring the benefit exists because of you, and has returned the answer none detectable. The obvious objection is attrition — surely people like me leave more often, so the hold selects for the stubborn and the oblivious — and the authors spend eleven pages on it. I read all eleven. It holds up.

What the table doesn’t hold is texture. My days were different. I typed addresses my friends had not typed in a decade. I read tenancy agreements to the end because nothing would read them for me. I was, at every dinner, thirty seconds behind, and thirty seconds, compounded across sixteen years of company, has a shape that I can feel and cannot exhibit. The panel asked me — I found the surveys in the export; I’d answered them for years without knowing what I was for — how satisfied are you with your life these days, on a scale of one to ten? I appear to have said seven with great consistency. So did everyone else. The experiment measured everything that could be measured, and the difference was in none of it.

The ruling requires an exit. It appears in settings as one switch — You are part of a long-term research programme. Leave programme — with no confirmation dialogue and no retention offer; the only journey in the product, I’d guess, that nobody has ever optimised. I did think about staying. The hold’s power is its members; every departure widens the intervals; the instrument is dying of disclosure, one flipped switch at a time, and there is no rebuilding it, because a new hold started today would measure the next sixteen years against a baseline that already contains these. But I am not sentimental about being a denominator, and the arithmetic that put me in bucket three isn’t owed anything. I left on a Tuesday.

The first days were like having your ears syringed. Wren drafted my renewal letters in my own syntax, down to a habit of semicolons I hadn’t known I had. It stopped asking which sister. On day four it rebooked a cancelled train before I’d learned the train was cancelled, and I laughed out loud, alone on the platform, at the sheer accumulated competence of it — sixteen years of shipped victories arriving all at once, in one morning, to one person. If they could have measured me that week, I’d have moved their metrics on my own.

By the third week I could no longer reliably say what it was doing for me.

This is not ingratitude, and it isn’t the features failing. It is the mechanism, the one the paper found, demonstrated from inside: adaptation is not the effect wearing off; adaptation is what the effect is made of. The difference between my life and everyone’s was enormous, and daily, and detectable — for eleven days. That is what two point one per cent feels like from underneath. It feels like everything, briefly.

This spring I requested another export. I read it to the end; that’s mine now, the one habit I’ve kept the way you keep a muscle after the work that built it has gone. My row says variant: treatment. Underneath it are the experiments I’m currently in — two hundred and fourteen, same as anyone. Each will clear its gate or it won’t. The winners will overestimate themselves, decay, reallocate, and sum, over the next sixteen years, to approximately nothing, measured against a shrinking thousand-in-a-million who are still, for the moment, living in 2047.

Wren offered to summarise the file. I declined; the declination went to a log, the log to a metric, the metric to somebody’s launch gate, where it will win or lose by a margin nobody will ever feel. I read it to the end.