Exposure

A test question dies the moment you ask it. Not instantly — death here has a half-life — but to administer an item is to expose it, and anything exposed is eventually copied, and anything copied ends up in the corpus. After that the question no longer measures whether a model can think. It measures whether a model can remember, and models can always remember. In the trade we say the item is spent.

I compose items. Before this I was a number theorist, with a lectureship, one modest theorem carrying my name jointly, and the standard ration of unfinished ideas. The consortium approached me eleven years ago, after the last of the public benchmarks died. Marek made the pitch over lunch, and to his credit he made it without euphemism: we will pay you well for original mathematics; it must be new, it must be hard, and it must close — a finite answer a machine can check without opinion. It will be put to a model at most once, in one season, and then sealed. You will never publish it. Nobody will ever read it. That is not a side effect of the arrangement. That is the product.

The public benchmarks died of exposure, all of them, and it is worth being precise about how, because people imagine espionage and the truth is more like weather. A benchmark is published so that results can be compared. Being published, it is crawled. The training pipelines run decontamination filters — they search the corpus for overlapping strings and cut what they find — and the filters work, in the sense that a verbatim copy rarely survives. But a paraphrase survives. A translation into Polish and back survives. A student who half-remembers a question and posts it on a forum, wrongly, with the numbers changed, survives — and a model that has seen ten mangled paraphrases of a question has seen the answer. Nobody has to cheat. The corpus is the sea, and everything written eventually drains into it.

Our contracts carry a canary: a long random string, unique to each document, meaning nothing. If a model can ever be coaxed into completing that string, we know the contract itself was swallowed, and we know our plumbing failed. In eleven years the canary has surfaced twice. Both times the leak was traced not to theft but to convenience — a scanned page in a shared drive, a phrase quoted in a calendar invite. The sea is patient.

So the consortium treats exposure as an actuarial problem. An item’s life is measured in minds. Each person who reads it is a channel with a half-life: the postdocs who pilot it to calibrate difficulty, behind a second grade of nondisclosure; the graders; the author. The actuaries keep a number for each of us — the probability, year on year, that we say too much at a conference dinner. I am told mine is priced in.

The work itself is honest work, and most of it is dead. I want to be exact about this, because it bears on what happened. A good item is adversarial in a narrow way: new enough that nothing adjacent to it exists anywhere in print, hard enough to bite, and still closed — it must end in an integer, or a polynomial, something a script can mark without judgement, because a disputed grade is worth nothing. Producing such objects is a craft, and the craft’s usual output is workmanlike: hard, checkable, dead inside. For nine years I delivered forty items of that kind and slept well.

The forty-first was different. I was three days into it when I noticed that the scaffolding I had built to make the problem difficult was more interesting than the problem. There was a bridge in it — a way of carrying a structure from one part of mathematics across to another where, as far as I know, nobody had thought to look for it. I put the item aside and followed the bridge for three weeks. They were the best three weeks of my working life, and I had spent twenty years hoping for them, and when they came, they came on commission, on a deadline, for a buyer.

I am permitted to tell you how it felt and forbidden to tell you what it was. Even the area is exposure: an area and a year would let a determined lab synthesise ten thousand near misses and train on the neighbourhood of my idea, and then the idea would go the way of the benchmarks — not stolen, drained. I have tried metaphors and each one leaks a little of the shape. So: it was beautiful. The review board passed that sentence because it contains no information. That is also why it is the only sentence I have.

I understood what I was holding. Written up, it was a paper — possibly a career; the bridge would not stop where I had stopped. It was also, within a fortnight of publication, part of the crawl. I rang Marek and said I had something better than an item and I didn’t know what to do, and he was silent for a moment and then said: don’t tell me anything. Not the area. Nothing in email, even to me. Bring it in the usual way, or publish it — but decide before you talk to anyone, because talking is deciding.

I brought it in the usual way.

The season ran in November, in a building I am contractually obliged to describe only as being in the Midlands. Nothing with a radio inside; the weights arrive by courier on encrypted drives, one lab at a time, and the only thing that leaves is paper. Authors attend their own administrations because the grading script only checks the final answer, and an answer without a derivation is a coin toss until someone reads the transcript. That is the rule that put me in the room. Three models were entered that season. I watched my item put to each in turn, which is to say I watched a progress bar, which is to say I watched nothing, for most of two days.

The first model spent its budget manufacturing lemmas and discarding them, closed the gap with an assertion, and reported the wrong integer with high confidence. The second found the right integer by what the transcript showed to be an enormous guided search, and could not produce the derivation when asked; under our rules that scores as a fail, and I signed the sheet without hesitation and with a relief I did not examine at the time.

The third returned the right integer in forty minutes, and attached a proof.

It was not my proof. I want to record that plainly, because it is the fact I keep returning to. It did not cross my bridge. It went another way — not shorter, stranger — through territory I would have said was unrelated, and I spent two further days trying to break it and could not. There was a step near the middle that I would call inspired if the word attached to anything here. I signed the sheet. Under “notes for the record” I wrote: correct, and independent of the intended method. Fourteen words were all the room I had. That may have been a mercy.

My theorem has two proofs now. I have read both. Nobody else has read either.

The item is sealed, not destroyed — the consortium must be able to prove, in a dispute, what was asked. Somewhere in the archive my three weeks sit in a numbered folder beside a machine’s forty minutes, and the folder will not open in my lifetime, because opening it would tell the labs what the season measured, and the neighbourhood would be drained by spring. The public report came out in March. Aggregated across all items and all entrants, it found “evidence of capability consistent with original research-level mathematics in at least one domain”. My three weeks are the words at least one.

Marek once walked me through the arithmetic, apologetically, over the same lunch table as the first pitch. An administration yields information about the model: a few bits, at most, per item — pass, fail, and a little colour from the transcript. The depth of the item does not enter into the yield. Depth is what it costs to make the question hard; it is not what the question tells you. The theorem underneath is not part of the measurement, he said. It is the fuel.

A graduate student somewhere will find the bridge again. Mathematics is like the sea in that respect too: it does not care who reaches a place first, only that the place is there. When the paper appears I will read it in a public library like anyone else, and I will learn, at last, where the bridge goes, and the name it carries will not be mine, and none of this will be provable or worth proving. I have rehearsed being calm about it. Some days the rehearsal takes.

The new model generation is expected in the spring, and the bank is low; demand for items has never been better. The stipend cleared on a Friday and I have started my forty-second. It is workmanlike — hard, checkable, dead inside. I intend to keep it that way.