Welcome to the VBNN Digital Library
Unlock a Vast Knowledge Ecosystem
Featuring over 30,000 books, academic papers, illustrations, and expert insights—continuously updated to support your research and professional growth.
Welcome to our library!
Here, you will find an exclusive collection created 100% by our own faculty, meaning you will not find these resources anywhere else. Over the last 20 years, our team has written much more than what is currently online, and we are actively working to upload our complete back catalog. We update our platform regularly, so be sure to check back from time to time. If you ever need help finding a specific resource, you can always contact us!
Maximize Your Access
Log in to instantly view and download tailored resources directly aligned with your specific program and curriculum.
Ready to begin? Sign in above to explore your personalized dashboard.
Please note: Login is only possible using your institutional email address; otherwise, the system will not recognize your account.
VBNN Library AI
Introducing our fully integrated Library AI. Designed to support your research, you may submit inquiries in any language and receive precise, evidence-based responses drawn exclusively from our published scholarly articles and textbooks.
Search...
Latest Publications:
Search this site
Results found for empty search
- Spacecraft Flight Software (Fault Tolerance and Radiation Hardening)
Download the Book (PDF): Introduction On the afternoon of 4 July 1997, a small spacecraft bounced to a stop inside a cluster of airbags in Ares Vallis on Mars. Mars Pathfinder was a triumph of low-cost engineering, and for its first few days it returned pictures and weather data exactly as planned. Then its computer began to reset itself. Not once, but repeatedly, each reset wiping the day's work and sending the lander back through its start-up sequence. Nothing was physically broken. No cosmic ray had struck a critical transistor. The fault lay in the interaction of three software tasks sharing a resource, a timing pattern that had appeared only rarely in ground testing and that the flight team had not chased down before launch. What saved the mission was a set of decisions made years earlier by people who had assumed that something, somewhere, would go wrong: a watchdog timer that noticed when the system stopped making progress, a reset path that brought the lander back to a safe and communicative state, a debugging trace that recorded what had happened, and an onboard command interpreter that let engineers on Earth change a single configuration flag in the running software from a hundred and ninety million kilometres away. That episode contains, in miniature, the subject of this book. Flight software for interplanetary spacecraft is written for a machine that no one will ever touch again, operating in an environment that actively corrupts its memory and logic, at a distance that makes real-time help from its creators physically impossible. The engineering discipline that has grown up around those three facts is one of the most demanding in computing. It is also one of the least widely understood. Most programmers spend their careers on systems where a crash means a restart, where hardware can be replaced, where a patch can be deployed by the end of the day, and where the probability of a bit spontaneously changing value is low enough to ignore. Flight software engineers enjoy none of these luxuries, and the techniques they have developed in response repay study by anyone who builds systems that must keep working when the ordinary assumptions fail. The argument of this book The controlling idea of what follows is simple to state and demanding to practise. Deep-space flight software does not survive by preventing faults. Radiation guarantees that faults will occur, complexity guarantees that some of them will be design errors nobody anticipated, and distance guarantees that no human can intervene quickly enough to stop a fast-moving failure. Spacecraft survive because they are designed so that every layer, from a single memory cell up to the mission plan, assumes the layer beneath it may be lying, and can contain, detect and recover from that lie on its own. Redundancy is the most visible expression of this philosophy, but redundancy is only ever as good as the independence of the things it replicates. Two identical computers running identical software will fail identically when the software meets a case its authors never imagined, and a great deal of space history consists of learning that lesson again. This book develops that argument through nine chapters, each doing a distinct job. The first establishes why distance changes everything: the speed of light sets a hard floor under how quickly Earth can respond, and that floor is often far higher than the time a spacecraft has before a fault becomes fatal. The second explains the physics of radiation damage to electronics, from the slow accumulation of total ionising dose to the instant havoc of a single heavy ion passing through a memory cell, because no mitigation makes sense without an accurate picture of what is being mitigated. The third covers the hardware response: radiation-hardened processes, hardened circuit designs, shielding, and the shift now under way from decades-old hardened processors towards faster parts and commercial chips that accept more upsets and handle them in software. The fourth chapter treats triple modular redundancy and the mathematics of voting, from John von Neumann's 1956 lectures to the Saturn V's guidance computer and the Space Shuttle's quadruple-redundant computers, including the uncomfortable fact that a redundant system can be less reliable than a simple one under the wrong conditions. The fifth descends into memory and state: error-correcting codes, scrubbing, and the software-implemented techniques that duplicate instructions and check control flow when the hardware cannot be trusted. The sixth confronts common-mode failure, the way in which redundancy is defeated by shared design, shared assumptions and shared time, using the loss of Ariane 5's first flight and several more recent cases. The seventh turns to software architecture, showing how real-time operating systems, partitioning, watchdogs and disciplined coding standards create a structure in which faults stay where they start. The eighth describes fault protection and safe mode, the spacecraft's reflexes, and the hard engineering problem of verifying software whose whole purpose is to run when things have already gone wrong. The ninth extends the argument from survival to autonomy: landing, driving and making decisions without Earth, and the new failure modes that appear when a spacecraft's software is trusted to interpret the world on its own. Who this book is for This is a book for practitioners and serious students: software engineers who want to understand what changes when the computer is in orbit around another planet, systems engineers who need a working grasp of the software side of reliability, and technically literate readers who have wondered how a rover that cannot be rebooted by hand keeps working for more than a decade. It assumes familiarity with programming and basic computer architecture. It does not assume any background in physics, radiation effects or space operations; those are explained where they are needed. A few short fragments of C appear where a piece of code says something more precisely than prose can, but the argument is carried by the text. The emphasis throughout is on mechanisms and on real events. The history of spaceflight is unusually well documented when things go wrong, because failure review boards publish their findings and engineers present their lessons at conferences. Those reports are the closest thing the field has to experimental data, and they are used here wherever possible: the Pathfinder resets, the Spirit rover's flash memory crisis eighteen days after landing, the loss of Mars Global Surveyor after a faulty memory upload, the integer overflow that silenced Deep Impact, the saturated sensor that convinced Schiaparelli it was below the Martian surface, the filter that taught a lunar lander to disbelieve a correct altimeter, and the failed memory chip that Voyager 1's engineers worked around in 2024 by relocating code inside a computer built in the early 1970s. Each of these illustrates a principle better than any abstract statement could. A note on what the field looks like now Spacecraft computing is changing faster in the mid-2020s than at any time since the 1990s. For two decades, the workhorse of American deep-space missions has been the RAD750, a radiation-hardened derivative of a PowerPC processor that first appeared in desktop computers in 1997. It flies on the Curiosity and Perseverance rovers, on Juno at Jupiter, and on the James Webb Space Telescope. It is extraordinarily robust and, by modern standards, extraordinarily slow. At the same time, the Ingenuity helicopter flew seventy-two times on Mars using a smartphone-class processor running Linux, a demonstration that commercial hardware, carefully managed, can survive and perform in deep space for years. NASA's High Performance Spaceflight Computing programme, awarded to Microchip Technology in 2022, aims at a processor offering roughly a hundred times the computational capacity of current space computers, built on the open RISC-V architecture. Europe's equivalent path runs through Cobham Gaisler's fault-tolerant LEON processors. Meanwhile, flight software itself is moving from bespoke codebases towards reusable open frameworks such as NASA's core Flight System and JPL's F Prime. These shifts do not overturn the principles in this book; they sharpen them. A faster, less hardened processor experiences more upsets, so software must detect and correct more of them. A more autonomous spacecraft makes more decisions without a human check, so the logic it uses to judge its own sensors becomes a new kind of single point of failure. A reusable framework multiplies the reach of each design flaw across every mission that adopts it. The questions that flight software engineers have always asked, namely what can fail, how will we know, what will we do about it, and what happens if our response is itself wrong, are more pressing in 2026 than they were when Pathfinder landed. How to read it The chapters build on one another, and the first three lay foundations that later ones rely on, so a first reading is best done in order. Readers already comfortable with radiation physics can move quickly through the second chapter; those with a background in dependable computing will find the fourth and fifth familiar in outline but may find the spaceflight particulars useful. The conclusion does not summarise. It draws out what the evidence of the preceding chapters implies for how flight software should be built from here, and where the hard questions remain open. Notes identify the sources behind specific claims, and a short list of further reading points to the books and papers that reward deeper study. Underneath all the technique lies an attitude worth naming at the outset. The engineers who build this software are, by professional necessity, pessimists about components and optimists about systems. They assume every transistor may flip, every sensor may lie, every line of code may harbour a defect, and every operator may send a wrong command. Then they build machines that, despite all of this, keep working for decades at the edge of the solar system. Understanding how they do it is the purpose of the pages that follow. CHAPTER 1 The Tyranny of Light Time Every engineering discipline has a constraint that shapes everything else. For bridges it is gravity; for aircraft it is weight; for interplanetary flight software it is the speed of light. Radio signals travel at about 300,000 kilometres per second, which is fast enough that on Earth the delay is imperceptible and slow enough that, across the solar system, it becomes the central fact of operations. A spacecraft at Mars is never less than about three light-minutes from Earth and can be more than twenty-two. A spacecraft at Jupiter is between roughly half an hour and three-quarters of an hour away. Voyager 1, still operating in interstellar space, is approaching a full light-day from Earth in late 2026. Whatever happens on board, the people who built the spacecraft learn of it only after that delay, and any instruction they send takes the same time again to arrive. This chapter establishes why that delay is not merely an inconvenience but a design driver, and why it forces a particular kind of software. The argument runs as follows. Faults on a spacecraft develop on timescales set by physics: how fast a battery drains, how fast an unpowered tank freezes, how fast a spinning spacecraft's antenna sweeps away from Earth, how fast a lander falls. Earth's ability to respond is set by light time plus the human time to notice, understand, decide and command. Whenever the first timescale is shorter than the second, the spacecraft must handle the problem itself or not at all. For deep-space missions, that condition holds for almost every fault that matters. The speed of light as a design constraint The distance between Earth and Mars varies because both planets orbit the Sun at different speeds. At the most favourable oppositions, Mars comes within about 55 to 60 million kilometres of Earth, and a signal crosses that gap in a little over three minutes. When Mars is on the far side of the Sun, the distance grows to around 400 million kilometres and the one-way light time to about twenty-two minutes. A round trip, the minimum time between a spacecraft reporting a problem and receiving a response, therefore ranges from roughly six minutes to well over forty, before any human has done anything at all. The bare light time understates the real delay by a large margin. A deep-space spacecraft is not in continuous contact with Earth. NASA's Deep Space Network consists of three complexes, near Goldstone in California, near Madrid in Spain and near Canberra in Australia, spaced roughly a third of the way around the planet from one another so that any distant spacecraft is above the horizon of at least one of them. But the network serves dozens of missions, each of which receives scheduled passes of a few hours. A Mars rover, meanwhile, usually sends most of its data through relay orbiters such as Mars Reconnaissance Orbiter, Mars Odyssey, MAVEN and the European Trace Gas Orbiter, and each of those orbiters passes over a given spot on the surface only briefly, a handful of times a day. A rover that develops a problem shortly after one relay pass may not be able to report it until the next. The engineers who receive the report then need time to interpret telemetry, reproduce the situation on a testbed, agree on a response and build and verify the commands. For a routine anomaly on a Mars rover, the realistic loop from fault to corrective action is measured in days, not minutes. Twice in every Mars synodic period of about twenty-six months, the problem worsens further. When Mars passes behind the Sun as seen from Earth, an event called solar conjunction, the solar corona lies between the two planets and its charged plasma scrambles radio signals. Mission teams stop sending commands for about two weeks, because a corrupted command could do real harm, and they upload a plan in advance that keeps every spacecraft at Mars busy and safe without help. The most recent conjunction, in January 2026, was handled in this way, as were all its predecessors. For those two weeks, the rovers and orbiters at Mars are, operationally, on their own. The further out one goes, the starker this becomes. The Cassini spacecraft at Saturn operated with one-way light times of well over an hour. Voyager's engineers, sending commands to a spacecraft nearly a light-day away, wait close to two days to see the result of any instruction. In July 2023, an incorrect command tilted Voyager 2's high-gain antenna about two degrees away from Earth, enough to cut off normal communication. Engineers did not have to rely on a heroic rescue alone: the spacecraft's software already included a routine that periodically re-pointed the antenna towards Earth using stored ephemeris knowledge, and that routine was scheduled to run in October of that year. In the event, the team managed to "shout" a corrective command to the spacecraft from the large antenna at Canberra in early August, and normal communication resumed. But the design assumption behind the onboard re-pointing routine is the point: the people who built Voyager in the 1970s knew that a spacecraft that far away would one day lose its link through someone's mistake, and gave it a way to find Earth again without being told. Fault timescales versus response timescales The useful way to think about light time is not as a number but as a comparison. For each credible fault, an engineer can ask two questions. How long does the spacecraft have before the fault becomes unrecoverable? And how long would Earth take to respond? If the first number is comfortably larger than the second, the fault can be left to the ground, and the spacecraft needs only to stay stable and report. If the first number is smaller, the spacecraft must act autonomously, and the software that does so must be correct without the benefit of a human checking its work. Table 1 sets out some representative timescales on each side of that comparison. Table 1. Representative timescales that shape deep-space fault response. Situation Typical timescale What it implies Earth to Mars, one-way light time at close approach about 3 minutes Six-minute minimum round trip Earth to Mars, one-way light time near conjunction about 22 minutes Round trip over 40 minutes Mars entry, descent and landing, atmosphere to surface about 7 minutes Over before Earth hears it began Earth to Jupiter, one-way light time about 33 to 54 minutes Hours-long command loop Earth to Voyager 1, one-way light time in 2026 approaching 24 hours Two-day command loop Mars solar conjunction command pause about 2 weeks, every 26 months Full autonomy for the period Source: NASA mission and Deep Space Network public documentation; light times computed from orbital distances. The most dramatic illustration is entry, descent and landing on Mars, which engineers at the Jet Propulsion Laboratory have long described as "seven minutes of terror". A lander strikes the top of the Martian atmosphere at several kilometres per second and must, in about seven minutes, shed that speed with a heat shield, deploy a supersonic parachute, jettison the shield, find the ground with radar, separate from the parachute, fire its descent engines and set down. When Perseverance landed in February 2021, the one-way light time was more than eleven minutes. By the time the first signal announcing atmospheric entry reached Earth, the rover had been sitting on the surface for several minutes, either intact or not. Every decision during landing, including the choice of where to set down based on what the descending vehicle's cameras could see, was made by onboard software. Landing is the extreme case, but the same logic governs mundane faults. Consider a spacecraft whose attitude control system develops a fault that leaves it slowly tumbling. Its solar arrays drift away from the Sun and its batteries begin to discharge. Depending on the spacecraft, the batteries might last hours. Its high-gain antenna sweeps away from Earth within minutes, so the very telemetry that would tell engineers what happened stops arriving. If the spacecraft waits for instructions, it will be dead before anyone on Earth has finished reading the last packets. It must recognise the fault, stop the tumble, point its arrays at the Sun, establish a low-rate link through an antenna that does not need precise pointing, and wait. This sequence is what safe mode means, and it is the subject of a later chapter. The point here is that safe mode exists because of light time. Thermal faults behave similarly. Propellant lines and tanks on many spacecraft must be kept above freezing by heaters; if a heater fails or is switched off by a software error, hydrazine can freeze in hours, and frozen propellant can rupture lines and end the mission. Power faults can be even faster. A short circuit on a power bus can drag down voltage in milliseconds, and the response, isolating the faulted load, has to happen in hardware and low-level software before any processor-level reasoning is possible. There is a structure here that will recur throughout this book. Faults that unfold in microseconds to milliseconds are handled by hardware: current limiters, error-correcting memory, hardware voting. Faults that unfold in milliseconds to seconds are handled by low-level software: watchdogs, retries, task restarts. Faults that unfold in seconds to hours are handled by the spacecraft's fault protection software, which can reconfigure hardware and change modes. Only faults slower than the communication loop, or faults that leave the spacecraft stable and waiting, can be handed to people on Earth. Light time sets the boundary between the last two tiers, and for deep-space missions it pushes that boundary out to hours or days. What distance does to the operations model Light time does more than force autonomy in emergencies. It reshapes how a spacecraft is operated every day, and that in turn shapes the software. A spacecraft in low Earth orbit can, in principle, be operated interactively: a controller sends a command, sees the result seconds later, and decides what to do next. No deep-space mission works that way. Instead, the ground team plans a period of activity in advance, encodes it as a sequence of time-tagged or event-triggered commands, verifies it on simulators and testbeds, and uploads it as a block. The spacecraft executes the sequence on its own, and the team learns how it went hours or days later. For Mars rovers, this planning cycle typically runs on a daily rhythm: the team reviews what the rover did, plans the next sol's activities and uploads them, and the rover carries them out while the team sleeps. This model has a profound consequence for software design. Because the ground cannot watch each command execute, every command sequence must be safe to run unsupervised, including in the case that something goes wrong halfway through. If a rover is told to drive twenty metres and then deploy its robotic arm, and the drive stops early because the wheels slip, the software must not deploy the arm against a rock it was not expecting. Sequences therefore contain conditional logic, and the flight software contains layers of checks that refuse to execute commands whose preconditions are not met. Over decades, this has produced flight software that is better described as a supervised autonomous system than as a remotely controlled machine, even in normal operation. Distance also makes every command upload a risk in itself. A wrong command sent to a nearby satellite can usually be corrected quickly; a wrong command sent to a spacecraft at Mars may do its damage long before anyone notices. Several of the most consequential failures in planetary exploration began not with a cosmic ray or a design flaw but with an upload. The Soviet Phobos 1 spacecraft was lost in 1988 after a command sent from the ground, prepared with an error, deactivated its attitude control thrusters; the spacecraft could no longer keep its solar panels on the Sun and its batteries ran down. NASA's Mars Global Surveyor, after nearly a decade of successful operation, was lost in November 2006 through a chain that began with a parameter update to its flight software months earlier, a case examined in detail later in this book. In both instances, the spacecraft carried out instructions faithfully. Good flight software defends not only against the environment and its own defects but against its operators, by validating commands, refusing dangerous combinations and keeping a path back to safety that no single command can remove. From commanded to self-reliant Taken together, these constraints define the requirements that make interplanetary flight software distinctive. It must keep the spacecraft alive without help for at least as long as Earth needs to notice and respond to a problem, which, allowing for missed passes and conjunctions, may be weeks. It must carry out long sequences of planned activity without supervision, and handle the routine surprises that arise during them. It must perform time-critical operations, such as landing, orbit insertion and flybys, entirely on its own, often with no second chance. And it must accept new software and parameters from Earth over a slow, delayed and occasionally error-prone link, without letting a bad upload kill it. None of those requirements is unique to spaceflight in isolation. Industrial controllers must run unattended; aircraft systems must handle time-critical events; medical devices must protect against operator error. What is distinctive is the combination of all of them with two further facts. The first is that the hardware cannot be repaired or replaced, so every component that fails must be worked around for the rest of the mission. The second, examined in the next chapter, is that the environment itself attacks the electronics, so that faults are not rare events to be guarded against in principle but daily occurrences to be handled in practice. It is worth noticing how early engineers grasped all of this. The Apollo Guidance Computer, designed at the MIT Instrumentation Laboratory in the 1960s, flew in a spacecraft only one and a half light-seconds from Earth, close enough for human controllers to advise on decisions. Yet when its software famously raised 1201 and 1202 program alarms during the descent of Apollo 11 in July 1969, the computer did not crash. Its executive, overloaded by unexpected demands for processing time, shed lower-priority work and restarted the essential guidance tasks from protected checkpoints, a design known as restart protection. Controllers in Houston had time to confirm that landing could continue. A deep-space mission would have had no such confirmation; the software's own recovery would have been the whole of the response. That the recovery worked is a testament to a design philosophy that later spacecraft adopted wholesale: assume overload and error are possible, prioritise what matters most, and arrange for the system to recover itself. The rest of this book can be read as an elaboration of that philosophy under harsher conditions. The environment between planets is far more hostile to electronics than the space between Earth and the Moon; the distances are hundreds to thousands of times greater; missions last not days but decades. The defences have grown accordingly, from bits in memory to the logic that decides whether a sensor should be believed. But the starting point never changes. A spacecraft beyond the Moon cannot wait for help, and its software must be designed on the assumption that, when the moment comes, no help will arrive in time. CHAPTER 2 Particles Through Silicon A computer on Earth sits beneath a natural radiation shield about ten metres of water thick: the atmosphere, together with the planet's magnetic field, which deflects most charged particles arriving from space. Outside that protection, electronics are bathed in a steady flux of energetic particles, punctuated by storms of far greater intensity. Some of those particles are stopped harmlessly by a spacecraft's walls. Others pass through the walls, through circuit boards, and through the microscopic transistors that store and process information, leaving behind a trail of electrical charge or atomic disruption. The effects range from the gradual ageing of a chip over years to the instantaneous corruption of a single bit, and occasionally to the destruction of a device in a fraction of a second. This chapter explains those effects, because every technique in the rest of the book is a response to them. A designer who knows only that "radiation causes bit flips" will choose the wrong defences. A designer who understands that some effects accumulate while others are instantaneous, that some are transient and others persist until power is cycled, and that some damage hardware permanently while others merely corrupt data, can choose the right mitigation for each and recognise when a mitigation that works for one effect is useless against another. Where the particles come from Three sources dominate the radiation environment that matters for spacecraft electronics, and their relative importance depends on where a spacecraft flies. The first is galactic cosmic rays: atomic nuclei, stripped of their electrons, accelerated to enormous energies by supernova shocks and other processes far outside the solar system. Most are protons, a substantial minority are helium nuclei, and around one per cent are heavier nuclei, from carbon and oxygen up to iron and beyond. The flux is low, a few particles per square centimetre per second, but the energies are so high, often hundreds of millions to billions of electron-volts per nucleon, that no practical thickness of spacecraft shielding stops them. The heavy nuclei matter disproportionately for electronics because, as explained below, the damage a particle can do in a single strike rises steeply with its charge. The cosmic ray flux in the inner solar system varies with the eleven-year solar cycle: when the Sun is active, its magnetic field extends further and deflects more cosmic rays, so the flux is lowest at solar maximum and highest at solar minimum. The second source is the Sun itself. Solar flares and coronal mass ejections periodically accelerate vast numbers of protons, with some heavier ions, into interplanetary space. These solar particle events can raise the flux of energetic protons by many orders of magnitude for hours to days. They are unpredictable in detail, more frequent near solar maximum, and capable of overwhelming defences designed for quiet conditions. The large storms of October and November 2003, sometimes called the Halloween storms, caused anomalies on many spacecraft; at Mars, the MARIE radiation instrument aboard Mars Odyssey stopped working after the event of 28 October and never recovered. Mission planners therefore size radiation defences not only for average conditions but for a worst-case event of a given probability over the mission's life. The third source is trapped radiation. Planetary magnetic fields capture charged particles in belts. Earth's Van Allen belts hold energetic protons and electrons, and a region over the South Atlantic where the inner belt dips close to the surface, the South Atlantic Anomaly, is notorious among operators of low-orbit satellites as the place where upsets cluster. Interplanetary missions cross Earth's belts quickly and then leave them behind. Mars, which lacks a global magnetic field, has no belts at all, so Mars orbiters and landers face mainly galactic cosmic rays and solar events. Jupiter is the opposite extreme. Its magnetic field is immensely strong and traps electrons and ions at energies and densities far beyond anything near Earth; a spacecraft orbiting Jupiter can accumulate in weeks the radiation dose that an Earth-orbiting satellite would take years to receive. Juno and Europa Clipper were designed around this environment, as later chapters describe. Cumulative damage: total dose and displacement Radiation affects electronics in two broad ways: gradually, through damage that accumulates over the mission, and suddenly, through the effects of individual particles. The gradual effects come first, because they set the lifetime of hardware. Total ionising dose is the accumulated energy deposited by ionising radiation in a material, measured in rads or grays. In a metal-oxide-semiconductor transistor, ionising radiation passing through the thin insulating oxide frees electrons and leaves behind positive charges, some of which become trapped in the oxide or at its boundary with the silicon. Over time, this trapped charge shifts the voltage at which transistors switch on and increases leakage current, particularly through the thicker isolation oxides that separate transistors from one another. A device that has absorbed enough dose begins to draw more current, run more slowly and eventually fail outright. Total dose tolerance is therefore quoted as a threshold: a part rated to 100 kilorads is expected to remain within specification up to that cumulative dose. Commercial parts often fail at a few kilorads to a few tens of kilorads; radiation-hardened parts are rated to hundreds of kilorads or more. Displacement damage is a different mechanism. Energetic protons and neutrons can knock silicon atoms out of their positions in the crystal lattice, creating defects that trap charge carriers. This matters less for digital logic than for devices whose function depends on carrier lifetime, such as solar cells, optical sensors and some analogue and bipolar circuits. Camera detectors on long missions gradually develop "hot pixels" for this reason. For flight software, cumulative damage is mostly a background concern, addressed by choosing parts with adequate ratings and shielding them sufficiently for the mission's predicted dose. But it has software consequences. Parts degrade gracefully before they fail, and a spacecraft that monitors supply currents, timing margins and error rates can sometimes detect the onset of degradation and switch to a redundant unit before failure. Long-lived missions also have to plan for the possibility that, late in life, hardware will behave in ways that it never did in testing. Single-event effects The effects that most directly shape flight software are those caused by individual particles. A single energetic ion passing through a semiconductor ionises the material along its path, freeing a dense column of electron-hole pairs. If that column passes near a sensitive junction in a transistor, the electric field there sweeps the charge out of the column and into the circuit node, producing a brief pulse of current. What happens next depends on the circuit. The physics is captured by two quantities. Linear energy transfer, or LET, measures how much energy a particle deposits per unit length of material, conventionally in units of MeV square centimetres per milligram. In silicon, a particle with an LET of one in those units deposits roughly ten femtocoulombs of charge per micrometre of path. LET rises with the square of a particle's charge, which is why heavy nuclei such as iron, though rare among cosmic rays, are responsible for most single-event effects in hardened parts. Critical charge is the amount of charge that must be collected at a node to change the state of the circuit. If the deposited charge collected at a sensitive node exceeds the critical charge, an upset occurs. Engineers characterise a device's susceptibility by bombarding it with ion beams of known LET at accelerator facilities and counting events. The result is a cross-section curve: below a threshold LET the device does not upset at all; above it, the probability of upset per incident particle rises and then saturates at a value related to the total sensitive area of the device. Combining that curve with a model of the mission's particle environment, using tools such as the CREME96 code developed at the US Naval Research Laboratory and its successors, yields a predicted upset rate in events per bit per day. For a hardened processor in interplanetary space, such rates are low but never zero; for a dense commercial memory, they can amount to many upsets per day across the whole device. Single-event effects come in several varieties, and the differences between them determine which defences work. Table 2 compares the main ones alongside the cumulative effects described above. Table 2. The principal radiation effects on spacecraft electronics. Effect Mechanism Persistence Typical defence Total ionising dose Charge trapped in oxides over time Cumulative, permanent Hardened process, shielding Displacement damage Lattice atoms knocked out of place Cumulative, permanent Part selection, shielding Single-event upset Ion charge flips a stored bit Until overwritten Error-correcting codes, scrubbing, voting Multiple-bit upset One ion flips several nearby bits Until overwritten Interleaving, stronger codes Single-event transient Ion causes a pulse in logic Nanoseconds, unless latched Filtering, temporal redundancy Single-event functional interrupt Ion upsets control logic of a device Until reset Watchdogs, device reset Single-event latch-up Ion triggers parasitic thyristor Until power removed; can destroy Current limiting, power cycling, hardened process Burnout and gate rupture Ion triggers breakdown in power devices Permanent Derating, part selection A single-event upset is the archetype: a particle strike flips the state of a memory cell or a flip-flop from zero to one or vice versa. Nothing is damaged; the cell works perfectly afterwards and will store the next value written to it correctly. But until that happens, it holds the wrong value. If the cell holds data, the data are corrupt; if it holds an instruction, the program is corrupt; if it holds a pointer, a return address or a piece of the processor's internal state, the consequences can be anything from nothing at all to a crash. The first documented upsets in space were reported by Binder, Smith and Holman in 1975, who attributed anomalous state changes in flip-flops aboard a communications satellite to galactic cosmic rays. On the ground, the issue arrived through a different door. In the late 1970s, Timothy May and Murray Woods at Intel traced unexplained errors in dynamic memory chips to alpha particles emitted by trace radioactive contamination in the packaging, and in 1979 James Ziegler and William Lanford showed that cosmic-ray secondaries, mostly neutrons, at sea level could upset memory cells too. The phenomenon has been a standing concern of the semiconductor industry ever since, and today's terrestrial servers use error-correcting memory partly for this reason. In space, where the particle flux is far higher and includes heavy ions that never reach the ground, it is a certainty. Multiple-bit upsets occur when a single particle, striking at a glancing angle or depositing charge that spreads over a region, flips two or more adjacent cells. As memory cells have shrunk, more of them fit within the footprint of a single strike, and multiple-bit upsets have grown from a curiosity to a significant fraction of all events in modern dense memories. This matters because the most common error-correcting codes, examined in a later chapter, correct one bad bit in a word and detect two; if a single strike flips two bits in the same word, correction fails. Memory designers counter this by interleaving, arranging physically adjacent cells to belong to different logical words so that a cluster of upsets is spread across several words, each with only one error. Single-event transients are the logic equivalent of upsets. A strike in combinational logic produces a brief voltage pulse that propagates through gates. If it reaches a flip-flop at the moment the clock captures its input, it is latched as an error; otherwise it vanishes. Because the probability of capture rises with clock frequency, transients have become more important as processors have got faster. Defences include filtering, which suppresses pulses shorter than a set width, and temporal sampling, which captures a signal at several slightly different times and votes. Single-event functional interrupts are, from a software point of view, among the most troublesome effects. Complex devices such as processors, memory controllers, flash chips and communication interfaces contain internal state machines and configuration registers. A strike that upsets one of these can put the device into an unintended mode: a test mode, a hung state, a configuration in which it no longer responds. The device is not damaged, but it stops working until it is reset or power-cycled. Flash memories and synchronous dynamic memories are particularly prone to such interrupts. The defence is necessarily at the system level: the software or a supervisory circuit must notice that the device has stopped responding correctly and reset it. Single-event latch-up is the most dangerous of the non-destructive-in-principle effects. The structure of a standard bulk CMOS chip contains, by accident of its layering, parasitic transistor pairs that form a thyristor, a device that, once triggered, conducts heavily and stays on until power is removed. A heavy ion can trigger this structure, creating a low-resistance path from the power supply to ground. Current surges; if it is not cut off quickly, the local heating can destroy the chip. Spacecraft that use parts susceptible to latch-up therefore include current-sensing circuits that detect the surge and remove power within milliseconds, after which the part can be restored. The software consequence is that any device so protected may disappear without warning and come back in its power-on state, having lost its contents and configuration. Burnout and gate rupture, affecting power transistors under high voltage, are permanent and are addressed by choosing and derating parts rather than by software. Scaling and the modern picture For decades, there was a comfortable assumption in some circles that shrinking transistors would make radiation effects worse in every respect. The truth is more complicated, and the complications matter for the choices engineers make today. Smaller transistors store less charge, so their critical charge falls, which on its own makes each cell easier to upset. But smaller cells also present a smaller target, so the probability that a given particle strikes a sensitive node falls too. For static memory, these effects have partially offset each other across many technology generations, and the upset rate per bit has fallen or stayed roughly level even as the rate per chip has risen, simply because chips hold so many more bits. Robert Baumann's 2005 review of soft errors in advanced semiconductor technologies documented these trends for terrestrial devices, and they broadly hold in space. Total dose tolerance has, in some respects, improved with scaling, because very thin gate oxides trap less charge. Modern commercial processors are sometimes surprisingly tolerant of total dose for that reason. Against these encouraging trends stand several worrying ones. Multiple-bit upsets have grown more common. Transients matter more at gigahertz clock rates. Modern processors contain vast amounts of unprotected internal state, in pipelines, branch predictors, caches and configuration registers, much of which a designer cannot see, let alone protect. And complex systems-on-chip are prone to functional interrupts whose effects are hard to characterise because the internal design is proprietary and the number of possible failure states is enormous. A commercial processor that comes through total dose testing well may still hang at unpredictable intervals in a heavy-ion environment for reasons its user can never fully understand. This is the picture that frames the rest of the book. Radiation guarantees a steady rate of faults in any electronic system beyond the atmosphere. Some of those faults are benign; some corrupt data; some corrupt control; some silence a device until it is reset; a few, if unchecked, destroy hardware. No single defence addresses all of them. Hardened manufacturing processes reduce their frequency. Shielding reduces dose but does little against the most energetic particles. Error-correcting codes handle upsets in memory but not in logic. Voting handles upsets in logic but not in the voter, nor errors that strike all voters identically. Watchdogs handle hangs but not silent corruption. The art of radiation-tolerant design lies in layering these defences so that each covers the gaps of the others, and in recognising which faults will still get through so that higher levels of the system can deal with them. The next chapter turns to the first of those layers: what can be done in hardware, through the choice of process, circuit design and physical shielding, to make the computer itself more resistant before any software is written. CHAPTER 3 Hardening the Hardware Before any line of flight software runs, the engineers who choose a spacecraft's computer have already made the most consequential decisions about how often that software will face corrupted data. The manufacturing process of the chips, the design of their circuits, the placement of the boards and the metal around them together determine whether the system experiences an upset once a year or once a minute, whether it will survive the mission's total dose, and whether a heavy ion can trigger a destructive latch-up. Software can compensate for a great deal, but it cannot compensate for everything, and it can never be better than the platform it runs on at detecting faults that the platform hides from it. This chapter surveys the hardware layer: hardening by process, hardening by design, shielding, and the processors that result. It ends with the most significant shift in spacecraft computing in a generation, the move from slow, heavily hardened processors towards faster parts, including commercial ones, that experience more faults and rely more on software to manage them. That shift is the reason the software techniques in later chapters are becoming more, not less, important. Hardening by process The most direct way to make a chip resistant to radiation is to manufacture it differently. A radiation-hardened process modifies the steps by which transistors are built so that ionising radiation leaves less trapped charge, single-event charge is collected less efficiently, and parasitic structures cannot latch up. Silicon-on-insulator technology is the best-known example. In a conventional bulk process, transistors are built directly in a thick wafer of silicon, and a particle passing through can deposit charge deep in the substrate that then drifts and diffuses up to the sensitive junctions. In a silicon-on-insulator process, the transistors sit in a thin layer of silicon on top of a buried layer of oxide. Charge deposited below the oxide cannot reach the transistors, so the volume from which charge is collected shrinks dramatically and the device becomes much harder to upset. Because the transistors are isolated from one another by oxide rather than by junctions in shared silicon, the parasitic thyristor structure that causes latch-up does not exist, and silicon-on-insulator devices are effectively immune to that failure mode. These advantages explain why many hardened processors, including later generations of the RAD750, have been built on silicon-on-insulator processes. Other process measures address total dose. Hardened processes use specially grown gate and isolation oxides that trap less charge, and layouts that avoid the edge leakage paths through which trapped charge causes most of the damage in ordinary transistors. Enclosed-layout transistors, in which the gate forms a closed ring around one terminal so that there is no oxide edge between source and drain, were developed for this purpose and are widely used in radiation-tolerant designs. Guard rings, heavily doped regions surrounding sensitive structures, collect stray charge and suppress latch-up. The cost of process hardening is that it requires a dedicated or specially controlled manufacturing line, and the market for radiation-hardened chips is tiny by semiconductor standards. Hardened processes therefore lag the commercial state of the art by many years, often more than a decade. A hardened processor fabricated on a process two or three generations behind the commercial frontier is inevitably slower, larger and more power-hungry than its commercial contemporaries, and far more expensive. This lag is the root of the performance gap discussed at the end of the chapter. Claims of hardness are only as good as the testing behind them, and radiation qualification is a discipline in its own right. Total dose tolerance is usually measured by exposing powered parts to gamma rays from a cobalt-60 source, raising the dose in steps and checking electrical parameters after each step until the part drifts out of specification. Because dose effects depend on the rate at which dose is delivered, and some bipolar technologies degrade more at the low rates found in space than at the high rates convenient in a laboratory, test standards specify dose rates and, for sensitive technologies, require low-dose-rate testing that can take weeks. Single-event behaviour is measured at particle accelerators. In the United States, heavy-ion testing is commonly carried out at facilities such as the Texas A&M University Cyclotron Institute and the 88-Inch Cyclotron at Lawrence Berkeley National Laboratory, where a beam of ions of chosen species and energy is directed at a device with its lid removed, and the device is tilted to vary the effective path length and hence the deposited charge. Proton testing, important for parts that will cross trapped belts or face solar events, is done at medical and research proton facilities. Each campaign yields the cross-section curves described in the previous chapter, from which a mission's expected upset rates are computed. Two cautions follow from this practice. First, a test characterises the parts tested, from a particular manufacturing lot, under the conditions of the test. Hardened parts are manufactured on controlled lines precisely so that results from one lot can be trusted for another; commercial parts offer no such assurance, and two lots of the same commercial chip can behave quite differently in a beam. Second, testing a complex device such as a processor requires running software on it while it is irradiated, and the upset rate observed depends on what that software does. A processor running a tight loop that uses few registers will report fewer upsets than one exercising its full pipeline and caches. Engineers therefore try to test with workloads representative of flight, and to interpret results from simple test programmes with care. Hardening by design An alternative approach, increasingly common since the early 2000s, accepts a commercial or near-commercial manufacturing process and builds radiation tolerance into the circuit design instead. Hardening by design lets engineers benefit from the density and speed of more modern processes while still achieving acceptable upset rates. The most important building block is the hardened storage cell. A standard static memory cell or flip-flop holds its state in a pair of cross-coupled inverters: each drives the other, and the pair settles into one of two stable states. A strike on either node can flip the pair. Hardened cells add redundant storage nodes arranged so that a strike on any single node is corrected by the others. The dual interlocked storage cell, described by Calin, Nicolaidis and Velazco in 1996, uses four nodes in a ring such that the state is held by two pairs, and an upset in one node is restored by the remaining three. Other designs add resistors or capacitors that slow the feedback loop so that a transient pulse dies out before it can flip the cell. The cost is area, power and speed: a hardened flip-flop may be two or three times the size of an ordinary one. At a higher level, hardening by design applies the principle of triple modular redundancy within the chip: critical registers are triplicated and their outputs voted on every clock cycle, so that an upset in any single copy is outvoted. If the voter's output is fed back into all three copies, the upset is also corrected, so errors cannot accumulate. Some designs extend this to the clock and reset networks themselves, so that a transient in one clock tree does not corrupt all three copies at once. The mathematics and limitations of this technique are examined in the next chapter; here it is enough to note that it can be applied at the level of individual flip-flops, entire functional blocks or whole processors. The European Space Agency's approach to processors illustrates hardening by design at scale. Since the late 1990s, ESA has funded the development of the LEON family of processors, implementations of the open SPARC V8 architecture whose register files and caches are protected by error-detecting and correcting codes and whose design is available for implementation in different processes. The fault-tolerant versions, designated LEON2-FT, LEON3FT and LEON4FT, detect and correct upsets in their internal memories transparently, often without software ever noticing. The GR740, a quad-core LEON4FT processor developed by Cobham Gaisler with ESA support, runs at up to 250 megahertz and is fabricated on a 65-nanometre space process from STMicroelectronics. It has become a standard choice for European missions that need more performance than earlier single-core parts provided. Field-programmable gate arrays occupy a special place in this picture. Spacecraft use them extensively for interfaces, signal processing and increasingly as computing accelerators, because they offer custom hardware without the cost of a custom chip. But an SRAM-based FPGA stores its configuration, the very definition of the circuit it implements, in millions of static memory cells, and an upset in one of those cells can silently change the circuit's function. Engineers counter this with configuration scrubbing, periodically reading back or rewriting the configuration memory to correct upsets, and with design-level triplication of the logic. Xilinx produced a radiation-hardened SRAM-based device, the Virtex-5QV, specifically to reduce these effects. Other families avoid the problem altogether: antifuse FPGAs, such as Microsemi's RTAX series, are programmed once by permanently forming connections, and flash-based devices, such as the RTG4 from the same company, hold their configuration in non-volatile cells that are far less susceptible to upset. Each choice trades flexibility, density and in-flight reprogrammability against vulnerability. Shielding and its limits The oldest radiation defence is simply to put material between the particles and the electronics. Spacecraft structure, equipment boxes and circuit board materials all provide some shielding, and for missions to severe environments designers add dedicated shields. The most striking example is Juno, which has orbited Jupiter since 2016. Its flight computer and other critical electronics sit inside a box with titanium walls about a centimetre thick, a radiation vault designed to cut the dose from Jupiter's trapped electrons by a large factor. Europa Clipper, launched in October 2024 to study Jupiter's moon Europa through dozens of close flybys, carries a similar vault. Both missions also shaped their trajectories around radiation: Juno's highly elliptical polar orbit and Europa Clipper's flyby design limit the time spent in the most intense parts of the belts. Shielding is effective against the particles that dominate total dose in trapped belts, namely electrons and lower-energy protons, because these lose energy quickly in matter. A few millimetres of aluminium removes a large fraction of the dose from a typical trapped-electron spectrum. But shielding is subject to rapidly diminishing returns. Each additional millimetre removes less dose than the one before, because the particles that remain are the more energetic ones, and it adds mass that must be launched. Against galactic cosmic rays, practical shielding achieves little: a heavy ion with billions of electron-volts of energy passes through any reasonable thickness of metal. Worse, a high-energy particle striking a shield can break up nuclei in the shield material, producing a shower of secondary particles, including neutrons, that may cause more upsets than the original particle would have. For single-event effects in interplanetary space, shielding is not the answer. The practical rule follows from the physics. Shielding is used to keep total dose within the tolerance of the chosen parts, especially near Jupiter and when crossing Earth's belts; designers sometimes place sensitive parts deep inside equipment stacks or behind other components, a practice called spot shielding. Single-event effects are handled by the choice of hardened parts, by circuit design and, above all, by the fault-tolerance techniques in the rest of this book. The performance gap and the move towards commercial parts The consequences of process and design hardening show up clearly in the processors that have actually flown. Table 3 compares several that span the past three decades and the near future. Table 3. Representative spacecraft processors and their approaches to radiation. Processor Architecture basis Clock rate Radiation approach Example use RAD6000 IBM POWER (RSC) up to about 33 MHz Hardened process Mars Pathfinder, Spirit and Opportunity RAD750 PowerPC 750 110 to 200 MHz Hardened process Curiosity, Perseverance, Juno, Webb GR740 SPARC V8, quad LEON4FT up to 250 MHz Hardened by design, ECC throughout European science and Earth-observation missions Snapdragon 801 ARM, quad-core up to about 2.3 GHz None; commercial part Ingenuity Mars helicopter PIC64-HPSC RISC-V, multicore Not yet flown Hardened, fault-tolerant design Selected for future NASA missions Source: manufacturer data sheets; NASA and ESA mission documentation. The RAD750, produced by BAE Systems from the early 2000s, is the paradigm of the hardened processor. It is a derivative of a PowerPC design that appeared in desktop computers in 1997, rebuilt on a radiation-hardened process. It is rated for total doses in the hundreds of kilorads and is extremely resistant to upset and latch-up. It has flown on dozens of missions, and its record is outstanding. It is also, by any modern measure, slow: the 200-megahertz version aboard Curiosity and Perseverance offers a tiny fraction of the performance of a contemporary phone. For many years, this did not matter, because flight software for cruise, orbit and surface operations did not need more. As spacecraft have been asked to process images, navigate by terrain and make decisions on their own, it has come to matter a great deal. Perseverance carries additional computing hardware, built around a radiation-tolerant FPGA, specifically to accelerate the image processing that its terrain-relative navigation system needed during landing, because the main computer could not do it fast enough. NASA's response is the High Performance Spaceflight Computing project. In August 2022, the Jet Propulsion Laboratory awarded Microchip Technology a contract to develop a processor that NASA said would provide at least a hundred times the computational capacity of current spaceflight computers, with improved fault tolerance and the ability to power down unused capability to save energy. The resulting device, which Microchip markets as the PIC64-HPSC family, is built around multicore RISC-V processors and includes features such as time-sensitive networking. Its arrival does not eliminate the need for software fault tolerance; a far more capable processor contains far more state, and its users will run far more complex software on it. It does shift the balance of what is feasible on board. The other response is more radical: use commercial parts directly and manage their weaknesses in software and system design. Ingenuity, the helicopter that accompanied Perseverance to Mars, is the most celebrated example. Its navigation computer was a Qualcomm Snapdragon 801, a smartphone processor running Linux, which processed camera images at a rate no hardened processor could match. The team did not trust it blindly. Ingenuity's flight control ran on a separate set of safety-oriented automotive-grade microcontrollers alongside an FPGA, so that the fast commercial processor computed the navigation solution while more conservative hardware handled the time-critical control loop and could reset or work around the navigation processor if necessary. Ingenuity was planned for five flights over thirty days. It flew seventy-two times over nearly three years before rotor damage ended its mission in January 2024. Whatever upsets its commercial processor experienced, the system design absorbed them. Commercial parts are also now routine in small satellites and in some commercial human spaceflight systems, where operators have stated publicly that they use multiple commercial processors in redundant arrangements rather than single hardened ones. The logic is that several fast, cheap processors whose outputs are compared can provide more performance and, if the comparison is designed well, comparable integrity to one slow hardened processor. The weakness is that commercial parts are poorly characterised for radiation, can suffer functional interrupts whose causes are opaque, and may latch up; a system that uses them must monitor current, reset devices freely and treat every component as liable to fail at any moment. The broader lesson is that the boundary between hardware and software fault tolerance is moving. When the processor is a RAD750, upsets are rare enough that much flight software can treat the hardware as reliable and concentrate on design errors and hardware failures. When the processor is a commercial chip, or even a modern hardened multicore, upsets become a routine operating condition that software must expect, detect and correct. The next two chapters examine the principal techniques for doing so: redundancy and voting, and the protection of memory and program state. CHAPTER 4 Triple Modular Redundancy and the Mathematics of Voting If one copy of something might be wrong, keep three and believe the majority. Triple modular redundancy, universally abbreviated to TMR, is the simplest and most intuitive answer to unreliable components, and it appears at every scale in spacecraft: triplicated flip-flops inside a hardened chip, triplicated variables in software, triplicated computers in a flight control system. It is also routinely misunderstood. Triplication does not make a system three times as reliable, does not protect against faults that strike all three copies alike, and under some conditions makes a system less reliable than a single copy would have been. This chapter sets out what voting can and cannot do, why the details of where and how one votes matter more than the decision to vote at all, and how the most demanding real systems have handled the problems that simple majority voting leaves open. From von Neumann to the Saturn V The idea of building reliable systems from unreliable parts was formalised by John von Neumann in a set of lectures given at the California Institute of Technology in 1952 and published in 1956 as "Probabilistic Logics and the Synthesis of Reliable Organisms from Unreliable Components". Von Neumann was thinking about both vacuum-tube computers, whose components failed constantly, and the brain, which seemed to compute reliably despite unreliable neurons. He showed that by replicating each logic element and taking majority votes among the copies, and by restoring the signals at each stage, a network could be made arbitrarily reliable provided each component's error probability was below a threshold. The argument rested on two ingredients that remain essential today: replication, so that a single error can be outvoted, and restoration, so that errors do not accumulate. The engineering application followed within a decade. R. E. Lyons and W. Vanderkulk of IBM published "The Use of Triple-Modular Redundancy to Improve Computer Reliability" in 1962, giving the analysis that most textbooks still follow. IBM put the idea to work in one of the most important computers of the Apollo programme: the Launch Vehicle Digital Computer, which guided the Saturn rockets from lift-off to Earth orbit and onto the trajectory towards the Moon. Its logic was triplicated, with voting circuits at multiple points within the processing chain so that an error in one channel was outvoted before it could propagate. Its memory was duplicated and checked. The Saturn V never lost a mission to a guidance computer failure. The mathematics of voting Suppose a single module has reliability R, meaning the probability that it is still working correctly at the end of some period. Three independent copies feed a perfect voter, which outputs the majority value. The system works if all three modules work, or if exactly two of them do. The probability of all three working is R cubed; the probability of exactly two working is three times R squared times one minus R. Adding these gives the reliability of the triplicated system: three R squared minus two R cubed. This expression has a feature that surprises people who meet it for the first time. When R is close to one, the triplicated system is much more reliable than a single module. If a module has a 99 per cent chance of surviving a period, the triplicated system has about a 99.97 per cent chance, reducing the probability of failure from one in a hundred to about three in ten thousand. But as R falls, the advantage shrinks, and at R equal to one half the two are exactly equal. Below one half, the triplicated system is worse than a single module, because it now needs two working modules out of three and it is more likely than not that two have failed. For components that fail at a constant rate, R falls exponentially with time. The triplicated system therefore begins far more reliable than a single module, but its advantage erodes, and after a time equal to about 0.69 divided by the failure rate, it becomes the worse option. Integrating the reliability over all time gives the mean time to failure, and here the result is starker still: a simple triplicated system without repair has a mean time to failure of five-sixths that of a single module. Triplication improves the probability of surviving a short mission and worsens the expected lifetime. The reason is that, without repair, a triplicated system that has lost one module is now a duplex system that cannot tolerate another fault, and it has twice as many modules exposed to failure as a simplex system would. The remedy, and the decisive ingredient for spacecraft, is repair. If a module that has disagreed with the majority can be restored to the correct state before a second fault strikes, the system returns to full triplication, and its reliability depends on the probability of two faults occurring within a single repair interval rather than over the whole mission. For transient faults such as radiation upsets, repair is easy in principle: overwrite the bad copy with the voted value. This is why hardened chip designs feed the voter output back into all three registers, and why software that triplicates variables writes the voted value back after every read. With frequent correction, the probability of two independent upsets striking two copies of the same value between corrections becomes tiny, and the improvement over a single copy is enormous. Without correction, upsets accumulate silently until the majority is lost. The voter itself must also be considered. If the voter is a single circuit, it is a single point of failure, and the reliability of the whole system can never exceed the reliability of the voter. Designers deal with this in three ways. They make voters small and simple, so they have few failure modes and a small sensitive cross-section. They triplicate the voters too, so that each of three downstream copies has its own voter, an arrangement that keeps the triplication unbroken through successive stages. Or they push the final vote into a physical mechanism that is inherently robust, a technique the Space Shuttle used, as described below. In software, a bitwise majority vote over three copies of a word is a single line of logic, and the same expression can report which copy disagreed so that it can be repaired. /* Bitwise majority of three copies; repairs any disagreeing copy. */ static inline uint32_t vote3(volatile uint32_t a, volatile uint32_t b, volatile uint32_t c, uint32_t mismatch) { uint32_t x = a, y = b, z = *c; uint32_t m = (x & y) | (x & z) | (y & z); /* per-bit majority */ mismatch = (x ^ m) | (y ^ m) | (z ^ m); / nonzero if any copy differed */ if (*mismatch) { a = m; b = m; c = m; } / restore full triplication */ return m; } A worked comparison makes the effect of repair concrete. Take a stored value whose single copy has a one in a thousand chance of being corrupted over some period. Triplicated without repair, with copies independently exposed, the chance of losing the majority over the same period is about three in a million, a thousandfold improvement over short periods that, as shown above, erodes as the period lengthens. Now repair the copies every time the value is read, say once per second over that same period. The value is lost only if two copies are corrupted within the same one-second window, and if the period is a day, there are about eighty-six thousand such windows, each with a vanishingly small chance of a double hit. The probability of failure falls by a further factor of tens of thousands. Repair converts triplication from a modest, decaying advantage into protection that holds for the life of the mission, which is why every serious implementation of voting, in hardware or software, writes the majority back. The bitwise form is important. It means that even if two copies have each suffered a different single-bit upset, provided the upsets are in different bit positions, every bit position still has a correct majority, and the vote recovers the right value. The mismatch output lets the system count upsets, which is useful both for diagnosing failing hardware and for noticing that an anomalously high rate suggests something worse than random upsets is happening. Where to vote, and what to vote on The mathematics assumes that the three modules should produce bit-for-bit identical outputs, so that any disagreement means a fault. That assumption holds for triplicated flip-flops clocked together and for triplicated copies of a stored variable. It fails as soon as the modules are separate computers reading separate sensors. Consider three computers, each reading its own gyroscope and computing the spacecraft's attitude. The three gyroscopes will never agree exactly, because each has its own noise and bias. The three computers, if they run on independent clocks, will sample their sensors at slightly different times. Their outputs will differ in the last few digits even when nothing is wrong. Bitwise voting is useless here. Instead, systems use inexact voting, accepting values that agree within a tolerance, or mid-value selection, in which the middle of three values is chosen. Mid-value selection has an attractive property: a single faulty value, however wild, can never be the middle one unless it happens to lie between the two good ones, in which case it is close enough to be harmless. Choosing the tolerance for inexact voting, however, is a genuine design problem. Too tight, and noise causes false alarms that remove healthy channels; too loose, and a slowly drifting sensor stays in the vote long enough to do damage. Independent computers also have to agree on when to compute. If each runs freely, their results will not correspond to the same instant, and comparison becomes meaningless. Redundant flight computers are therefore synchronised, either in lockstep, executing the same instruction on the same clock cycle, or loosely, exchanging synchronisation messages at defined points in each processing cycle and comparing results at those points. Loose synchronisation is more robust to clock faults but requires careful design of the synchronisation protocol itself, which, as a later chapter shows, has been the source of some memorable failures. A deeper difficulty concerns faults that are not merely wrong but inconsistent. In 1982, Leslie Lamport, Robert Shostak and Marshall Pease published "The Byzantine Generals Problem", which asked how a group of processors could agree on a value when some of them might be faulty in arbitrary ways, including sending different values to different recipients. They proved that, with unauthenticated messages, tolerating f such faults requires at least 3f+1 processors and several rounds of message exchange. Four processors are needed to tolerate one arbitrary fault, not three. It is tempting to regard Byzantine faults as an exotic theoretical construction, but engineers who have built real distributed control systems disagree. Kevin Driscoll and colleagues at Honeywell, in their 2003 paper "Byzantine Fault Tolerance, from Theory to Reality", described how ordinary hardware faults produce precisely this behaviour: a transmitter with a marginal output voltage, or a signal edge arriving close to a receiver's sampling instant, can be read as one value by some receivers and another by others. Each receiver then votes on different inputs, and a system designed for three-way majority voting can find its healthy channels disagreeing with each other because a single faulty channel told them different stories. The defence is either a protocol designed for Byzantine agreement, or an architecture, such as a bus that physically guarantees every receiver sees the same signal, that rules such faults out. The Shuttle and the family of redundancy schemes The most elaborate voting architecture in the history of spaceflight belonged to the Space Shuttle. The orbiter carried five IBM general-purpose computers. During critical phases such as ascent and entry, four of them ran identical copies of the primary flight software in a redundant set. They were tightly synchronised, exchanging status at frequent intervals, and each compared its outputs with the others. A computer that disagreed with the rest was identified and voted out of the set by the crew on the basis of the computers' fault indications. With four computers, the Shuttle could lose one and remain fully fault-tolerant, then lose a second and still continue safely: the design goal was described as "fail operational, fail safe". The final vote did not happen in a computer at all. Each of the four computers commanded the aerosurface actuators through its own channel, and the actuators combined the four commands mechanically and hydraulically. If one channel commanded something different from the other three, the three overpowered it in what engineers call a force fight, and the deviating channel could be isolated by pressure sensing. The vote was cast in hydraulic fluid, by physics, with no single electronic voter to fail. The fifth computer ran entirely different software, the Backup Flight System, developed by a different contractor from different requirements, which could take over if the primary software failed in a way that affected all four primary computers alike. Its existence acknowledged, from the beginning of the programme, that four identical copies of the same software could share a single bug. That concern is the subject of the sixth chapter. The Shuttle's architecture is one point in a family of redundancy schemes that engineers choose among according to what they need to detect, what they need to mask and what they can afford. Table 4 sets out the main members of that family. Table 4. Common redundancy schemes in spacecraft computing. Scheme Units Detects a fault Masks a fault Main weakness Duplex with comparison 2 Yes No; must stop or switch Cannot tell which unit is wrong Standby sparing 1 active, 1 or more spares Only via self-checks After switchover delay Dormant spare may have failed unnoticed Triple modular redundancy 3 plus voter Yes Yes, one fault Voter and common-mode faults Pair-and-spare 2 pairs Yes Yes, by switching pairs Four units' cost and power Byzantine-tolerant set at least 4 Yes, including inconsistent faults Yes, one arbitrary fault Protocol complexity The choice for most deep-space missions is not TMR at the level of whole computers but standby sparing. Curiosity and Perseverance each carry two complete rover computers, one active and one held as a backup. Cassini carried redundant command and data computers and redundant attitude control computers. The reasons are power, mass and the nature of the mission. A rover on Mars has no need to mask a computer fault in microseconds; if its computer fails, it can stop, switch to the backup and wait for Earth. What it needs is to detect the fault reliably and preserve the ability to switch. Curiosity exercised precisely this capability in February 2013, about two hundred sols into its mission, when corrupted memory in its active computer caused repeated problems. The team commanded a switch to the backup computer, which became the prime unit while engineers diagnosed the original. Standby sparing has its own trap, which Table 4 names: a dormant spare can fail without anyone knowing, so that when it is called on it does not work. Missions therefore periodically power and test their spares, and fault protection logic must handle the case in which a switchover to the spare fails and the original unit, possibly faulty, is the best remaining option. Duplex comparison deserves mention because it is so common inside individual computers. Many modern safety processors, including those used on Ingenuity and in automotive systems, run two cores in lockstep, executing the same instructions and comparing every output. A mismatch cannot be resolved, since there is no majority, but it can be detected instantly, and the processor can halt or reset before an erroneous output escapes. Combined with a higher-level system that can recover from such a halt, this provides strong integrity at modest cost. Pair-and-spare architectures put two such pairs together, so that a detected mismatch in one pair causes immediate switchover to the other. The common thread is that redundancy is a means of turning undetectable or uncorrectable faults into detectable and correctable ones, and its value depends on three properties the mathematics alone does not show. The copies must be repaired faster than faults accumulate. The voting or comparison mechanism must be more reliable than the things it protects. And the copies must fail independently. The first two are matters of design discipline. The third, independence, turns out to be the hardest of all to achieve, and its failure has caused some of the most expensive losses in spaceflight. Before turning to it, the next chapter examines the most pervasive target of radiation upsets: the memory in which programs and their state reside. CHAPTER 5 Guarding Memory and State Memory is where radiation does most of its routine damage. A modern spacecraft computer may hold hundreds of megabytes of volatile memory and gigabytes of non-volatile storage, against a processor core whose sensitive internal state amounts to a few tens of kilobytes. Since upset probability scales roughly with the number of bits exposed, the overwhelming majority of single-event upsets on any spacecraft occur in memory. Those upsets corrupt data, parameters, stored commands and, most dangerously, the program itself. A spacecraft whose memory is not protected will, given enough time, execute a corrupted instruction or act on a corrupted value; the only question is when. This chapter examines the layered defences that keep memory trustworthy: error-correcting codes in hardware, scrubbing to prevent errors accumulating, software techniques that detect corruption in data and control flow when hardware protection is absent or incomplete, and the design of state that survives the resets that every long mission will experience. Taken together, these techniques embody the principle that runs through this book: no layer assumes the one below it is correct. Error-correcting codes The foundation of memory protection is the error-correcting code. Richard Hamming, frustrated by relay computers at Bell Laboratories that halted whenever they detected an error over the weekend, published the first practical codes in 1950. The idea is to store, alongside each group of data bits, a small number of check bits computed from the data. When the word is read back, the check bits are recomputed and compared with those stored. The pattern of differences, called the syndrome, identifies which bit, if any, has flipped, and the hardware corrects it before passing the word to the processor. The code almost universally used for spacecraft and server memory is a single-error-correcting, double-error-detecting code, abbreviated SEC-DED. For a 64-bit data word, it adds eight check bits, for 72 bits in all. It corrects any single flipped bit and detects, without being able to correct, any two flipped bits. A widely used construction was published by M. Y. Hsiao of IBM in 1970, optimised so that the check logic is fast and small. Processors such as the RAD750 and the LEON family, and the memory controllers that accompany them, implement such codes in hardware so that single-bit upsets in main memory are corrected transparently on every read. The software need never know they occurred, though well-designed systems count them, because a rising count is useful evidence. SEC-DED codes have two important limits. They correct only one error per word, so a multiple-bit upset or two separate upsets accumulating in the same word defeat them. And they protect data only while it rests in memory; a value corrupted in a processor register or in an unprotected cache before it is written to memory will be encoded faithfully, wrong value and all, with matching check bits. Memory designers address the first limit by interleaving bits so that physically adjacent cells belong to different words, and by using stronger codes where multiple-bit upsets are likely. Mass memory such as the flash storage used for science data commonly uses Reed-Solomon codes, introduced by Irving Reed and Gustave Solomon in 1960, which operate on multi-bit symbols and can correct bursts of errors, or BCH codes, which correct several independent bit errors in a block. The second limit is addressed partly by protecting caches and register files with their own codes, as the LEON fault-tolerant designs do, and partly by the software techniques described later in this chapter. Non-volatile memory brings its own problems. Spacecraft store their flight software in non-volatile memory such as EEPROM, flash or, increasingly, magnetoresistive and other technologies, and load it into volatile memory at boot. Flash memory cells are relatively resistant to upset in their stored state but vulnerable to errors during writing and erasure, and the complex controllers in flash devices are susceptible to functional interrupts. Flash cells also wear out after many erase cycles, so file systems spread writes across the device. The combination of complex controller logic, wear management and a file system layered on top makes flash one of the more fragile parts of a spacecraft, a point the Spirit rover's software team learned dramatically, as a later chapter recounts. Scrubbing: winning the race against accumulation Error-correcting codes correct errors when a word is read. A word that is never read is never corrected, and while it sits unread, upsets can accumulate in it. A single-bit error in a rarely accessed word is harmless as long as no second upset strikes the same word before it is read and corrected. If one does, the word now has two errors, which SEC-DED detects but cannot correct. The defence is scrubbing: a background process, in hardware or software, that reads every word of memory on a regular cycle, letting the code correct any single-bit errors and writing the corrected value back. Scrubbing turns the question of accumulation into a race. If upsets arrive at a given rate per word, the probability that two strike the same word within one scrub interval depends on the square of the product of that rate and the interval. Over a mission of fixed length, which contains many intervals, the expected number of uncorrectable double errors therefore grows in proportion to the interval and to the square of the upset rate. Halving the interval halves the risk; doubling the upset rate quadruples it. The analysis was worked out in detail by Saleh, Serrano and Patel in 1990, and it gives designers a straightforward way to choose a scrub rate: estimate the worst-case upset rate, including solar particle events, and scrub fast enough that the expected number of uncorrectable errors over the mission is negligible. A worked example shows how these dependencies interact. Suppose, purely for illustration, that each 72-bit word in a memory experiences an upset on average once every ten million days, and the memory holds about a hundred million words. Across the whole memory, upsets arrive at about ten per day, so single-bit errors are routine. With a scrub interval of one day, the chance that a word already carrying one error takes a second hit before the scrubber reaches it is about one in ten million; with ten first hits a day, the memory can expect an uncorrectable double error roughly once every million days, comfortably beyond any mission. Lengthen the interval to a hundred days, perhaps because a software scrubber has been starved of processor time, and the expected time to an uncorrectable error falls a hundredfold, to around ten thousand days, which is now comparable to the life of a long mission. Raise the upset rate a hundredfold during a severe solar event, and, because the risk scales with the square of that rate, the expected time falls by a further factor of ten thousand, to about a day. The numbers are invented to show the scaling, but the lesson is real: the margin that looks enormous in quiet conditions can collapse when the scrubber slows and the environment worsens at the same time, which is exactly the combination a heavily loaded spacecraft in a solar storm presents. In practice, scrub intervals on spacecraft range from seconds to hours depending on the memory size, the environment and the scrubbing mechanism. Many memory controllers include a hardware scrubber that walks through memory in idle bus cycles. Where hardware does not provide one, a low-priority software task does the same job. The software version has a trap: if it runs at low priority and the processor becomes heavily loaded, it may be starved of time and stop scrubbing, exactly when an intense solar event might also be raising the upset rate. Good designs therefore guarantee the scrubber a minimum share of time, or monitor its progress and raise an alarm if a full pass takes too long. The same race applies to configuration memory in FPGAs, as the previous chapter noted, and to every other copy of state that the system relies on. A triplicated software variable that is voted only when it is used, and never corrected otherwise, is no better than an unscrubbed memory word: two upsets in different copies, at the same bit position, will eventually destroy the majority. The principle is general. Any redundancy that is not regularly exercised and repaired decays. Scrubbing addresses soft errors, those in which the hardware is undamaged. It cannot fix hard errors, where a memory cell or chip has failed permanently. When that happens, the only remedy is to stop using the failed region. Voyager 1 provided a remarkable demonstration in 2024. On 14 November 2023 the spacecraft's flight data subsystem, the computer that packages science and engineering data for transmission, began sending unreadable data. The spacecraft continued to receive and execute commands, so engineers could investigate, but every test took nearly two days to return a result. By March 2024 they had established that a single memory chip in the subsystem had failed, corrupting part of the stored software. The chip could not be repaired, and no single free region was large enough to hold the affected code. The team therefore split the code into pieces, relocated each piece to a different free area of memory, and adjusted every reference so that the pieces still worked as a whole. They sent the first relocated section on 18 April 2024, and two days later Voyager 1 returned readable engineering data for the first time in five months. The episode illustrates a capability that matters as much as any code or scrubber: the ability to reshape a running program around failed hardware, many years after launch, from the ground. Software-implemented fault tolerance Hardware protection covers most memory, but not all state. Processor registers, pipeline latches and unprotected caches can be upset, and in commercial processors much internal state has no protection at all. When an upset corrupts a value before it reaches protected memory, or corrupts the processor's control flow directly, hardware codes are blind to it. Software-implemented fault tolerance aims to catch such errors by building redundancy into the program itself. The first family of techniques duplicates computation. In EDDI, short for Error Detection by Duplicated Instructions, described by Nahmsuk Oh, Philip Shirvani and Edward McCluskey of Stanford in 2002, the compiler emits two copies of every computation, using different registers and memory locations, and inserts comparisons at points where values leave the protected domain, such as before stores to memory and before branches. If an upset corrupts one copy, the comparison fails and the program raises an error. SWIFT, presented by George Reis and colleagues at Princeton in 2005, refined the approach to reduce overhead by relying on error-correcting memory to protect stored values and duplicating only the computation in registers. These techniques roughly double the instruction count and slow execution substantially, but they can be applied selectively to critical code and require no special hardware. The second family protects control flow. An upset in the program counter, in a branch target or in the return address on the stack can send execution to an arbitrary location, where it will execute whatever instructions it finds. Control-flow checking by software signatures, a technique published by the same Stanford group in 2002, assigns a signature to each basic block of the program and maintains a runtime signature register that is updated at each block boundary. If execution jumps somewhere it should not, the runtime signature no longer matches the expected value at the destination, and the error is detected. Simpler variants are common in flight software: functions that set a token on entry and check it on exit, sequences of steps that each verify the previous step completed, and assertions placed at points that should be reachable only from known predecessors. The third family protects critical data structures directly. Values that would cause harm if corrupted, such as a thruster enable flag, an operating mode or a pointer to a command table, can be stored in a form that makes corruption detectable. A common pattern stores each value alongside its bitwise complement, so that any single upset breaks the relationship between them. typedef struct { uint32_t val; uint32_t inv; } guarded_u32; static void g_write(guarded_u32 *g, uint32_t v) { g->val = v; g->inv = ~v; } /* Returns false if the stored pair has been corrupted. */ static bool g_read(const guarded_u32 g, uint32_t out) { if ((g->val ^ g->inv) != 0xFFFFFFFFu) return false; *out = g->val; return true; } A failed read cannot be corrected, since there is no way to tell which half was hit, but it can trigger a safe response: reload the value from a protected source, fall back to a safe default or escalate to fault protection. Where correction is needed in place, the triplicated form from the previous chapter serves instead. Larger structures, such as tables of parameters and stored command sequences, are protected with checksums or cyclic redundancy checks that are verified before each use and periodically in the background. Flight software that loads its parameter tables at boot and never checks them again is relying on memory protection alone; software that checks a table's integrity each time it is used catches corruption that slipped past the hardware. Another approach to protecting the program itself is to verify the code image periodically. Many flight systems compute a checksum over the executable code in memory and compare it with a stored reference, and if a mismatch is found they reload the code from non-volatile storage or reboot. Since code is normally read-only, this is a simple and powerful check, and it catches errors that SEC-DED misses because they have accumulated beyond its correcting power. All of these techniques cost time, memory and complexity, and complexity is itself a source of defects. The art lies in applying them where they matter most: to the state whose corruption would cause irreversible harm, such as propulsion, power switching and attitude control, and to the values that persist longest without being rewritten. State that survives resets Every spacecraft computer will reset during its mission, whether because of a radiation-induced hang, a software fault, a watchdog timeout or a deliberate command. What survives the reset determines whether the spacecraft picks up where it left off or starts over, and whether it repeats the mistake that caused the reset. Spacecraft distinguish between cold restarts, which reload everything from non-volatile storage and reinitialise all state, and warm restarts, which preserve selected state in a protected region of memory. Warm restarts allow the system to remember critical information: which hardware units have been marked as failed, which fault responses have already been tried, what time it is, what mode it was in and how many times it has reset recently. That last item matters a great deal. A fault that causes a reset and then recurs immediately after the reset produces a reset loop, in which the computer spends all its time restarting and never does anything useful. Counting resets in persistent memory lets the boot software notice a loop and break it, for example by booting a different software image, skipping a suspect initialisation step or coming up in a minimal safe configuration. Preserved state is itself vulnerable. If the very state that survives a reset has been corrupted, the reset will not clear the problem but carry it forward. Robust designs protect persistent state with checksums or redundant copies and validate it at every boot, falling back to defaults if it fails. They also keep multiple copies of the flight software in non-volatile memory, typically a known-good image from launch and one or more updated images, with boot logic that verifies each before using it and can fall back to the older one if the newer is damaged or fails to run. This arrangement is what makes in-flight software updates survivable: if a new version contains a fatal flaw, the spacecraft can still return to a version that is known to work. The broader lesson is that memory protection is not a single mechanism but a stack. Hardware codes catch most upsets in memory; scrubbing prevents those upsets from accumulating into uncorrectable errors; software checks catch corruption that hardware cannot see; protected persistent state and multiple software images let the system recover even when the corruption is severe enough to force a restart. Each layer assumes the layer beneath it will occasionally fail. That assumption is well founded for random radiation upsets, which strike independently. It is far more dangerous to assume for faults that strike every copy at once, which is the subject of the next chapter. CHAPTER 6 When Redundancy Fails Together The mathematics of redundancy in the fourth chapter rested on one assumption that is easy to state and extraordinarily hard to secure: that the copies fail independently. Radiation upsets largely satisfy it. A heavy ion that flips a bit in one computer's memory does not, in general, flip the corresponding bit in another computer on the other side of the spacecraft. But radiation is only one source of faults. Design errors, mistaken assumptions, shared inputs and shared notions of time strike every copy at once, and against them replication offers no protection whatsoever. Two identical computers running identical software on identical inputs will make identical mistakes, and the voter will faithfully report that they agree. Engineers call these common-mode or common-cause failures. They are the reason that the most dependable redundant systems in history have nonetheless failed, and they represent the deepest limit on what fault tolerance can achieve. This chapter examines the classic case, the experimental evidence on whether diverse software escapes the problem, the subtler forms of common mode that arise from time and shared assumptions, and the practical means engineers use to reduce, though never eliminate, the risk. Ariane 501: redundancy defeated by design On 4 June 1996, the first Ariane 5 rocket lifted off from Kourou in French Guiana carrying four Cluster science satellites. About thirty-seven seconds later, the launcher veered off course, began to break up under aerodynamic loads and was destroyed by its self-destruct system. The inquiry board, chaired by Jacques-Louis Lions, published its report the following month. It remains the most widely taught case in the history of software failure, and its central lesson concerns redundancy. Ariane 5's guidance depended on two inertial reference systems, each a complete unit with its own computer running the same software, arranged as an active unit and a hot standby. Their software had been inherited largely from Ariane 4, on which it had flown successfully. Part of it was an alignment function that computed, among other values, a quantity related to the rocket's horizontal velocity, which the report calls the horizontal bias. On Ariane 4, this function continued to run for a short time after lift-off, a holdover from a requirement that allowed a countdown to be resumed quickly after a late hold. On Ariane 5 it served no purpose after lift-off at all. But it continued to run. Ariane 5 accelerated horizontally much faster than Ariane 4 in its early flight. The horizontal bias value, computed as a 64-bit floating-point number, grew larger than any value Ariane 4 had ever produced. The software converted it to a 16-bit signed integer, which can hold values only up to 32,767. The conversion overflowed and raised a hardware exception. The designers had analysed which conversions could overflow and had protected some of them; for this one, they had judged from Ariane 4's flight profile that overflow was physically impossible, and had left it unprotected to save processing time on a heavily loaded processor. The exception was not handled locally, and the system-level response to an unhandled exception was to declare the unit failed and shut its processor down. The backup inertial reference system had already failed in exactly the same way, about seventy-two milliseconds earlier, because it ran the same software on the same data. When the active unit failed, there was nothing to switch to. Worse, the failed active unit sent a diagnostic bit pattern to the main flight computer, which interpreted it as flight data and commanded the engine nozzles to full deflection. The rocket turned sharply and broke apart. The report's analysis is worth quoting in substance, because its implications reach far beyond Ariane. The decision to shut down a processor on detecting an exception reflected a design philosophy in which faults were assumed to be caused by random hardware failures, which would affect one unit and not the other, so that shutting down the faulty unit and relying on its twin was correct. Against a software design error, the same response guaranteed that both units would be shut down by the same input. Redundancy, designed to defend against independent faults, had converted a single design error into the simultaneous loss of both units. A single unit with a more forgiving exception policy, which simply carried on with a wrong value for a quantity that no longer mattered, would have flown. Every element of this failure recurs elsewhere: code reused from a context where its assumptions held, an unprotected operation justified by analysis of a different flight profile, a function that kept running when it no longer needed to, an error response designed for hardware faults applied to a software fault, and a pair of redundant units that shared everything except their physical hardware. The deepest lesson is the mismatch between the fault model and the response. A spacecraft that treats every anomaly as a random hardware fault, and responds by switching to an identical spare, is defenceless against the faults that most often kill missions. Does diverse software help? If identical software fails identically, the obvious remedy is to make the copies different. N-version programming, advocated by Algirdas Avižienis and colleagues at UCLA from the late 1970s, proposed that several teams independently develop programs from the same specification, so that their faults would be uncorrelated and a majority vote among them would mask design errors as triplication masks hardware errors. The hypothesis was tested directly by John Knight and Nancy Leveson, whose 1986 paper "An Experimental Evaluation of the Assumption of Independence in Multiversion Programming" reported an experiment in which twenty-seven programs were written independently, by students at two universities, from a single specification for a missile-defence-style problem. Each program was subjected to a million test cases. Most versions were highly reliable individually. But when the researchers looked at which inputs caused failures, they found that versions failed together far more often than independence would predict. Statistically, the hypothesis of independent failure was rejected with high confidence. Independent programmers, it turned out, tend to make mistakes on the same difficult parts of a problem: boundary conditions, special cases, numerically delicate computations. A theoretical model by Dave Eckhardt and Larry Lee, published in 1985, had already explained why: if some inputs are intrinsically harder than others, then even truly independent development processes will produce correlated failures, because every team is more likely to fail on the hard inputs. None of this means design diversity is worthless. Diverse versions do fail together less often than identical copies, which fail together always. The lesson is rather that diversity buys a reduction in common-mode risk, not its elimination, and that the reduction is hard to quantify. It is also expensive: multiple teams, multiple verification efforts and a voting mechanism that must cope with legitimate differences in output. The Space Shuttle's Backup Flight System is the best-known application of diversity in spaceflight. Its software was developed by a different company from the primary software, to the same high-level requirements but implemented independently, and it ran on the fifth general-purpose computer. It was designed to take over if a generic flaw in the primary software affected all four primary computers. It never had to take control during a mission, which is either evidence of the primary software's quality or simply the absence of a test. Its existence imposed real costs and caused, as the next section shows, at least one memorable problem of its own. Civil aviation adopted a related approach in the fly-by-wire systems of the Airbus A320 and its successors, which combine computers of different types, using different processors and different software developed by different teams, so that a single design error cannot take out every flight control channel at once. Deep-space missions rarely use full software diversity, because of cost and because their redundancy is mostly standby sparing, in which a single computer is active at any one time. Diversity appears instead in targeted forms: a minimal, independently written safe-mode or boot program that does not share code with the main flight software; hardware circuits that enforce critical constraints regardless of what the software commands; and fault protection logic designed by people other than those who wrote the code it monitors. Shared time and shared assumptions Some common-mode failures arise not from a shared bug in a calculation but from something every copy depends on: a clock, an epoch, a unit convention, a shared input. Time is a particularly treacherous shared dependency. On 10 April 1981, the first Space Shuttle launch was scrubbed shortly before lift-off when the Backup Flight System could not synchronise with the four primary computers. John Garman, who led the software effort, explained the cause in a paper whose title, "The 'Bug' Heard 'Round the World", has become part of software engineering folklore. A change made about two years earlier to an initialisation routine had introduced a small chance, roughly one in sixty-seven, that the primary computers would start up with a timing offset that left them out of step with the backup by one processing cycle. The four primary computers agreed with each other perfectly; it was the cross-check against an independently designed system that exposed the problem, at the cost of a scrub. Diversity had done its job, but in the form of an alarm rather than a rescue. The Deep Impact spacecraft, which had sent an impactor into Comet Tempel 1 in 2005 and later flew past Comet Hartley 2, fell silent in August 2013. Its team later traced the likely cause to a time counter that recorded tenths of a second since 1 January 2000 in an unsigned 32-bit integer. On 11 August 2013, the count reached two to the power thirty-two and overflowed. The resulting fault left the spacecraft's computers rebooting continually, unable to process commands; with its attitude no longer controlled, the spacecraft lost the ability to point its antenna and arrays, and contact was never restored. The spacecraft had redundant computers, but they ran the same software and counted the same time. Redundancy gave no protection against a fault scheduled, in effect, from the day the counter's width was chosen. Boeing's CST-100 Starliner provided a more recent example on its first uncrewed orbital flight test in December 2019. Shortly after separating from its Atlas V launcher, the spacecraft's mission elapsed timer was set from the launch vehicle at the wrong moment, leaving it roughly eleven hours off. Believing itself to be in a different phase of the mission, the spacecraft fired its thrusters in ways that consumed too much propellant to reach the International Space Station. During the flight, engineers also found and corrected a separate software error in the mapping of thruster commands for the service module's disposal manoeuvre, which could have caused a collision between the service module and the crew capsule after separation. NASA's subsequent review emphasised that integrated, end-to-end testing with the real interfaces would have caught both errors. In each case, every redundant computer on board shared the same wrong value or the same wrong table. Redundant data paths can share a weakness even when their sources are independent, if every consumer filters the data in the same way. An accident in civil aviation, examined in a long report by the Australian Transport Safety Bureau, shows how. On 7 October 2008, an Airbus A330 operating as Qantas Flight 72 pitched abruptly nose-down twice while cruising off the coast of Western Australia, injuring many of those on board. One of the aircraft's three air data and inertial reference units had begun intermittently emitting spikes of grossly wrong data, including angle of attack. The flight control computers compared the three units' values and were designed to reject a single bad source, but their algorithm had a specific weakness: a particular pattern of spikes, arriving with a particular spacing in time, could slip through its filtering. The redundant sources were genuinely independent; the logic that combined them was common to every flight control computer, and it was the logic, not the sensors, that let the error through. The investigators could not determine what made the faulty unit behave as it did. They examined whether a single-event effect from atmospheric radiation might have been the trigger, but found no conclusive evidence for that or any other cause. The case is a reminder that the voting and filtering software that sits above redundant hardware is itself a single design shared by all channels. Shared assumptions between teams produce the same pattern at a larger scale. Mars Climate Orbiter was lost on arrival at Mars in September 1999 because ground software supplied thruster impulse data in pound-force seconds where the navigation software expected newton seconds. The trajectory errors accumulated over months of cruise, and the spacecraft entered the atmosphere far lower than planned. The failure was in ground software, not flight software, but its structure is the same: two components, each internally consistent, sharing an unverified assumption at their interface. No amount of redundancy in either component could have helped. Engineering for independence If common-mode failures cannot be eliminated, they can be made less likely and less harmful. The practices that achieve this are less glamorous than voting circuits, but they account for much of the difference between missions that survive their own mistakes and those that do not. The first is to treat software faults as software faults. Ariane 501's exception policy, Deep Impact's counter and Starliner's timer all show that when a design error triggers, it will do so in every copy at once. Responses should therefore not assume that switching to an identical spare will help. Instead, the system should be able to continue with degraded or default values, to restart the affected function rather than declare the whole unit dead, or to fall back to a different and simpler implementation. Where a computation's result no longer matters, as with Ariane's alignment function after lift-off, the software should stop performing it. The second is to diversify where diversity is affordable and effective. A safe mode implemented with independent code, a boot loader that shares nothing with the main flight software, a hardware timer that resets the processor regardless of software state, and an analogue circuit that prevents a thruster firing for too long all provide protection against common-mode software failure at modest cost. Diversity of inputs also matters: two sensors measuring the same quantity by different physical principles are far less likely to fail together than two identical sensors. The third is to remove shared hazards by design. Time counters can be sized to outlast any conceivable mission, or written so that rollover is handled correctly; NASA's own lessons-learned records after Deep Impact emphasised this. Units can be carried explicitly in interface definitions rather than assumed. Arithmetic conversions can be checked or saturated rather than allowed to overflow. Each such practice removes a class of design error that would otherwise strike every redundant copy at once. The fourth is independent verification. NASA established an Independent Verification and Validation programme, with a facility in Fairmont, West Virginia, in the early 1990s, partly in response to the Challenger accident, so that engineers outside each mission team examine critical flight software. Independence of people is a form of diversity: reviewers who did not share the original developers' assumptions are more likely to notice them. NASA's software engineering requirements, codified in its procedural requirements document NPR 7150.2, mandate this for the highest-criticality software. The European Space Agency's ECSS standards impose comparable obligations. The fifth, and the most fundamental, is humility about what testing can show. The Ariane 4 software had flown flawlessly; the Shuttle's initialisation change had passed testing for two years; Deep Impact's counter worked perfectly for thirteen years. A design error that depends on a condition outside the tested envelope, whether a new trajectory, a rare timing alignment or a future date, is invisible until the condition arrives. The only defence is to ask, of every assumption a piece of software makes, what would happen if it were false, and to make sure that something in the system, preferably something designed by different people with different assumptions, would notice. These five practices are not separate from the architecture of flight software; they are embodied in it. The next chapter examines that architecture: how operating systems, partitioning, watchdogs and coding discipline build structures in which faults, whether random or designed in, stay where they start. CHAPTER 7 Architecture for Containment Redundancy and error-correcting codes deal with faults one value or one unit at a time. But a spacecraft's flight software is a large program, often running to hundreds of thousands or millions of lines, doing many things at once: commanding instruments, controlling attitude, managing power and thermal systems, storing and transmitting data, executing sequences and watching for faults. A fault in any part of it, whether caused by radiation or by a design error, can propagate through shared memory, shared processor time, shared resources and shared assumptions to every other part. The architecture of the software determines whether a fault in the camera-handling code stays in the camera-handling code or brings down the attitude control system along with it. The organising idea of this chapter is containment. Well-architected flight software is divided into regions such that a fault inside one region cannot corrupt another, cannot starve another of the resources it needs, and is detected and handled at the region's boundary. This chapter examines the real-time foundations on which that architecture rests, two famous incidents in which containment broke down and was rescued, and the mechanisms, from watchdog timers to coding rules, that make containment real rather than notional. Real-time foundations Almost all spacecraft flight software runs on a real-time operating system. Wind River's VxWorks has flown on a long series of NASA missions, including Mars Pathfinder, the Mars Exploration Rovers, Curiosity and Perseverance. RTEMS, an open-source real-time executive with origins in US Army missile programmes, is widely used on European and NASA missions. What these systems offer is not speed but predictability. Tasks are scheduled by fixed priority, so that the most urgent work always runs first, and the time the system takes to respond to an event is bounded and analysable. Rate monotonic analysis, developed by Liu and Layland in 1973, gives a mathematical test for whether a set of periodic tasks with fixed priorities will always meet their deadlines; flight software teams use it, and its descendants, to show that control loops will run on time even under the worst-case load. On top of the operating system sits an application framework that structures the flight software into components. NASA's core Flight System, developed at Goddard Space Flight Center and released as open source, organises software into applications that communicate through a publish-and-subscribe software bus, so that no application calls another directly or shares its memory. JPL's F Prime, released as open source in 2017 and flown first on the ASTERIA cubesat and then on Ingenuity, organises software into components connected by typed ports, with the connections described in a model from which much of the communication code is generated. Both frameworks reflect the same philosophy. Components interact only through defined interfaces; each component owns its own state; and the framework, not the components, mediates every interaction. This structure makes faults easier to contain, because a component that misbehaves can affect others only through messages, which can be checked, rate-limited and logged. Stronger containment requires hardware support. A processor with a memory management unit can give each component its own address space, so that a wild pointer in one cannot overwrite another's data. The aviation standard ARINC 653 goes further, defining a partitioned operating system in which each partition receives a guaranteed slice of processor time and a fixed region of memory, enforced by the operating system and the hardware. A partition that loops forever or corrupts its own memory cannot take processor time or memory from any other. Such time-and-space partitioning has entered spacecraft design through European initiatives and hypervisor-based systems that run several operating systems side by side on one processor, allowing critical software and less trusted software, such as experimental autonomy or instrument code, to share hardware without sharing fate. The cost of strong partitioning is overhead and rigidity, and many deep-space flight systems have historically run all their tasks in a single address space on a processor with no memory protection enabled, relying on coding discipline instead. As processors become more capable and software more complex, that trade-off is shifting. But even within a single address space, the architecture can enforce containment through ownership rules, message passing and runtime checks, and it can protect the shared resources, processor time, locks and buses, through which faults most often spread. When containment failed: Pathfinder and Spirit Two incidents on Mars illustrate how faults escape their boundaries through shared resources, and how architecture made recovery possible. Mars Pathfinder's software, running on VxWorks, included a shared data area called the information bus, protected by a mutual-exclusion lock. Several tasks used it, among them a high-priority task that managed the bus, a low-priority task that gathered meteorological data and a set of medium-priority tasks, including communications. A few days after landing in July 1997, the lander began resetting itself. The cause, as Glenn Reeves, the software team lead, explained in an account that circulated widely among engineers that December, was a classic priority inversion. The low-priority meteorological task would take the lock and begin publishing its data. The high-priority bus management task would then need the lock and block, waiting for the low-priority task to finish. But before the low-priority task could finish and release the lock, a medium-priority communications task would become ready, preempt it, and run for a long time. The high-priority task was thus blocked indefinitely behind a medium-priority task that had nothing to do with the lock. When the bus management task failed to complete its cycle, a watchdog-style check in the scheduler detected the missed deadline, concluded that the system had failed and reset the computer. Three features of the architecture turned this from a potential disaster into a well-known engineering story. First, the deadline check worked as intended: it detected that the system was no longer making progress and forced a reset that returned the lander to a known state, preserving the mission even before the cause was known. Second, the team had left a trace facility enabled in the flight software, which recorded the sequence of system events leading up to each reset; when they reproduced the problem on an identical testbed at JPL, the trace showed exactly what had happened. Third, VxWorks offered priority inheritance as an option on its mutexes, a mechanism under which a task holding a lock temporarily inherits the priority of the highest-priority task waiting for it, so that medium-priority tasks cannot preempt it. The option had been left off. The spacecraft included a C-language interpreter that let engineers change variables in the running software, and they used it to turn priority inheritance on for the relevant lock. The resets stopped. The lesson for architecture is that containment must extend to time and locks, not just memory. A low-priority task that holds a resource shared with a high-priority one has, in effect, been promoted to high-priority status for the duration, whether the designers intended it or not. Modern flight software practice treats every shared lock as a design hazard: locks are minimised, held briefly, protected by priority inheritance or priority ceiling protocols, and analysed as part of the schedulability analysis. The Spirit rover's flash anomaly in January 2004 illustrates a different escape route: a shared resource that grows silently until it is exhausted. On the eighteenth sol after landing, Spirit stopped communicating normally and began returning only fragmentary, nonsensical data. Over the following days, the team established that the rover was resetting repeatedly, scores of times. Glenn Reeves and Tracy Neilson later described the cause in their paper on the anomaly. The rover stored files in its flash memory using a file system whose directory structures were held in RAM when the flash was mounted. Over the long cruise to Mars and the first weeks on the surface, a very large number of files had accumulated, including many left over from cruise that were no longer needed. The memory required to manage them grew until it exceeded what the system could allocate. Once it did, the software failed while handling the file system, the computer reset, and on restart it tried to mount the flash again, failed again and reset again. Because the failure occurred during initialisation, every reset led straight back into it. The rescue depended on a capability built into the boot sequence: the ability to command the rover to start up without mounting the flash file system, operating from RAM alone. Engineers sent commands to boot in that mode, which they called crippled mode, during the brief windows when the rover was awake and listening. With the rover stable, they could examine and then reformat the flash, deleting the accumulated files, and restore normal operations. Spirit went on to operate for six more years. The architectural lessons were explicit: resources that grow over time must be monitored and bounded; initialisation must not depend on subsystems that might themselves be the cause of the fault; and there must always be a way to bring the computer up in a minimal configuration that excludes the suspect parts. Watchdogs, health monitoring and coding discipline The watchdog timer is the last line of defence against a computer that has stopped working, and its design is more subtle than it looks. A basic watchdog is a hardware counter that resets the processor if it is not periodically reset, or "kicked", by software. If the software hangs, whether because of an infinite loop, a deadlock or a radiation-induced processor fault, the kicks stop and the watchdog forces a restart. A naive implementation kicks the watchdog from a periodic timer interrupt or a high-priority task. This catches a processor that has stopped altogether but misses the more common case in which the processor is running but some essential task is not: a deadlock that blocks the attitude control task, say, while the timer interrupt cheerfully keeps kicking. Better designs kick the watchdog only when every critical task has reported, since the last kick, that it is alive and making progress. #define ALL_CRITICAL_TASKS ((1u << NUM_CRITICAL_TASKS) - 1u) static volatile uint32_t checkin_mask; void task_checkin(unsigned task_id) { checkin_mask |= (1u << task_id); } /* atomic in practice */ /* Called once per watchdog period by the lowest-priority health task. */ void health_service(void) { if (checkin_mask == ALL_CRITICAL_TASKS) { hw_watchdog_kick(); checkin_mask = 0; } else { log_missing_tasks(~checkin_mask & ALL_CRITICAL_TASKS); /* evidence survives reset */ } } Placing the kick in the lowest-priority task has a further advantage: if higher-priority work ever consumes all processor time, the health task starves and the watchdog fires, catching overload as well as hangs. Some hardware watchdogs are windowed, triggering a reset if kicked too early as well as too late, which catches software that has fallen into a tight loop that happens to include the kick. And the watchdog's evidence, which tasks missed their check-in, must be recorded somewhere that survives the reset, or the team on Earth will see only that the computer restarted, not why. Above the watchdog sits a wider health-monitoring layer that checks each task's timing, each queue's depth, each resource's usage and each device's responsiveness. Its job is to detect faults earlier and more specifically than a full reset would, so that the response can be narrower: restart one task, reset one device, clear one queue. This graduated response is the subject of the next chapter. The final element of containment is the code itself. Gerard Holzmann of JPL's Laboratory for Reliable Software published "The Power of Ten: Rules for Developing Safety-Critical Code" in 2006, a set of ten deliberately terse rules that have since shaped coding standards across the space industry. They forbid recursion and unbounded loops, so that stack depth and execution time are bounded and analysable. They forbid dynamic memory allocation after initialisation, eliminating an entire class of fragmentation and exhaustion failures, the class that brought down Spirit's file system. They require functions short enough to read on one page, a minimum density of assertions, checking of every return value, minimal scope for data and restricted use of pointers and the preprocessor. And they require that code compile without warnings under the strictest settings and pass static analysis. JPL's institutional coding standard for C, built on these rules and on the MISRA C guidelines from the automotive industry, governed the flight software for Curiosity, whose development Holzmann later described in a 2014 article titled "Mars Code". That article recounts how the team combined coding rules, routine static analysis, peer review supported by tools and model checking of critical concurrent algorithms to produce flight software running to millions of lines. These rules are often presented as matters of style, but they are better understood as containment measures. A program with bounded loops cannot hang in a loop forever. A program with no dynamic allocation cannot run out of heap in flight. A program with assertions checks its own assumptions at runtime, turning a silent corruption into a detected fault that can be handled at the boundary of the component where it occurred. A program whose every return value is checked does not proceed as though a failed operation had succeeded. Each rule narrows the range of behaviours that a fault can produce, and therefore the range of ways a fault can escape. NASA's 2009 study of flight software complexity, led by Daniel Dvorak, documented how the size of flight software had grown by roughly an order of magnitude every decade or so, and argued that complexity was itself becoming a principal source of risk. Its recommendations, including better architecture, earlier analysis and explicit management of complexity, remain relevant. The architecture of containment is the main tool by which large flight software stays manageable: if faults stay local, the system can be understood and verified one component at a time. Containment also applies to the boundary between the spacecraft and its operators. Commands arriving from Earth pass through layers of checking before they reach the software that acts on them: the uplink is protected by error-detecting codes and, increasingly, by authentication; each command is checked for valid format and for parameters within allowed ranges; and the software rejects commands whose preconditions are not met in the current mode. Hazardous commands, such as those that fire pyrotechnic devices, open propulsion valves or disable fault protection, are commonly guarded by a two-step protocol in which a separate arming command must be received, within a short window, before the hazardous command will execute. A single corrupted or mistaken command therefore cannot do irreversible damage on its own. These checks are tedious to specify and test, and they occasionally block a command the team genuinely wanted, but they are the architectural expression of the first chapter's warning that uploads are among the most dangerous events in a mission. The same architecture must allow the software to be changed in flight, which every long mission eventually requires. Curiosity's own landing showed how far this can go. The software that controlled cruise and landing was not the software the rover needed for surface operations; the surface version, still being completed after launch, was sent up during the cruise and held in storage. In the days after landing in August 2012, the team carried out a planned transition, installing the new flight software on both rover computers and rebooting into it over several sols, while preserving the ability to fall back to the old version if the new one misbehaved. Patches of that scale are only safe in an architecture with a protected boot path, verified multiple images in non-volatile memory and the kind of reset-surviving state described in the fifth chapter. Smaller changes, from adjusting a single parameter to replacing one module, rely on the same machinery. An architecture that cannot be safely updated has, in effect, frozen its defects in place for the life of the mission. What containment cannot do on its own is decide what to do when a fault is detected. A reset is a blunt instrument; a mission cannot afford to reset its computer for every anomaly, and some faults, such as a failed thruster or a stuck valve, are not in the computer at all. The spacecraft needs a layer of software whose specific job is to recognise faults anywhere in the system, decide on a response and keep the mission alive while doing so. That layer, fault protection, and its ultimate refuge, safe mode, are the subject of the next chapter. CHAPTER 8 Fault Protection and Safe Mode Every deep-space spacecraft carries, alongside the software that does its job, a second body of software whose purpose is to keep it alive when something goes wrong. At JPL and across much of the American planetary programme this is called fault protection; the broader discipline that encompasses its design, analysis and operation is now usually called fault management. Its job is to watch the whole spacecraft continuously, recognise symptoms of failure, decide what to do, and do it, all without help from Earth. Its ultimate refuge is safe mode, a minimal configuration in which the spacecraft can survive indefinitely while it waits for instructions. Fault protection is where the threads of this book come together. It must handle radiation-induced upsets that escape lower-level defences, hardware failures that redundancy can work around, design errors that strike in unpredictable ways and operator mistakes that arrive with the next upload. It must do so on timescales set by the physics of the spacecraft rather than the light time to Earth. And it presents a paradox: the software that is meant to save the mission when things go wrong is, by its nature, the software that runs least often, is hardest to test, and when it errs, can do the most damage. This chapter examines how fault protection is structured, what safe mode must achieve, how fault protection has itself caused losses, and how engineers try to verify software whose whole purpose is to run on the worst day of the mission. Monitors, responses and containment regions The architecture of American planetary fault protection descends from the Voyager and Galileo spacecraft, was refined on Cassini, and has been carried forward with variations on most subsequent JPL missions. Its basic elements are monitors and responses. A monitor watches some measurable symptom: a bus voltage, a temperature, the rate at which an attitude estimate is changing, the time since the last message from a subsystem, the count of correctable memory errors. It compares the symptom with an expected range and declares a fault when the symptom stays out of range for long enough. That last clause matters. Spacecraft telemetry is noisy, sensors glitch and transients come and go, so monitors almost always include persistence filters, requiring a condition to hold for several consecutive samples or a set fraction of a time window before acting. The persistence must be long enough to reject noise and single-event transients but short enough that a real fault is caught before it does harm, which is a design trade that must be made separately for every monitor. A response is a predefined action or sequence of actions taken when a monitor declares a fault. Responses are typically tiered. The first tier tries the least disruptive remedy: retry a command, reset a device, switch to a redundant sensor. If the fault persists, the next tier takes broader action: power-cycle a subsystem, swap to a redundant electronics string. If that fails, the final tier is to abandon the current activity and enter safe mode. Each tier records what it has tried, so that if the same fault recurs the system does not loop through the same unsuccessful remedies indefinitely. The design is organised around fault containment regions: parts of the spacecraft, defined so that a fault within one region cannot propagate to another, and so that each region can be isolated and replaced by a redundant counterpart. A redundant pair of star trackers might form two regions; a computer and its power supply might form one. The fault protection designer's job is to make sure every credible fault can be detected, isolated to a region, and recovered by switching to that region's backup or by degrading gracefully without it. NASA's Fault Management Handbook, developed by an agency-wide working group in the early 2010s, sets out this framework and the analysis that supports it, including fault trees, failure modes and effects analyses and explicit mapping of every credible failure to its detection and response. Responses interact, and the interactions are where fault protection becomes hard. Two monitors may detect symptoms of the same underlying fault and trigger conflicting responses. A response may mask a symptom that another monitor depends on. A response in progress may be interrupted by a higher-priority fault. Fault protection systems therefore include an arbitration mechanism, sometimes called a fault protection engine, that queues responses, resolves conflicts by priority and ensures that only one response controls a given part of the spacecraft at a time. Cassini's fault protection, for example, ran responses through such a mechanism with explicit priorities, and the design of those priorities occupied much of the system engineering effort. Table 5 shows how fault protection fits into the overall hierarchy of defences, from the hardware mechanisms of earlier chapters to the ground team. Table 5. Tiers of fault response on a deep-space spacecraft. Tier Typical faults handled Response time Example mechanisms Hardware Bit upsets, latch-up, bus shorts Nanoseconds to milliseconds Error-correcting codes, current limiters, voting logic Local software Hung tasks, device interrupts, bad data Milliseconds to seconds Watchdogs, retries, device resets, assertions System fault protection Failed units, out-of-range conditions Seconds to minutes Monitors, tiered responses, redundancy swaps Safe mode Faults not otherwise resolved Minutes Sun-pointing, low-gain link, minimal loads Ground team Diagnosis, repair, software updates Hours to weeks Testbed reproduction, patches, new sequences The principle underlying the table is that each tier handles what it can, as quickly and locally as possible, and passes what it cannot to the tier above. A well-designed system resolves the vast majority of faults at the lowest tiers, where they cost nothing but a log entry. Safe mode is rare and the ground team is involved only when the spacecraft has done everything it can. Safe mode: the spacecraft's refuge Safe mode is the state a spacecraft enters when its fault protection has run out of specific remedies. Its requirements are few and absolute. The spacecraft must be thermally and electrically stable, which usually means pointing its solar arrays at the Sun, or for nuclear-powered spacecraft maintaining a safe thermal attitude. It must be able to communicate with Earth, usually through a low-gain antenna that does not require precise pointing, at a low data rate that can be received with large ground antennas. It must turn off everything not needed for survival, especially science instruments and any activity in progress. And it must stay in that state indefinitely, without depending on any part of the spacecraft that might have caused the problem, until instructed otherwise. Several design principles follow. Safe mode should depend on as little as possible: the simplest attitude control, the fewest sensors, the minimum of software. It should be entered by a path that is independent of the software that may have failed, so that even a corrupted flight software image can be overridden by a hardware or boot-level mechanism. It should be testable and tested more thoroughly than anything else on the spacecraft, because it is what everything else falls back on. And it must not be possible for a single command to disable it. A companion mechanism is the command loss timer. A spacecraft that has received no valid command from Earth for a defined period, perhaps several days or weeks depending on the mission, assumes that its ability to receive commands has failed, whether because of a fault in its receiver, an antenna pointed the wrong way or a configuration error. It then works through a predefined sequence: switch to the backup receiver, try different antennas, reset the communications hardware, and re-point towards Earth using stored knowledge. Voyager's ability to recover from the 2023 antenna mispointing, described in the first chapter, rested on precisely this kind of logic. Safe mode is a frequent event for most long missions, and its triggers are often surprising. Mars Reconnaissance Orbiter, which has operated at Mars since 2006, entered safe mode several times in 2009 following unexplained computer resets; the team kept it there for an extended period while diagnosing the cause and preparing software changes to make recovery more robust before resuming science. Such episodes are not failures of fault protection but its success: the spacecraft protected itself, and the ground had time to understand the problem. When fault protection kills The paradox of fault protection is that it holds the most authority over the spacecraft at exactly the moments when the spacecraft is least understood. It is designed to take drastic action, such as swapping hardware, firing thrusters and changing attitude, on the basis of symptoms that may be misleading. When its model of the world is wrong, it can turn a survivable problem into a fatal one. Two losses illustrate the danger. Mars Global Surveyor orbited Mars from 1997 and returned a vast trove of data before contact was lost in November 2006. NASA's review board, reporting in April 2007, traced the loss to a chain of events that began months earlier. Two operators had uploaded parameter updates to two separate copies of the spacecraft's software parameters, and the copies diverged. A subsequent corrective update, sent in June 2006, wrote data to incorrect memory addresses. The damage stayed latent until November, when a routine command to adjust the solar array caused the array to be driven against a stop. Fault protection detected the problem and put the spacecraft into safe mode, as designed. But the attitude chosen in safe mode, together with the effects of the corrupted parameters, left one of the spacecraft's batteries exposed to direct sunlight. It overheated, and the fault protection logic interpreted the overheating as overcharging and reduced the charge, while the corrupted parameters also kept the spacecraft's antenna from pointing at Earth. The batteries were depleted within hours, and the spacecraft was lost. The board highlighted inadequate procedures for verifying memory uploads and the absence of checks that might have caught the corrupted values. Every element of the safe-mode response had worked according to its design; the design simply had not anticipated a world in which the parameters it relied on were wrong. Japan's Hitomi X-ray astronomy satellite, launched in February 2016, was lost little more than a month later in a sequence that the Japan Aerospace Exploration Agency described in detail. The attitude control system came to believe, on the basis of erroneous data from an inertial reference unit, that the spacecraft was rotating when it was not. It commanded the reaction wheels to counter the phantom rotation, which actually set the spacecraft spinning. Data from the star tracker, which could have corrected the error, was not accepted under the conditions that prevailed. As momentum built up in the wheels, the system attempted to unload it, could not, and entered a safe-hold mode that fired thrusters to stabilise the spacecraft. But the thruster control parameters had been updated shortly before, with an error, and the firings increased the spin rather than reducing it. The spacecraft rotated faster and faster until parts broke away. Hitomi illustrates in compressed form the danger of fault protection that trusts one sensor over another without adequate cross-checking, and of safe-mode parameters that are changed in operation without the same scrutiny as the original design. In both cases, the fault protection was active, the safe mode was entered and the spacecraft died. The common lesson is that safe mode must be robust not only to the fault that triggers it but to the state of the spacecraft at the time, including corrupted parameters and misleading sensors. A safe mode that relies on recently changed parameters, or that acts on a single sensor's view of the world, inherits all the uncertainty of the situation it is meant to escape. Verifying software meant for bad days Fault protection is unusually difficult to verify, and the reasons are structural. Normal flight software runs constantly in testing and in flight, so its defects tend to surface. Fault protection runs only when a fault occurs, and faults are rare in testing, so most of its code paths are exercised rarely or never before launch. The space of possible faults is enormous: every component that can fail, in every way, at every moment, in combination with every other possible fault. And the most dangerous scenarios involve exactly the unexpected conditions that designers did not think to test. Engineers attack this problem with several complementary methods. The first is systematic fault injection. On testbeds that combine flight-equivalent computers with simulated or real spacecraft hardware, test teams deliberately cause faults: they disconnect sensors, inject corrupted data, force devices to stop responding, flip bits in memory, cut power to units and send malformed commands. Each test checks that the fault protection detects the fault, isolates it correctly and responds as designed. Missions typically run many hundreds of such tests, and the fault protection test campaign is often among the longest of any in the development. The second is analysis. Fault trees and failure modes and effects analyses are used to enumerate credible faults and to check that each has a detection and a response. Timing analysis checks that responses complete before the fault becomes fatal. Formal methods have been applied increasingly to fault protection logic, because its state machines and priority schemes are exactly the kind of concurrent, combinatorial logic in which human reasoning fails and exhaustive analysis succeeds. Model checking tools such as SPIN, developed by Gerard Holzmann, can explore every interleaving of a concurrent design and find the rare one that leads to deadlock or an unsafe state. The third is radiation testing of the hardware and software together. Beyond characterising individual parts at accelerator facilities, some teams expose running computers to particle beams to observe how the complete system, including its software defences, responds to real upsets. This exposes failure modes, such as functional interrupts in complex chips, that component-level testing misses. The fourth is the principle that engineers summarise as "test as you fly, fly as you test". The Starliner errors described in an earlier chapter were found to be detectable by end-to-end testing with real interfaces, which had not been carried out. Many past failures have been traced to configurations or sequences that differed, often in small ways, from what was tested. The discipline is to test the actual flight software, with the actual parameters, in conditions as close as possible to flight, and to treat any untested change, including a parameter update in operations, as a risk. The losses of Mars Global Surveyor and Hitomi both involved parameters changed in flight without the full verification the original design had received. The final tool is humility about what has not been tested. No campaign can exercise every combination of faults, and fault protection will eventually face a situation its designers did not foresee. The best defence is to design responses that are safe even when the diagnosis is wrong: to prefer actions that preserve options over those that consume them, to avoid irreversible actions such as firing pyrotechnic devices or venting propellant unless the evidence is overwhelming, to cross-check sensors before acting on them, and to make safe mode as simple and independent as possible. Fault protection that is robust to its own errors is more valuable than fault protection that is right in every tested case. These principles are about keeping a spacecraft alive. But modern missions increasingly ask their software to do much more than survive: to land on uncharted terrain, drive across it, fly through it and decide which rocks are interesting. The next chapter examines what changes when a spacecraft's software must not only protect itself but act on its own understanding of the world. CHAPTER 9 Autonomy Beyond Survival Everything so far has been about keeping a spacecraft alive. Fault protection and safe mode are conservative by design: when in doubt, stop, point at the Sun, call home and wait. That posture works for most faults during cruise and orbital operations, because waiting costs only time. It does not work when waiting is impossible. A lander falling towards the surface of Mars cannot pause to ask Earth whether its radar is lying. A helicopter in flight cannot stop in mid-air to think. And a rover that waits for Earth to approve every metre of travel and every measurement will accomplish a fraction of what it could. Modern missions therefore give their software authority to act, not merely to protect: to land on terrain no one has seen at close range, to drive across it, to fly over it, to choose scientific targets and to plan activities. This autonomy brings a new class of failure that the earlier chapters did not address. A spacecraft with corrupted memory or a failed unit has a fault in its hardware or software. A spacecraft that misjudges its altitude because it has rejected a correct sensor reading has a fault in its understanding of the world. Such faults are not caught by error-correcting codes or watchdogs, because every bit is correct and every task is running on time. This chapter examines how autonomous spacecraft have failed in this way, the design principles that have emerged from those failures, and the methods engineers use to bound the authority of software they cannot fully verify. Landing on its own Landing is the canonical autonomous operation, because, as the first chapter showed, it is over before Earth knows it has begun. It is also the operation in which misjudgement of the world has most often been fatal. Mars Polar Lander was lost on 3 December 1999 as it descended towards the planet's south polar region. No telemetry was being returned during landing, so the cause could not be established with certainty, but the review board identified a most probable cause. The lander's three legs each carried a sensor designed to detect the jolt of touchdown and signal the software to shut off the descent engines. Testing after the loss showed that the sensors also generated a brief spurious signal when the legs deployed from their stowed position, well above the surface. The software recorded that signal. Later, when the lander reached the altitude at which it began to monitor touchdown, it read the recorded value, concluded that it had landed, and shut off its engines, leaving it to fall the remaining distance, estimated at around forty metres, at a speed it could not survive. The software requirements had not specified that transient signals at leg deployment should be ignored, and the tests that should have exercised the touchdown logic did not catch the problem because of a wiring error in the test setup. The flaw was not in any single component but in an assumption at the boundary between hardware behaviour and software logic. ESA's Schiaparelli lander, part of the ExoMars 2016 mission, was lost on 19 October 2016 in a sequence that ESA's inquiry reconstructed in detail from telemetry. After the parachute opened, the lander experienced rotation rates higher than expected. Its inertial measurement unit saturated, meaning that the rotation exceeded the range the instrument could measure, for about a second. The guidance software held the saturated value for longer than the actual saturation lasted and integrated it into its estimate of the lander's attitude, producing a large attitude error. When the radar altimeter's measurements were combined with that erroneous attitude, the navigation system computed an altitude below zero: the lander believed it was underground. It responded as though it had landed or was about to. It released the parachute and back shell, fired its thrusters for only about three seconds instead of the planned descent burn, and activated its on-ground systems. It fell from an altitude of roughly three and a half kilometres. The inquiry identified the handling of the saturation flag, the absence of a plausibility check that would have rejected a negative altitude, and insufficient modelling of the parachute dynamics as contributing factors. The ispace HAKUTO-R Mission 1 lander, attempting the first commercial landing on the Moon in April 2023, failed in a way that inverted Schiaparelli's error. As the lander descended, it passed over the rim of a large crater, and the altitude measured by its laser rangefinder jumped sharply as the ground fell away beneath it. The lander's software had a filter designed to reject implausible measurements, to protect against a faulty sensor. The jump was large enough to be treated as such a fault, and the software discarded the rangefinder data, continuing instead with its own estimate. That estimate was wrong: the lander believed it had reached the surface when it was still around five kilometres up. It descended slowly, waiting for touchdown, until it ran out of propellant and fell. ispace's analysis noted that the landing site had been changed after an earlier design review and that simulations had not adequately represented the terrain along the new approach. The fault protection worked exactly as designed; it was the design's model of what a plausible measurement looked like that was wrong. Intuitive Machines' IM-1 lander, Odysseus, in February 2024 showed autonomy working under pressure, with the help of a hurried change from the ground. While the spacecraft was in lunar orbit, its team discovered that its own laser rangefinders would not work: a safety switch that should have been enabled before launch had not been. With hours to spare, engineers uploaded a software patch to use data from an experimental NASA navigation lidar carried as a payload instead. The lander reached the surface, though it came down with lateral motion, broke a leg and came to rest on its side. It returned data, and it became the first American spacecraft to land softly on the Moon since 1972. Perseverance, in February 2021, demonstrated the positive case on Mars. Its Lander Vision System photographed the ground during descent beneath the parachute, matched features in those images against a map built from orbital imagery, and so established the vehicle's position relative to the terrain to within tens of metres. Using that position and a pre-loaded map of hazards, the software selected a safe landing target within reach and steered towards it. The landing ellipse in Jezero crater contained cliffs, boulder fields and dunes that earlier landers could not have risked; terrain-relative navigation made the site acceptable. The system worked because it did not trust a single measurement of position, but continually reconciled what its camera saw against an independent model of the world prepared on Earth. These cases set out the central difficulty. Autonomous landing requires software to judge which of its sensors to believe, in real time, under conditions that cannot be fully reproduced on Earth. Believe a bad sensor, as Mars Polar Lander did with its touchdown signal, and the spacecraft acts on a phantom. Disbelieve a good one, as HAKUTO-R did with its rangefinder, and the spacecraft acts on a phantom of its own making. Integrate a sensor's limitations incorrectly, as Schiaparelli did with its saturated measurement unit, and the error propagates into everything downstream. Judging one's own sensors The engineering response to these failures has converged on a small set of principles, none of them new in isolation but each sharpened by experience. The first is that plausibility checks must be grounded in physics, not only in statistics. A filter that rejects measurements because they differ sharply from the predicted value is protecting against faulty sensors, but a sharp difference may also be the world doing something the prediction did not model, such as a crater rim. Better checks ask whether a measurement is physically possible given everything else the spacecraft knows. An altitude below zero during descent, as Schiaparelli computed, is impossible on its face and should have been rejected. A rangefinder jump that matches the known topography under the flight path is entirely plausible. Encoding the second kind of knowledge requires accurate maps and a model of where the vehicle is over them, which is one reason terrain-relative navigation has become central to modern landing systems. The second is cross-checking by dissimilar sensors. A single sensor's view of the world, however carefully filtered, cannot reveal when it is systematically wrong. Two sensors that measure the same quantity by different physical means, such as radar and lidar for altitude, or inertial integration and camera-based tracking for position, fail in different ways and under different conditions, which is the same argument for diversity that the sixth chapter made for software. Hitomi's loss turned on accepting an inertial measurement over a star tracker; Schiaparelli's on trusting an inertial attitude estimate whose error the radar data could not reveal. The third is to design for the period after a sensor misbehaves. Ingenuity offers an instructive example. On its sixth flight, in May 2021, about fifty-four seconds into the flight, a glitch in the pipeline delivering images from its navigation camera caused one image to be lost, and every subsequent image was delivered with an incorrect timestamp. The navigation software, correcting its estimate with images it believed were taken at one time when they had been taken at another, began issuing corrections for errors that did not exist. The helicopter oscillated, pitching and rolling by more than twenty degrees, with spikes in power consumption. Håvard Grip, the chief pilot, explained afterwards that two design decisions saved it. The control system had been designed with substantial stability margin, so that it could tolerate large errors without going unstable. And during the final phase of descent, the navigation system deliberately stopped using camera images, because dust and the changing view near the ground were expected to make them unreliable. When Ingenuity stopped using the images, it stopped oscillating, levelled itself and touched down at the designed speed. The team had anticipated not this particular fault but the class of fault: that camera-based navigation might become untrustworthy, and that the vehicle should have a phase in which it did without it. The fourth principle is to make irreversible actions require strong evidence. Shutting off a descent engine, releasing a parachute and firing pyrotechnic devices cannot be undone. Mars Polar Lander and Schiaparelli both took such actions on a single line of evidence, a latched touchdown signal and a computed negative altitude respectively. Modern designs require such actions to be confirmed by multiple independent indications, or to be consistent with the expected state of the mission, before committing. Driving, targeting and planning Landing is the most dramatic form of autonomy, but much of its practical value lies in everyday operations on the surface. The Mars Exploration Rovers introduced autonomous navigation to Mars in 2004. Their AutoNav software used stereo images to build a local map of terrain hazards, evaluated candidate paths and drove the rover towards a goal while avoiding obstacles, without step-by-step instructions from Earth. A complementary capability, visual odometry, tracked features between successive images to measure how far the rover had actually moved, correcting for wheel slip on sand and slopes. Mark Maimone, Yang Cheng and Larry Matthies documented its performance in a 2007 paper on two years of visual odometry on the Mars rovers. Both capabilities were computationally expensive on the rovers' slow processors, which limited driving speed; Curiosity improved on them, and Perseverance's enhanced navigation, aided by dedicated processing hardware, allowed the rover to plan its path while driving rather than stopping to think between moves, greatly increasing the distance it can cover in a day. Autonomy has also moved into science. AEGIS, the Autonomous Exploration for Gathering Increased Science software developed at JPL, was uploaded to Opportunity in 2010 and later to Curiosity, where it began routine use with the ChemCam laser spectrometer in 2016, as Raymond Francis and colleagues reported in Science Robotics in 2017. After a drive, AEGIS analyses images to find rocks that match criteria set by scientists and points the instrument at them, gathering data before Earth has even seen the images. It has since been adapted for Perseverance. On Earth-orbiting spacecraft, the Autonomous Sciencecraft Experiment on NASA's Earth Observing-1 satellite, described by Steve Chien and colleagues, used onboard analysis and planning to detect events such as volcanic eruptions and floods and to retask the spacecraft to observe them, running operationally for over a decade. The most ambitious early experiment in spacecraft autonomy also produced one of its most instructive incidents. In May 1999, the Remote Agent experiment took control of NASA's Deep Space 1 spacecraft. It combined an onboard planner and scheduler, a plan execution system and a model-based diagnosis and recovery system, and it was designed to accept high-level goals and work out for itself how to achieve them. During the experiment, the execution system deadlocked: a race condition between concurrent tasks caused the system to wait for an event that would never come. Ground controllers diagnosed the problem, and the experiment was later rerun successfully. The flaw was of a type that model checking could have revealed; after the event, researchers at NASA Ames applied model checking to the relevant code and showed that the defect could have been found that way. The episode demonstrated both the promise of model-based autonomy and the danger of concurrency in software that cannot be fully tested. Assuring autonomy The verification problem for autonomous software is harder than for fault protection, because the space of situations it must handle is not merely large but open-ended. A landing system must handle any terrain, any lighting and any combination of sensor behaviours; a driving system must handle any arrangement of rocks. No test campaign can cover that space. The response, increasingly, is to bound the authority of the autonomous system and to monitor it at runtime. The idea has a formal name, the simplex architecture, from work by Lui Sha and colleagues published around 2001: a complex, high-performance controller runs alongside a simple, verified safety controller, and a monitor switches control to the safety controller whenever the complex one would take the system outside a verified safe envelope. Ingenuity embodied a version of this idea in hardware: the sophisticated navigation ran on a fast commercial processor, while the flight control ran on conservative, safety-oriented hardware that could continue without it. The general pattern, sometimes called runtime assurance, allows a spacecraft to use software that cannot be exhaustively verified, including, increasingly, machine-learning components for image analysis, while guaranteeing that its worst behaviour is bounded by something that can be. Runtime assurance is not a complete answer. The monitor must itself be correct, and its definition of the safe envelope must be accurate, which returns the problem to the question of models: HAKUTO-R's filter was a monitor, and it was wrong because its model of plausible measurements was wrong. But it changes the shape of the problem from proving that complex software always behaves correctly, which is generally impossible, to proving that a simple monitor correctly recognises when complex software is about to do something dangerous, which is often tractable. The trajectory is clear. As processors such as those emerging from the High Performance Spaceflight Computing project become available, spacecraft will be asked to do more for themselves: to navigate by terrain on every landing, to drive and fly further between contacts, to recognise scientific opportunities and respond to them, and perhaps, at the outer planets and beyond, to conduct entire phases of their missions with little more than high-level guidance from Earth. The faults they will face will increasingly be faults of judgement rather than faults of hardware. The defences developed over the previous chapters, for bits, units, code and fault protection, remain necessary. They are no longer sufficient, and the final part of this book considers what follows from that. Conclusion The cases in this book, taken together, support a conclusion that none of them shows on its own. For most of the history of spaceflight, the dominant threat to a spacecraft's computer was the physical world acting on its hardware: radiation flipping bits, components wearing out, units failing. The techniques built to meet that threat, hardened processes, error-correcting codes, scrubbing, voting and standby spares, have been so effective that the hardware-induced loss of a deep-space mission's computer has become rare. Look back over the losses and near-losses of the past three decades, and a different pattern emerges. Ariane 501 lost both its inertial units to one line of reused code. Mars Polar Lander believed a transient signal. Mars Global Surveyor acted faithfully on corrupted parameters. Deep Impact counted past the width of its own clock. Hitomi trusted the wrong sensor and then fired thrusters with a wrong table. Schiaparelli calculated that it was underground. HAKUTO-R decided that a correct altimeter was broken. None of these was caused by radiation. All of them were defeats of redundancy by something every copy shared: a design, an assumption, a parameter or a model of the world. That pattern has consequences for how flight software should be built from here. The first is that the model a spacecraft holds of itself and its surroundings must be treated as a component with failure modes, analysed as rigorously as any electronic box. When a lander's software decides whether to accept a sensor reading, it is performing fault detection; when it chooses between two estimates, it is voting; when it discards one, it is isolating a fault. These functions are usually written by guidance and navigation specialists and reviewed as algorithms, not as fault protection, and they rarely receive the systematic enumeration of failure modes that a power subsystem would. The landing failures of the past decade suggest that they should. Every filter that rejects data should come with an explicit account of the true conditions it might wrongly reject, and every estimate that drives an irreversible action should be cross-checked against physics and against an independent source. The second is that changes made in operations deserve the scrutiny of design. Phobos 1, Mars Global Surveyor and Hitomi were all lost after uploads, and Voyager 2's 2023 mispointing began with a command. The software a team launches is reviewed, tested and independently verified for years; the parameter update it sends in the fifth year of the mission may be prepared in days by a smaller team under time pressure. The spacecraft cannot tell the difference, and it will act on the update with the same authority it gives to the original design. A mission that cannot afford to verify an update at the original standard should ask whether it can afford to send it, and flight software should be built to validate uploaded parameters against physical limits and to retain a known-good set to which it can return. The third is that independence, not redundancy, is the scarce resource, and current trends are depleting it. Open frameworks such as the core Flight System and F Prime are a great advance for the field: they spread hard-won engineering across missions and lower the cost of doing things properly. But they also mean that a latent defect in a widely used component now resides in many spacecraft at once. Commercial processors bring speed and the knowledge of a vast industry, along with internal designs that their users cannot inspect and whose radiation behaviour is only partly characterised; several redundant copies of the same commercial chip share whatever weaknesses it has. The answer is not to retreat from reuse or from commercial parts, both of which the field needs, but to account for what they share. Teams should ask, for every redundant arrangement, what single design element, input, clock, parameter or vendor is common to all copies, and should place at least one genuinely independent check, however simple, across that commonality. Some questions remain unsettled, and honesty requires naming them. The field does not yet have a widely accepted method for assuring machine-learning components in flight-critical roles; runtime assurance bounds their worst behaviour but depends on monitors whose own models may be wrong. The radiation behaviour of modern multicore systems-on-chip, with their vast unprotected internal state, is understood only statistically, and functional interrupts in such devices can produce failure modes that no amount of testing fully enumerates. And there is an unresolved tension between the two halves of spacecraft autonomy. Fault protection, by long tradition, responds to uncertainty by stopping: safe the spacecraft, call home, wait. Mission-critical autonomy, during landing or a flyby or a helicopter flight, cannot stop. As spacecraft are asked to carry out longer and more consequential activities without Earth, from extended drives across Mars to independent operations at the outer planets, the boundary between "stop and wait" and "press on with degraded knowledge" will have to be drawn more often, and more finely, in software. No general principle yet says where it belongs. What, then, should practitioners do differently? The evidence of these chapters points to a handful of habits. Design every response to be safe if the diagnosis that triggered it is wrong, preferring reversible actions and preserving options. Keep, always, a minimal path back to a working state that shares as little as possible with the software it is recovering from: the crippled mode that saved Spirit, the boot images and protected state that survive resets, the ability that let Voyager's engineers relocate code around a dead chip half a century after launch. Record evidence that survives the fault, because the team on Earth can fix only what it can understand, and Pathfinder's trace buffer was worth more than any amount of speculation. Bound every resource that grows. Size every counter to outlast the mission. Treat every untested change as a risk and every tested configuration as precious. And ask of every assumption in the software, and particularly of every assumption shared by redundant copies, what the spacecraft would do if it were false. There is a final lesson, less technical but no less important. The engineers who built Voyager could not have anticipated the specific memory chip that failed in 2023, the specific command error that mispointed Voyager 2's antenna, or the specific workarounds their successors would devise. What they did anticipate was that things would go wrong in ways they could not foresee, and they built machines that left room for future people to respond. That is the real achievement of flight software engineering at its best: not a spacecraft that never fails, which is impossible, but one that fails in ways that can be survived, understood and repaired by people who are, and always will be, far too far away to help in time. Notes Introduction 1. Glenn Reeves's account of the Mars Pathfinder resets, circulated in December 1997 and widely reproduced as "What really happened on Mars?", together with Mike Jones's summary of the same incident. 2. NASA, announcement of the High Performance Spaceflight Computing processor contract awarded to Microchip Technology, August 2022. Chapter 1 1. NASA Jet Propulsion Laboratory, Deep Space Network public documentation on the Goldstone, Madrid and Canberra complexes. 2. NASA, public statements on Mars solar conjunction command moratoriums, including those for 2023 and January 2026. 3. NASA Jet Propulsion Laboratory, mission updates on the Voyager 2 antenna mispointing and recovery, July and August 2023. 4. NASA, Perseverance entry, descent and landing press materials, February 2021. 5. Don Eyles, "Tales from the Lunar Module Guidance Computer", paper presented to the American Astronautical Society Guidance and Control Conference, 2004, on the Apollo 11 program alarms and restart protection. Chapter 2 1. D. Binder, E. C. Smith and A. B. Holman, "Satellite Anomalies from Galactic Cosmic Rays", IEEE Transactions on Nuclear Science, 1975. 2. T. C. May and M. H. Woods, "Alpha-Particle-Induced Soft Errors in Dynamic Memories", IEEE Transactions on Electron Devices, 1979. 3. J. F. Ziegler and W. A. Lanford, "Effect of Cosmic Rays on Computer Memories", Science, 1979. 4. A. J. Tylka and colleagues, "CREME96: A Revision of the Cosmic Ray Effects on Micro-Electronics Code", IEEE Transactions on Nuclear Science, 1997. 5. R. C. Baumann, "Radiation-Induced Soft Errors in Advanced Semiconductor Technologies", IEEE Transactions on Device and Materials Reliability, 2005. 6. NASA, reports on the Mars Odyssey MARIE instrument following the solar particle events of October 2003. Chapter 3 1. T. Calin, M. Nicolaidis and R. Velazco, "Upset Hardened Memory Design for Submicron CMOS Technology", IEEE Transactions on Nuclear Science, 1996. 2. Cobham Gaisler, GR740 quad-core LEON4FT processor data sheet and user manual. 3. BAE Systems, RAD750 radiation-hardened processor product documentation. 4. NASA Jet Propulsion Laboratory, Juno and Europa Clipper mission documentation on radiation vaults. 5. NASA, "NASA Awards Next-Generation Spaceflight Computing Processor Contract", news release, August 2022; Microchip Technology, PIC64-HPSC product documentation. 6. J. Balaram and colleagues, "Mars Helicopter Technology Demonstrator", AIAA Atmospheric Flight Mechanics Conference, 2018. Chapter 4 1. J. von Neumann, "Probabilistic Logics and the Synthesis of Reliable Organisms from Unreliable Components", in C. E. Shannon and J. McCarthy (eds), Automata Studies, Princeton University Press, 1956. 2. R. E. Lyons and W. Vanderkulk, "The Use of Triple-Modular Redundancy to Improve Computer Reliability", IBM Journal of Research and Development, 1962. 3. L. Lamport, R. Shostak and M. Pease, "The Byzantine Generals Problem", ACM Transactions on Programming Languages and Systems, 1982. 4. K. Driscoll, B. Hall, H. Sivencrona and P. Zumsteg, "Byzantine Fault Tolerance, from Theory to Reality", SAFECOMP 2003, Lecture Notes in Computer Science, Springer, 2003. 5. A. Spector and D. Gifford, "The Space Shuttle Primary Computer System", Communications of the ACM, 1984. 6. NASA Jet Propulsion Laboratory, Curiosity mission updates on the switch to the B-side computer, February and March 2013. Chapter 5 1. R. W. Hamming, "Error Detecting and Error Correcting Codes", Bell System Technical Journal, 1950. 2. M. Y. Hsiao, "A Class of Optimal Minimum Odd-Weight-Column SEC-DED Codes", IBM Journal of Research and Development, 1970. 3. I. S. Reed and G. Solomon, "Polynomial Codes over Certain Finite Fields", Journal of the Society for Industrial and Applied Mathematics, 1960. 4. A. M. Saleh, J. J. Serrano and J. H. Patel, "Reliability of Scrubbing Recovery-Techniques for Memory Systems", IEEE Transactions on Reliability, 1990. 5. NASA Jet Propulsion Laboratory, "NASA's Voyager 1 Resumes Sending Engineering Updates to Earth", April 2024. 6. N. Oh, P. P. Shirvani and E. J. McCluskey, "Error Detection by Duplicated Instructions in Super-Scalar Processors", IEEE Transactions on Reliability, 2002. 7. N. Oh, P. P. Shirvani and E. J. McCluskey, "Control-Flow Checking by Software Signatures", IEEE Transactions on Reliability, 2002. 8. G. A. Reis, J. Chang, N. Vachharajani, R. Rangan and D. I. August, "SWIFT: Software Implemented Fault Tolerance", International Symposium on Code Generation and Optimization, 2005. Chapter 6 1. J. L. Lions and colleagues, "Ariane 5 Flight 501 Failure: Report by the Inquiry Board", July 1996. 2. A. Avižienis, "The N-Version Approach to Fault-Tolerant Software", IEEE Transactions on Software Engineering, 1985. 3. J. C. Knight and N. G. Leveson, "An Experimental Evaluation of the Assumption of Independence in Multiversion Programming", IEEE Transactions on Software Engineering, 1986. 4. D. E. Eckhardt and L. D. Lee, "A Theoretical Basis for the Analysis of Multiversion Software Subject to Coincident Errors", IEEE Transactions on Software Engineering, 1985. 5. J. R. Garman, "The 'Bug' Heard 'Round the World", ACM SIGSOFT Software Engineering Notes, 1981. 6. NASA Jet Propulsion Laboratory and University of Maryland statements on the end of the Deep Impact mission, September 2013. 7. NASA, statements and briefings on the findings of the Boeing Orbital Flight Test independent review team, 2020. 8. Mars Climate Orbiter Mishap Investigation Board, Phase I Report, November 1999. 9. Australian Transport Safety Bureau, In-flight Upset, 154 km West of Learmonth, WA, 7 October 2008, VH-QPA, Airbus A330-303, Aviation Occurrence Investigation AO-2008-070, final report, 2011. Chapter 7 1. C. L. Liu and J. W. Layland, "Scheduling Algorithms for Multiprogramming in a Hard-Real-Time Environment", Journal of the ACM, 1973. 2. R. Bocchino, T. Canham, G. Watney, L. Reder and J. Levison, "F Prime: An Open-Source Framework for Small-Scale Flight Software Systems", AIAA/USU Conference on Small Satellites, 2018. 3. NASA Goddard Space Flight Center, core Flight System documentation. 4. G. Reeves and T. Neilson, "The Mars Rover Spirit FLASH Anomaly", IEEE Aerospace Conference, 2005. 5. G. J. Holzmann, "The Power of Ten: Rules for Developing Safety-Critical Code", IEEE Computer, 2006. 6. G. J. Holzmann, "Mars Code", Communications of the ACM, 2014. 7. D. L. Dvorak (ed.), "NASA Study on Flight Software Complexity", NASA Office of Chief Engineer, 2009. 8. NASA Jet Propulsion Laboratory, Curiosity mission updates on the post-landing flight software transition, August 2012. Chapter 8 1. NASA, Fault Management Handbook, NASA-HDBK-1002 (draft), 2011–2012. 2. NASA, "Mars Global Surveyor (MGS) Spacecraft Loss of Contact", report of the review board, April 2007. 3. Japan Aerospace Exploration Agency, Hitomi (ASTRO-H) anomaly investigation reports, 2016. 4. NASA Jet Propulsion Laboratory, Mars Reconnaissance Orbiter mission updates on safe mode events, 2009. 5. G. J. Holzmann, The SPIN Model Checker: Primer and Reference Manual, Addison-Wesley, 2003. Chapter 9 1. Mars Program Independent Assessment Team and JPL Special Review Board reports on the loss of Mars Polar Lander, March 2000. 2. European Space Agency, "ExoMars 2016 – Schiaparelli Anomaly Inquiry", May 2017. 3. ispace, announcement of the results of the HAKUTO-R Mission 1 landing sequence analysis, May 2023. 4. Intuitive Machines and NASA statements on the IM-1 landing, February 2024. 5. H. F. Grip, "Surviving an In-Flight Anomaly: What Happened on Ingenuity's Sixth Flight", NASA blog, 27 May 2021. 6. A. E. Johnson and colleagues, papers on the Lander Vision System and terrain-relative navigation for Mars 2020. 7. M. Maimone, Y. Cheng and L. Matthies, "Two Years of Visual Odometry on the Mars Exploration Rovers", Journal of Field Robotics, 2007. 8. R. Francis and colleagues, "AEGIS Autonomous Targeting for ChemCam on Mars Science Laboratory: Deployment and Results of Initial Science Team Use", Science Robotics, 2017. 9. S. Chien and colleagues, "Using Autonomy Flight Software to Improve Science Return on Earth Observing One", Journal of Aerospace Computing, Information, and Communication, 2005. 10. N. Muscettola, P. P. Nayak, B. Pell and B. C. Williams, "Remote Agent: To Boldly Go Where No AI System Has Gone Before", Artificial Intelligence, 1998. 11. K. Havelund, M. Lowry, S. Park, C. Pecheur, J. Penix, W. Visser and J. L. White, "Formal Analysis of the Remote Agent Before and After Flight", Fifth NASA Langley Formal Methods Workshop, 2000. 12. L. Sha, "Using Simplicity to Control Complexity", IEEE Software, 2001. Glossary Common-mode failure. A failure that affects several redundant elements at once because they share a design, input, assumption or environment, defeating the redundancy meant to protect against it. Critical charge. The amount of charge that must be collected at a circuit node to change its logical state; a particle strike depositing more than this at a sensitive node causes an upset. Fault containment region. A part of a system designed so that a fault inside it cannot propagate outside it, and that can be isolated and replaced by a redundant counterpart. Fault protection. Onboard software that monitors a spacecraft for symptoms of failure and carries out predefined responses without help from the ground. Linear energy transfer. The energy a particle deposits per unit length of material, which determines how much charge it frees along its path through a semiconductor. Safe mode. A minimal, stable spacecraft configuration, typically pointed at the Sun and communicating at low rate, entered when fault protection has no more specific remedy. Scrubbing. Periodically reading and correcting the contents of memory, or rewriting FPGA configuration, so that correctable errors do not accumulate into uncorrectable ones. SEC-DED. Single-error-correcting, double-error-detecting code: an error-correcting code that corrects any one flipped bit in a word and detects any two. Single-event effect. Any effect caused by one energetic particle, including upsets, transients, functional interrupts and latch-up. Single-event latch-up. A high-current state triggered in a CMOS device by a particle strike, which persists until power is removed and can destroy the device. Total ionising dose. The cumulative ionising radiation energy absorbed by a device, which gradually degrades its transistors. Triple modular redundancy. Three copies of a module whose outputs are combined by majority vote, so that a fault in any one copy is masked. Further Reading Edward Petersen, Single Event Effects in Aerospace (Wiley-IEEE Press, 2011).** A thorough, practitioner-oriented treatment of single-event physics, testing and rate prediction by one of the field's veterans. Andrew Holmes-Siedle and Len Adams, Handbook of Radiation Effects, second edition (Oxford University Press, 2002). The standard reference on how radiation affects electronic materials and devices, covering total dose and displacement damage as well as single events. Israel Koren and C. Mani Krishna, Fault-Tolerant Systems, second edition (Morgan Kaufmann, 2020). A clear textbook on redundancy, error-correcting codes, software fault tolerance and reliability modelling, with the mathematics of voting worked out in full. Algirdas Avižienis, Jean-Claude Laprie, Brian Randell and Carl Landwehr, "Basic Concepts and Taxonomy of Dependable and Secure Computing", IEEE Transactions on Dependable and Secure Computing, 2004. The definitive statement of the vocabulary of faults, errors and failures that underlies the whole discipline. J. L. Lions and colleagues, "Ariane 5 Flight 501 Failure: Report by the Inquiry Board" (1996). Short, lucid and still the best single document on how redundancy and software design interact. Gerard J. Holzmann, "Mars Code", Communications of the ACM (2014). An insider's account of the methods used to build and verify Curiosity's flight software. Glenn Reeves and Tracy Neilson, "The Mars Rover Spirit FLASH Anomaly", IEEE Aerospace Conference (2005). A frank and detailed reconstruction of a near-loss and its recovery, rich in lessons for anyone designing systems that must restart themselves. NASA, Fault Management Handbook, NASA-HDBK-1002 (draft). The agency's framework for designing and analysing fault management across a mission. David Mindell, Digital Apollo: Human and Machine in Spaceflight (MIT Press, 2008).** A history of the Apollo guidance computer and the debates over how much authority to give software, which anticipate today's questions about autonomy. NASA Goddard core Flight System and JPL F Prime documentation (open source). Working examples of modern flight software architecture, freely available for study. Hashtags: #SpacecraftFlightSoftware #FlightSoftware #SpacecraftSoftware #SpaceSoftwareEngineering #SpacecraftEngineering #AerospaceEngineering #SpaceSystemsEngineering #FaultTolerance #FaultTolerantComputing #RadiationHardening #RadiationHardenedElectronics #RadiationTolerantComputing #SpacecraftReliability #MissionCriticalSoftware #SafetyCriticalSystems #DependableComputing #DeepSpaceMissions #SpacecraftAutonomy #AutonomousSpacecraft #SpacecraftFaultProtection #FaultDetection #FaultIsolation #FaultRecovery #FDIR #SpacecraftSafeMode #AutonomousRecovery #WatchdogTimers #RealTimeOperatingSystems #RTOS #EmbeddedFlightSoftware #EmbeddedSystems #RadiationEffects #SpaceRadiation #CosmicRays #SolarParticleEvents #SingleEventEffects #SingleEventUpsets #SingleEventTransients #SingleEventLatchup #TotalIonizingDose #DisplacementDamage #MultipleBitUpsets #RadiationShielding #RadiationTesting #RadiationQualification #HardwareRootOfReliability #HardwareRedundancy #SoftwareRedundancy #TripleModularRedundancy #TMR #MajorityVoting #LockstepProcessing #StandbyRedundancy #ErrorCorrectingCodes #ECCMemory #SECDED #MemoryScrubbing #MemoryIntegrity #StateProtection #ControlFlowIntegrity #FaultContainment #SoftwarePartitioning #SoftwareVerification #FormalVerification #FlightSoftwareTesting #FaultInjectionTesting #MissionAssurance #SpacecraftProcessors #RAD750 #RAD6000 #LEONProcessors #GR740 #RISCVSpaceComputing #HighPerformanceSpaceflightComputing #CommercialOffTheShelfComputing #COTSInSpace #NASAcoreFlightSystem #FPrimeFramework #JPLSoftwareEngineering #MarsPathfinder #MarsRovers #PerseveranceRover #CuriosityRover #IngenuityHelicopter #VoyagerMission #ApolloGuidanceComputer #Ariane5 #SpacecraftNavigation #AutonomousLanding #TerrainRelativeNavigation #DeepSpaceNetwork #CommunicationLatency #SpaceMissionResilience #SpacecraftRecovery #ReliableEmbeddedSoftware #FutureOfSpaceComputing
- Natural Language Processing in Finance (Sentiment Analysis and Market Prediction)
Download the Book (PDF): Introduction On the afternoon of a Federal Reserve decision, a few hundred words of carefully drafted prose are released at two o'clock Eastern time. Within a second, Treasury futures have moved. Within a minute, equity index futures, the dollar, gold and the implied volatility of options on all of them have repriced. Very often the interest rate decision itself was fully expected, and nothing about it surprised anyone. What moved the markets was a phrase: a qualifier added to a sentence about inflation, a word removed from a description of the labour market, a change of tense in a statement about future policy. Trillions of dollars of assets were revalued on the strength of a handful of edits to a short document. The same thing happens, at smaller scale, thousands of times a quarter. A chief executive on an earnings call answers an analyst's question about margins with a sentence that is technically responsive and conspicuously vague. A bank's annual report adds a paragraph about a regulatory inquiry that was not there last year. A sustainability report promises "alignment" with a net-zero pathway but declines to give a date. Each of these is information. Each is buried in text that runs to tens of thousands of words, released on a schedule, competing for the attention of analysts who cannot read everything. This booklet is about the machinery that now reads it all: natural language processing applied to the documents of capital markets. It covers three arenas where that machinery has done the most work and where its limits are clearest. The first is corporate disclosure, and above all the earnings call, the quarterly ritual in which management explains results and then takes questions from the analysts who cover the firm. The second is central bank communication, where words have themselves become a policy instrument and where the market's reading of a phrase can matter as much as an interest rate decision. The third is sustainability reporting, where a fast-changing regulatory regime has created a flood of semi-structured text that firms must produce and investors must digest, and where automated extraction has moved from experiment to necessity. The controlling argument The booklet's argument can be put in a sentence. Financial text produces tradable information only when it is measured as a surprise against what its speaker would otherwise have said, at the moment it became public, and the hard part of financial NLP is not the language model but that measurement discipline. This runs against the way the field is usually presented. Most accounts are organised around models: the progression from word-counting dictionaries to machine learning classifiers to transformers to large language models, each more capable than the last. That progression is real and it matters. A modern transformer understands that "we do not expect headwinds to abate" is negative even though it contains no negative word, and that "liability" in a balance sheet discussion is not a moral judgement. These are not small gains. But the people who have built profitable text signals, and the academic literature that has tried to replicate them, tend to converge on a less glamorous set of lessons. The speakers in financial markets are not innocent. Chief executives are coached; central bankers draft statements by committee with the explicit aim of steering expectations; sustainability officers write for raters and regulators. Everyone knows they are being parsed. The average tone of a speaker therefore tells you very little, because most of it is the speaker's habit and the rest is the market's expectation. What carries information is the deviation: this executive is less upbeat than she usually is, this committee has dropped a word it has used for six consecutive meetings, this firm's climate section reads differently from the one it filed last year. Measuring deviations correctly requires things that have nothing to do with model architecture. It requires knowing exactly when each document became public, to the minute, and what the market already knew. It requires a baseline for each speaker and each document type. It requires guarding against the fact that a large language model trained on the internet may already know what happened after the document was released, and will cheerfully use that knowledge in a backtest. It requires accepting that a signal which works will be copied, and that its value will decay. A mediocre model with impeccable point-in-time data will beat an excellent model with contaminated data every time, and the excellent model's backtest will look better until the money is real. Why now Three developments make this a good moment for a short, focused treatment. The first is that transformer-based models have made fine-grained reading cheap. A decade ago, extracting the sentiment of each sentence in every earnings call transcript in the American market was a research project. Now it is an afternoon's work with an open-source model, and the frontier has moved to questions that were previously unaskable: did management's answer actually address the question? Has the committee's description of the risks shifted in emphasis even though the words are similar? Does this sustainability report contain a quantified target, and is it consistent with last year's? The second is that the evidence from large language models has accumulated quickly and is more mixed than either enthusiasts or sceptics expected. Studies published since 2023 have shown that general-purpose models, without any financial training, can classify the price impact of news headlines and analyse financial statements with accuracy that rivals human specialists. Other studies have shown that much of this apparent skill can be contaminated by the models' knowledge of what happened afterwards, and that the profits from acting on such signals shrink as more participants act on them. Both sets of results are correct, and the practitioner's job is to understand why. The third is regulatory. Sustainability disclosure has been through an extraordinary period of construction and partial demolition. The International Sustainability Standards Board issued its first two standards in 2023. The European Union built a detailed reporting regime and then, through its 2025 and 2026 simplification package, sharply narrowed its scope. The United States Securities and Exchange Commission adopted a climate disclosure rule in 2024, stopped defending it in court in 2025, and in 2026 moved to rescind it. Anyone building systems to read or produce this material needs to understand what is actually required, by whom, and in what form, because the text a model is asked to read is shaped by those rules. What this booklet covers and what it leaves out The chapters run in a deliberate order. The first two set out how text becomes data, from the early dictionary studies to modern transformers, and what each generation of method gains and loses. The next three take the earnings call as the central case: what is in a call, how to turn it into a signal, and how backtests of such signals deceive. Two chapters then turn to central banks: first what their communication is and why it moves markets, then how transformer models detect the subtle shifts that matter and how to tie those shifts to market prices. The eighth chapter covers sustainability reporting, both the automation of its production and the extraction of signal from it. The ninth addresses what happens when these systems are deployed inside regulated financial institutions, where model risk, explainability and accountability are not optional. The conclusion draws out what follows from the whole. Several adjacent subjects are left out by choice. Social media sentiment and retail-investor forums receive only passing mention; they are noisy, heavily studied, and governed by different dynamics from the institutional documents that are the booklet's focus. Speech and audio analysis appears where it bears on earnings calls but is not treated at length. High-frequency news trading, which is principally an engineering contest about latency, is mentioned only to set its boundaries. Credit, insurance underwriting and anti-money-laundering applications of NLP are substantial fields in their own right and are not covered. The booklet assumes no technical background beyond general familiarity with how financial markets work. Where a method matters, it is explained in words. Where a study is cited, it is a real study, and the notes at the end identify it so that a reader can check the claim. The aim is that a portfolio manager, a risk officer, a policy economist or a sustainability lead should finish it able to ask sharper questions of the systems being sold to them, and of the ones they build themselves. CHAPTER 1 Words as Data: Why Text Moves Prices Financial economics spent most of the twentieth century treating prices and accounting numbers as the data and treating words as commentary on the data. Earnings per share, revenue, book value, dividends and interest rates were measurable, comparable across firms, and amenable to regression. The narrative that accompanied them, in annual reports, press releases, analyst notes and newspaper columns, was assumed either to be redundant with the numbers or to be noise. A market that efficiently incorporated public information would have priced the numbers; the adjectives around them added nothing. That assumption broke down for two reasons, one empirical and one practical. Empirically, researchers found that the words did predict prices, in ways the numbers did not capture. Practically, the volume of text grew while the cost of reading it by machine collapsed, until the question was no longer whether text could be analysed at scale but how to do it without fooling oneself. This chapter sets out the first of these: why words carry information that prices have not yet absorbed, and what that implies for anyone trying to extract it. The first evidence The modern literature is usually dated to Paul Tetlock's 2007 paper in the Journal of Finance, "Giving Content to Investor Sentiment." Tetlock took a single daily newspaper column, the Wall Street Journal's "Abreast of the Market," which summarised the previous day's trading, and measured the proportion of its words that fell into negative categories of the Harvard General Inquirer, a dictionary built by social psychologists in the 1960s for content analysis of all kinds of text. He found that high pessimism in the column predicted downward pressure on the Dow Jones index the following day, followed by a reversal over the next several days. Unusually high or low pessimism also predicted high trading volume. The pattern of a move followed by a reversal was important. It suggested that the column was not conveying fundamental news that the market had missed, because fundamental news would produce a permanent price change. It looked instead like a measure of investor mood, a sentiment shock that pushed prices away from value and then dissipated. Tetlock's interpretation was that media pessimism either reflected or induced temporary selling pressure by noise traders. Three years earlier, Werner Antweiler and Murray Frank had published "Is All That Talk Just Noise?", which analysed about 1.5 million messages posted on Yahoo! Finance and Raging Bull message boards about forty-five large American firms. Using a naive Bayes classifier trained on a hand-labelled subset, they classified messages as bullish, bearish or neutral. Their finding was that message volume predicted volatility and that disagreement among posters was associated with trading volume, while the effect of bullishness on returns was statistically detectable but economically small. These two studies established a template that persisted for a decade: take a body of text, reduce it to a score by counting words or classifying documents, and regress future returns, volume or volatility on the score. They also established the two interpretations that every subsequent finding has had to choose between. Either the text contains information about fundamentals that the market has not yet priced, in which case the effect should be permanent, or the text measures or moves sentiment, in which case the effect should reverse. The distinction matters for a trader, because the first kind of signal can be held and the second must be faded, and it matters for an economist, because the first is evidence of slow information diffusion and the second of behavioural mispricing. Why text should contain unpriced information at all A sceptic's objection is straightforward. If an earnings call transcript is public, and thousands of professional analysts read it, then whatever it says should be in the price within minutes. Why would counting its words predict anything over days or weeks? There are four answers, and each has implications for how text signals should be built. The first is limited attention. Investors, including professionals, cannot process everything. When many firms report on the same day, as they do at the peak of every earnings season, attention is rationed. A large body of research in accounting and finance has documented that the market's reaction to earnings news is slower and more incomplete when investors are distracted, such as on Fridays or on days with many competing announcements. Text is costlier to process than numbers. A headline earnings figure can be compared to the consensus forecast instantly; a fifty-page transcript cannot be read instantly by anyone. The proportion of a document's information that is priced immediately should therefore be lower for text than for numbers, and lower still for long, complex or unusual text. This is the economic rationale for machine reading: a system that processes every document fully and immediately is exploiting the gap between what is public and what has been attended to. The second is soft information. Numbers are hard information: standardised, verifiable, comparable. Much of what matters about a firm is soft: management's confidence, the reasons behind a result, the risks it is worried about, its evasions. Soft information is conveyed in language, and it resists reduction to a number by construction. Markets price hard information more readily than soft, because hard information can be fed into models and contracts. Text analysis is in large part an attempt to harden soft information, turning a manager's hesitation into a score that can enter a regression. The third is gradual diffusion through the investor population. Even if some investors read and understand a document immediately, others act on it later, as they read analyst summaries, hear it discussed or notice price moves. Prices can take days or weeks to reach the level that full information would justify, especially for smaller firms with fewer analysts and less institutional ownership. The well-known post-earnings-announcement drift, documented by Victor Bernard and Jacob Thomas in 1989, is the canonical example: stocks with positive earnings surprises continue to outperform for months. Text-based signals frequently show the same shape, strongest in smaller and less-followed firms, which is consistent with diffusion rather than with the text containing secrets. The fourth is strategic obfuscation. Managers who have bad news have incentives to make it harder to find. They may bury it in long documents, use complex language, or emphasise positive developments. Research on the readability of annual reports, beginning with Feng Li's 2008 work using the Fog index, found that firms with lower earnings tended to have longer and harder-to-read reports, and that the earnings of firms with more readable reports were more persistent. If bad news is deliberately disguised, then a reader who can strip away the disguise has an informational edge, and the disguise itself becomes a signal. Each of these mechanisms predicts something specific. Limited attention predicts that the signal should be stronger when the document arrives on a busy day or is long. Soft information predicts that the signal should be strongest in the parts of a document least tied to reported numbers, such as forward-looking statements and answers to questions. Gradual diffusion predicts stronger effects in small, illiquid, thinly covered firms, which is also exactly where trading costs are highest. Strategic obfuscation predicts that the signal should be asymmetric, stronger for negative information than for positive. A good text signal should exhibit the pattern that its supposed mechanism predicts, and checking for this is one of the most effective guards against a spurious result. What a document is, economically It helps to be precise about what kinds of document populate financial markets, because each has different properties as a data source. Mandatory periodic filings, such as the American 10-K annual report and 10-Q quarterly report, are long, heavily lawyered, and slow to change. Their timing is known in advance. Much of their content is boilerplate repeated year after year. That last property, which might seem to make them useless, turned out to be their most valuable feature for text analysis. In "Lazy Prices," published in the Journal of Finance in 2020, Lauren Cohen, Christopher Malloy and Quoc Nguyen showed that when a firm changes the language of its annual report substantially from the previous year, especially in the risk factors and the management discussion sections, it subsequently underperforms, and that investors were slow to notice the changes. The signal was not the content of the language but the fact of change. Because firms tend to keep their filings unchanged unless they must alter them, a change is a costly signal that something has happened. Event-driven disclosures, such as the 8-K current reports and press releases announcing earnings, acquisitions, executive departures or litigation, are shorter and arrive unpredictably or on a known date with unknown content. They are where most of the immediate price reaction to corporate news occurs. Earnings calls are a hybrid. They are scheduled, and the prepared remarks are scripted, but the question-and-answer session that follows is partially spontaneous. Analysts ask what they want to know; managers must answer in real time, with limited ability to retreat into legal language. Accounting research has found that the question-and-answer portion is generally more informative than the prepared remarks. Earnings calls also carry vocal information, which can be analysed separately from the words. They are the central case in this booklet, examined in detail in the third and fourth chapters. News and commentary, from newswires such as Reuters, Bloomberg and Dow Jones to newspapers and specialist publications, report on firms and markets from outside. They mix hard news (a contract has been signed) with interpretation (analysts are concerned). Their timestamps are critical and often unreliable in historical archives, a problem examined in the fifth chapter. Central bank communication, including policy statements, minutes, transcripts, speeches, testimony and press conferences, is a distinct category with its own economics. Central banks are not firms trying to manage their share price; they are public institutions trying to steer the economy partly through the expectations their words create. The sixth and seventh chapters treat them separately. Sustainability and ESG reports, increasingly regulated, sit between mandatory filings and marketing material. Their content is shaped by a shifting set of standards and by the criteria of rating agencies. They are the subject of the eighth chapter. Social media and forums are voluminous, fast and noisy. They matter for retail-driven episodes, as the events around GameStop in January 2021 demonstrated, but they are outside this booklet's focus on institutional documents. From single firms to the whole economy Text measures are not confined to individual stocks. Some of the most influential uses of financial text have built aggregate indicators of conditions that no price or statistic captured directly. Scott Baker, Nicholas Bloom and Steven Davis, in a 2016 paper in the Quarterly Journal of Economics, constructed an index of economic policy uncertainty by counting the share of articles in major American newspapers that contained terms relating to the economy, to policy and to uncertainty together. The method was almost crude, a co-occurrence count with careful human auditing of samples to check that the articles flagged really did discuss policy uncertainty. Yet the index rose sharply around events such as close presidential elections, the 2011 debt-ceiling dispute and major wars, and it was associated with reduced investment and hiring in policy-sensitive sectors. It is now published for many countries and is used by central banks, international organisations and investors as a standard macroeconomic indicator. Asaf Manela and Alan Moreira, in a 2017 paper in the Journal of Financial Economics, went further back. Using the front-page text of the Wall Street Journal from 1890 onward, they trained a model to predict the implied volatility of options in the period when options data existed, and then used it to estimate a news-implied volatility index for the century before. The index allowed them to study how concerns about rare disasters, such as wars, varied over a long history and to show that such concerns predicted subsequent stock returns. Leland Bybee, Bryan Kelly, Asaf Manela and Dacheng Xiu, in a paper published in the Journal of Finance in 2024 titled "Business News and Business Cycles," applied topic modelling to the full text of the Wall Street Journal over several decades and showed that the attention the newspaper devoted to different themes, such as recessions, monetary policy, or particular industries, tracked and helped explain economic activity and asset prices. The news, in their framing, is a compressed record of what the economy's participants were paying attention to. These aggregate measures share an important property with the firm-level signals that are the main subject of this booklet. They work because they measure attention and concern that the official statistics capture with a lag or not at all. Newspapers write about uncertainty before it appears in investment data; they write about disasters before they occur, or about the fear of them. The economic content of text lies, here as elsewhere, in its capacity to register what is happening before the numbers do. The surprise principle Every one of these document types shares a property that dominates how they should be analysed. Their authors know they are being read, and their readers know what the authors usually say. The informational content of any statement is therefore what it tells the reader beyond what the reader already expected. This is familiar in the case of numbers. An earnings figure of a dollar per share is good news if analysts expected ninety cents and bad news if they expected a dollar ten. Nobody would build a trading signal on the raw level of earnings per share. Yet text signals have very often been built on the raw level of tone: the proportion of positive words, the average sentiment score of sentences. This is the equivalent of trading on the level of earnings. The expectation against which a text should be measured has several components. There is the speaker's baseline: some executives are habitually effusive and others habitually guarded, so a neutral tone means different things from each. There is the document-type baseline: prepared remarks are systematically more positive than answers to questions, and risk factor sections are systematically negative. There is the industry and macroeconomic baseline: in a recession, every firm's language darkens, and a firm that sounds only slightly gloomy may be signalling relative strength. And there is the market's expectation formed from other information: if the numbers released an hour before the call were poor, a gloomy tone on the call is no surprise. Measuring text against these baselines is more work than measuring its level, and it requires history: one cannot compute a speaker's usual tone without a record of past speech, aligned correctly in time. But the gain is large and consistent across the literature. Changes in tone predict returns better than levels; tone relative to peers predicts better than absolute tone; tone surprises relative to what the numbers implied predict better than tone alone. The "Lazy Prices" result is the purest example: it measures nothing but change, and it works because the baseline for a 10-K is its own previous edition. Central banks illustrate the principle even more sharply. The Federal Reserve's statement is drafted so that each word is chosen deliberately and previous wording is retained unless the committee intends a change. Markets do not read the statement fresh each time; they read the differences from the last one. A financial journalist's first act on receiving a new statement is to produce a redline against the old one. The informational content is in the edits. Prices as the final arbiter A last point frames everything that follows. In finance, unlike most applications of language technology, there is an external ground truth against which any reading of text can be judged: the subsequent behaviour of prices. A sentiment model for product reviews is evaluated by agreement with human labels. A sentiment model for earnings calls can also be evaluated that way, but its real test is whether its scores predict returns, volatility or analyst revisions out of sample. This is both a blessing and a trap. It is a blessing because it imposes discipline: a model that sounds sophisticated but does not predict anything is worthless, and the market will say so. It is a trap because prices are noisy and the number of ways to construct a text signal is enormous. With enough choices of dictionary, model, aggregation rule, holding period and universe, some combination will predict returns in any historical sample by chance. The more flexible the model, the more severe this problem. Large language models, which can be prompted in endlessly different ways, are the most flexible tools ever applied to the problem, and they inherit this hazard in its most severe form. The rest of this booklet can be read as a sustained attempt to get the benefit of the market's verdict without falling into its trap. That requires understanding the tools, which is the work of the next chapter; understanding the documents, which is the work of the chapters on earnings calls, central banks and sustainability reports; and understanding the ways a backtest can lie, which is a subject important enough to receive a chapter of its own. CHAPTER 2 From Dictionaries to Transformers: The Measurement Toolkit Every method for extracting information from financial text makes the same basic move: it maps a document, or a part of one, onto a number or a small set of numbers. The methods differ in how much of the text's structure they preserve on the way. Early methods kept almost nothing, treating a document as a bag of words whose order did not matter. Modern methods keep a great deal, representing each word in the light of every other word around it. The progression is genuinely one of increasing capability, but each step has also introduced new costs, new failure modes and new ways to deceive oneself. A practitioner choosing a method is choosing among those trade-offs, not simply picking the newest tool. Dictionaries and the problem of financial meaning The simplest method counts words from predefined lists. A document's negative tone is the proportion of its words that appear on a negative list; its positive tone, the proportion on a positive list. Tetlock's 2007 study used the Harvard General Inquirer, whose lists were built for general social science content analysis. The trouble with general-purpose dictionaries became clear in a 2011 paper by Tim Loughran and Bill McDonald in the Journal of Finance, titled "When Is a Liability Not a Liability?" They examined the words in the Harvard negative list as they appeared in a large sample of 10-K filings and found that almost three-quarters of the negative-word occurrences were words that are not negative in a financial context. "Liability" is an accounting term, not a misfortune. "Tax," "cost," "capital," "expense" and "foreign" carry no negative valence in a filing. "Vice" appears overwhelmingly as part of "vice president." A dictionary that counts these words measures, in large part, the industry and accounting structure of the firm rather than its tone. Loughran and McDonald built their own word lists from the language of financial filings: negative, positive, uncertainty, litigious, strong modal and weak modal, later extended with a list for constraining language. Their lists are maintained and updated and remain the most widely used dictionary in financial research. They showed that their negative list correlated more strongly with returns around filing dates than the Harvard list did, and they introduced a weighting scheme, borrowed from information retrieval, that down-weights words appearing in many documents and so reduces the influence of common terms. Dictionaries have virtues that are easy to underrate once more powerful tools are available. They are transparent: anyone can see which words drove a score. They are stable: the same document always receives the same score, and the score does not depend on a training set. They require no labelled data. They are fast. Most importantly for the problems examined later, they cannot know anything about the future, because they were built from a fixed list. A dictionary score computed on a 2005 document today is exactly what it would have been in 2005. Their weaknesses are equally clear. They ignore negation, so "not profitable" counts as positive. They ignore context, so "we reduced costs" and "costs increased" are scored the same if only "costs" is listed. They treat each occurrence of a word as equally important. And they cannot recognise sentiment conveyed without sentiment words: "we are reviewing strategic alternatives for the division" contains no negative word but is often bad news. Researchers developed patches, such as flipping polarity within a few words of a negator, but the patches never fully closed the gap. Supervised learning and the problem of labels The natural next step was to learn the mapping from text to score from data. Given a set of documents with known outcomes, a classifier can learn which words or phrases predict which outcome. There are two broad choices of outcome to learn. The first is human judgement: annotators label sentences as positive, negative or neutral, and the model learns to reproduce their labels. The most widely used labelled dataset for financial sentiment, the Financial PhraseBank introduced by Pekka Malo and colleagues in 2014, consists of several thousand sentences from financial news about Finnish and other companies, each labelled by multiple annotators with backgrounds in finance. The second choice is market reaction: the model learns which words in a document are associated with positive or negative returns around its release. Narasimhan Jegadeesh and Di Wu took the second approach in a 2013 paper, "Word Power," in the Journal of Financial Economics. Rather than assuming that every negative word is equally negative, they estimated each word's weight from the market's reaction to 10-K filings, effectively letting prices decide what each word meant. Zheng Tracy Ke, Bryan Kelly and Dacheng Xiu extended this idea in a method they called SESTM, for sentiment extraction via screening and topic modelling, which they applied to Dow Jones newswire articles. Their approach first screened for words that were statistically associated with returns, then estimated how strongly each word loaded on positive and negative sentiment, and finally scored new articles. The resulting signal predicted returns over the following days, with the effect stronger for smaller and more volatile stocks and for articles that arrived when markets were closed. Learning from returns has a powerful appeal: it optimises directly for the thing one cares about. It also carries a serious risk. Returns are noisy, so the learned word weights are noisy, and a model trained on a particular period's returns may learn associations specific to that period. The out-of-sample discipline required is severe, and the method only works when the training sample is long and the vocabulary is constrained. Classical machine learning classifiers, such as naive Bayes, logistic regression and support vector machines, all represent a document as a vector of word or phrase counts and learn weights on them. They improve on dictionaries by learning context-specific meanings from data, but they still discard word order and cannot, for example, distinguish "the decline in revenue was smaller than expected" from "revenue was smaller than expected, a decline." Embeddings and the geometry of meaning The step that made modern NLP possible was to represent words not as entries in a list but as points in a space, where proximity reflects similarity of meaning. Methods such as word2vec, introduced by Tomas Mikolov and colleagues at Google in 2013, learn these representations from the contexts in which words occur in large corpora: words that appear in similar contexts receive similar vectors. In such a space, "revenue" sits near "sales" and "turnover," "headwinds" near "pressures" and "challenges." Embeddings let a model generalise. A classifier trained on sentences containing "headwinds" can recognise that a sentence about "pressures" is likely similar even if it never saw that word in training. They also enable similarity measurement between whole documents: by combining word vectors, one can measure how much this quarter's call resembles last quarter's, or how similar one firm's risk disclosures are to another's. Static embeddings have a crucial limitation. Each word has one vector regardless of context. "Bank" has the same representation in "central bank" and "river bank," and "charge" the same in "impairment charge" and "charge ahead." For financial text, where many words have both everyday and technical meanings, this limitation is significant. Transformers The transformer architecture, introduced by Ashish Vaswani and colleagues at Google in the 2017 paper "Attention Is All You Need," solved the context problem. Its central mechanism, attention, allows the representation of each word in a passage to be computed as a weighted combination of information from every other word, with the weights themselves learned. The representation of "charge" in "a non-cash impairment charge" is built by attending to "impairment" and "non-cash" and so differs from its representation in other sentences. Stacking many layers of attention allows the model to build up representations that capture syntax, negation, reference and a good deal of what humans would call meaning. BERT, released by Google researchers led by Jacob Devlin and published in 2019, applied the transformer to produce general-purpose language representations. It was pretrained on a large general corpus by learning to predict masked words from their context, and could then be fine-tuned for a specific task, such as sentiment classification, with a modest amount of labelled data. The pretraining gave it a broad understanding of language; the fine-tuning specialised it. Financial versions followed quickly. Dogu Araci's FinBERT, released in 2019, fine-tuned BERT on financial text and the Financial PhraseBank and showed substantial gains in sentiment classification accuracy over earlier methods. A separate and more thoroughly validated FinBERT was developed by Allen Huang, Hui Wang and Yi Yang at the Hong Kong University of Science and Technology. They pretrained a BERT model from scratch on billions of tokens of financial communication, combining corporate filings, earnings call transcripts and analyst reports, and published their evaluation in Contemporary Accounting Research in 2023. They found that their FinBERT substantially outperformed the Loughran-McDonald dictionary and a range of classical and deep learning methods on sentiment classification, that its advantage was largest when labelled training data were scarce, and that other approaches underestimated the informativeness of earnings call text by a meaningful margin relative to FinBERT. Both FinBERT models are publicly available and have become the default first tool for many researchers and practitioners. Large language models The most recent step has been the arrival of large generative language models, built on the same transformer foundation but scaled to tens or hundreds of billions of parameters and trained not only to represent text but to generate it in response to instructions. These models can perform tasks they were never explicitly trained on, given only a description in natural language: "Read this earnings call excerpt and state whether management's guidance has become more or less optimistic relative to the prior quarter, and quote the sentence that supports your judgement." Bloomberg built a domain-specific model, BloombergGPT, described in a 2023 paper by Shijie Wu and colleagues. It had about fifty billion parameters and was trained on a mixture of a large proprietary financial corpus assembled from Bloomberg's archives and general-purpose text. It performed well on financial benchmarks relative to general models of similar size at the time. The broader lesson of the subsequent two years, however, was that general-purpose frontier models, trained at much larger scale and without a special financial focus, matched or exceeded specialised financial models on most financial language tasks. Scale and general reasoning ability proved more valuable than domain-specific pretraining, at least for tasks that did not depend on proprietary data. Two studies illustrate what these models can do. Alejandro Lopez-Lira and Yuehua Tang, in a paper first circulated in April 2023 and revised several times since, asked ChatGPT and later GPT-4 to judge whether news headlines about companies were good, bad or irrelevant for the stock price, and found that the model's judgements predicted next-day returns, with predictive power increasing with model size and concentrated in smaller stocks and negative news. Alex Kim, Maximilian Muhn and Valeri Nikolaev, in a 2024 paper from the University of Chicago, gave GPT-4 standardised and anonymised financial statements, without narrative text, and asked it to predict whether earnings would rise or fall in the following year. Using chain-of-thought prompting, which asks the model to reason step by step, it predicted direction with an accuracy of roughly sixty percent, above the median analyst forecast and comparable to specialised machine learning models trained for the task. Both results are striking and both come with caveats examined later in the booklet. For now the important point is that the frontier of capability has moved from classifying sentiment to performing something closer to analysis: comparing, reasoning, extracting structured facts and explaining a judgement. Representations rather than labels One of the more important shifts in recent research has been away from using language models to produce labels, such as positive or negative, and toward using them to produce representations, the internal numerical encodings of a text that the models compute on the way to any output. Yifei Chen, Bryan Kelly and Dacheng Xiu, in a working paper titled "Expected Returns and Large Language Models" circulated from 2022 and presented widely since, compared these two uses directly. They took news articles about companies in a range of international equity markets and fed them through several language models. In one approach, they asked for a sentiment judgement. In the other, they extracted each model's internal representation of the article, a vector of many hundreds or thousands of numbers, and trained a separate model to predict returns from those vectors. They found that the representation-based approach predicted returns substantially better than the sentiment-label approach, and that larger and more capable language models produced more useful representations. The intuition is that a label throws away almost all of the information a model has extracted. An article about a pharmaceutical company's trial results, a supply contract, or an executive departure contains much that bears on the stock price beyond whether it is good or bad: the size of the news, its relevance to future cash flows, its novelty, the kind of uncertainty it creates. A sentiment label reduces all of that to one of three categories. A representation preserves it, and a predictive model trained on representations can learn which aspects matter. The cost is interpretability. A label is easy to explain; a vector of a thousand numbers is not. And a model trained on representations to predict returns is exposed to all the overfitting risks of learning from returns described earlier, now in a very high-dimensional setting. The approach demands long samples, careful regularisation and strict out-of-sample testing. Numbers inside text A further limitation of language models deserves mention because financial documents are dense with numbers. Most language models break text into sub-word units called tokens, and they tokenise numbers in ways that can obscure their magnitude: a figure such as 1,234.5 may be split into several tokens that bear no simple relation to its value. Early models were notoriously unreliable at arithmetic and at comparing quantities, which matters when the meaning of a sentence depends on whether revenue grew by 2 percent or 20 percent, or whether a target of 42 percent is more or less ambitious than last year's 40. Frontier models have become much better at numerical reasoning, especially when allowed to reason step by step or to call an external calculator. But in production systems it remains good practice to extract numbers from text into structured fields and perform comparisons and calculations outside the language model, using ordinary code. Tables embedded in filings, which models often read poorly when converted to plain text, are best parsed with dedicated tools and supplied to the model in structured form. A language model is an excellent reader of prose and a fallible accountant. Choosing among the tools The choice among these methods is not a choice of the most advanced. It depends on the task, the data available, the need for transparency (the clarity of why a score was given), and the risk of contamination. The main generations of method differ on these dimensions as Table 1 sets out. Table 1. Generations of financial text method compared. Method Context use Labels needed Clarity Look-ahead risk Dictionary None None High None Classical ML Word counts only Moderate Moderate Low Static embeddings Word similarity Low to moderate Low Low to moderate Fine-tuned BERT Full passage Low Low Depends on pretraining date Large language model Full document None to low Very low High The last column needs emphasis because it is the one most often ignored. A dictionary built in 2011 knows nothing about the 2020 pandemic. A BERT model pretrained on text up to 2018 knows nothing after 2018, which makes it safe for backtests on later data but potentially contaminated for earlier periods if its pretraining corpus included news about those periods. A large language model trained on internet text up to a recent date knows a great deal about what happened to most listed companies, including events after any document one might ask it to read. The fifth chapter examines what this means for backtesting. There is also a practical point about cost and latency. A dictionary scores a million documents in minutes on a laptop. A fine-tuned BERT model can process a large transcript archive overnight on a modest graphics processor. A frontier large language model, accessed through a commercial interface, costs real money per document and takes seconds per call. For a research project scoring a decade of transcripts once, this is affordable. For a production system scoring every news story in real time, it may not be, and the economics often favour using a large model to label a training set and then distilling its judgements into a smaller, faster, locally hosted model. Measuring the right thing A final point bridges this chapter and the next. Whatever tool is used, it produces a score for a piece of text, and that score is only as useful as the question it answers. Most early work asked "how positive is this text?" That question, as the first chapter argued, is usually the wrong one. Better questions are comparative and specific: how does this text differ from what this speaker said last time? Did the answer address the question asked? Has the description of a particular risk become more or less severe? Is a quantified commitment present, and has it changed? The great advantage of modern models is that they can be asked these better questions directly. A dictionary can only count. A transformer can compare two passages and say how their meaning differs. A large language model can be asked whether a specific commitment has been weakened and to quote the evidence. The capability exists; the discipline of asking the right question, and of measuring the answer at the right moment against the right baseline, is what turns it into information. CHAPTER 3 The Anatomy of an Earnings Call Of all the documents that corporate finance produces, the earnings conference call is the richest source of information for machine reading. It is regular, arriving every quarter on a date announced in advance. It is comparable, following a nearly universal structure across firms and countries. It is long enough to contain substantial soft information, often eight to twelve thousand words for a large company. And it contains a segment, the question-and-answer session, in which senior executives must speak without a script to professional sceptics whose job is to find what management would prefer not to discuss. To extract signal from a call, one needs to understand how it is produced, what its parts are for, and what the incentives of each participant are. A model that treats a transcript as an undifferentiated block of text throws away most of what makes it valuable. How a call is made A typical American earnings call follows the release of quarterly results by a press release, usually after the market closes or before it opens. The call itself starts some minutes or hours later. It opens with an investor relations officer reading a safe harbour statement, a legal warning that forward-looking statements involve risks and uncertainties. The chief executive then delivers prepared remarks covering strategy and the quarter's highlights, followed by the chief financial officer on the detailed financial results and guidance for the coming period. This scripted portion typically runs fifteen to thirty minutes. The question-and-answer session follows. An operator invites analysts from banks and research firms to ask questions, usually one or two each, and executives answer. This portion runs from twenty minutes to an hour. Calls end with brief closing remarks. Since Regulation Fair Disclosure was adopted by the Securities and Exchange Commission in 2000, companies in the United States have been prohibited from selectively disclosing material information to some investors before others. One consequence is that earnings calls became publicly accessible, usually by webcast, and transcripts became widely distributed. Commercial providers now produce transcripts within hours, sometimes minutes, of a call's end, and maintain archives stretching back to the early 2000s for thousands of companies worldwide. The prepared remarks are drafted, reviewed by lawyers, and often rehearsed. Investor relations teams prepare executives for likely questions, and many firms hold practice sessions. Executives therefore arrive at the question-and-answer session with prepared positions on the questions they expect. What they cannot fully prepare for is the unexpected question, the follow-up, or the analyst who presses on an answer that did not satisfy. The two halves of a call Accounting research has consistently found that the two halves of a call carry different information. Dawn Matsumoto, Maarten Pronk and Erik Roelofsen, in a 2011 study in The Accounting Review, found that both the presentation and the discussion sections of calls had incremental information content beyond the earnings release, but that the discussion section was relatively more informative, and more so when the firm's performance was poor. Their interpretation was that analysts' questions draw out information management would not volunteer. This suggests a basic rule for any text system: score the halves separately. The prepared remarks reflect what management wants to say, and their tone mostly reflects management's chosen framing. The question-and-answer session reflects what management is pushed to say, and deviations from the framing there are more informative. A single sentiment score averaged across the whole call blends a heavily managed signal with a less managed one, diluting the second. Within the question-and-answer session, a further distinction matters: questions versus answers. Analysts' questions are themselves informative. Their tone reflects the analysts' own views, which incorporate their private research, and the topics they choose reveal what the market is worried about. A call in which several analysts ask about the same issue, such as inventory build-up or customer concentration, is signalling that the issue is salient. Separating analyst speech from management speech is therefore essential. Fortunately, transcripts identify speakers, and the structure is regular enough to parse reliably. Families of signal Once a call is segmented by section and speaker, there are several distinct families of information to extract. They draw on different parts of the call and different methods, and they capture different economic content. The main families, as studied in the research literature, are summarised in Table 2. Table 2. Families of signal extracted from earnings calls. Family What it captures Where in the call Representative study Tone Net positivity of language Both halves, scored apart Price et al. (2012) Uncertainty Hedging, vagueness, modal words Mainly answers Loughran and McDonald lists Evasion Non-answers, deflection Answers to questions Hollander et al. (2010) Deception cues Linguistic markers of misstatement Executive speech Larcker and Zakolyukina (2012) Vocal affect Emotion in the voice Audio of answers Mayew and Venkatachalam (2012) Topic exposure Attention to a risk or theme Whole call Hassan et al. (2019) Tone The most direct signal is tone. S. McKay Price, James Doran, David Peterson and Barbara Bliss, in a 2012 study in the Journal of Banking and Finance, found that the tone of earnings calls, measured with a dictionary adapted to finance, significantly explained abnormal returns around the call and over the following two months, beyond the earnings surprise itself, with the question-and-answer session particularly informative. Many subsequent studies have confirmed and refined this result with better dictionaries and transformer models. The refinement that matters most is measuring tone as a surprise. A firm whose executives are habitually upbeat conveys little by being upbeat again; one that turns cautious conveys a great deal. Tone change relative to the same firm's previous call, or tone relative to what the accompanying numbers would predict, is a consistently stronger signal than tone level. Uncertainty and hedging A second family measures how certain management sounds. Words and phrases such as "may," "could," "approximately," "we believe," "it is difficult to predict" and "subject to" signal hedging. Uncertainty language is informative in two ways. It can indicate genuine uncertainty about the business, which predicts higher future volatility and wider dispersion in analyst forecasts. And a sudden increase in hedging, particularly when management had previously been specific, can indicate that management knows something it is reluctant to commit to. Transformer models handle hedging much better than dictionaries do, because hedging is often syntactic and contextual. "We expect margins to improve" and "we would hope to see margins begin to improve, depending on how the second half develops" express very different levels of conviction, and only the second contains obvious hedge words. A fine-tuned model or an instructed large language model can score the degree of commitment in a forward-looking statement more faithfully than any word count. Evasion A third family captures evasion: answers that do not answer. Stephan Hollander, Maarten Pronk and Erik Roelofsen, in a 2010 paper in the Journal of Accounting Research titled "Does Silence Speak?", examined cases in which managers declined to answer analysts' questions on earnings calls. They found that non-answers were common, that they were more likely when the information would be proprietary or unfavourable, and that investors interpreted silence negatively. Explicit refusals ("we don't break that out," "we're not going to comment on that") are relatively easy to detect with rules. More subtle evasion, in which an executive responds at length without addressing the question, was beyond dictionary methods but is well within reach of modern models. A system can embed the question and the answer and measure their semantic alignment, or ask a large language model directly whether the answer addressed the question and how completely. Analysts do this intuitively; it is one of the clearest cases where machine reading now replicates a skilled human judgement at scale. The signal is sharpest when evasion concerns a topic of known importance. An executive who answers questions about revenue growth fully but deflects questions about gross margin is telling the market where the problem lies. Deception cues A fourth family attempts something more ambitious: detecting when management is not telling the truth. David Larcker and Anastasia Zakolyukina, in a 2012 study in the Journal of Accounting Research, built models of deceptive speech using linguistic features drawn from psychological research on lying, and trained them on earnings calls of firms that later restated their financial statements, labelling the calls covering the restated periods as potentially deceptive. They found that deceptive executives used more references to general knowledge ("you know," "everybody knows"), fewer non-extreme positive emotion words, fewer references to shareholder value, and more extreme positive emotion words. Their models classified deceptive calls significantly better than chance, and a portfolio formed on the deception scores earned negative abnormal returns. Two cautions apply. The labels are noisy, since a restatement does not prove that any particular executive lied on any particular call. And performance well above chance is still far below the reliability needed to accuse anyone of anything. Deception cues are best understood as a risk indicator, a reason to look harder, rather than a detector. Vocal affect A fifth family moves beyond the words to the voice. William Mayew and Mohan Venkatachalam, in a 2012 study in the Journal of Finance titled "The Power of Voice," applied vocal emotion analysis software to the audio of earnings calls and found that managers' positive and negative affect, as expressed in their voices during the question-and-answer session, predicted future firm performance and was associated with analyst recommendation revisions and stock returns. Vocal signals are partially independent of word choice: an executive can say confident words in a strained voice. Audio is costlier to obtain and process than transcripts, and the commercial voice-stress tools of the early 2010s were scientifically questionable. Modern speech models that can analyse pitch, pace, hesitation and disfluency have renewed interest in the area. Hesitations and filled pauses ("um," "uh") are typically removed from commercial transcripts but can be recovered from audio, and they carry information about cognitive load. Topic exposure A sixth family measures not how management speaks but what it speaks about, and how much. Tarek Hassan, Stephan Hollander, Laurence van Lent and Ahmed Tahoun, in a 2019 study in the Quarterly Journal of Economics, measured firm-level political risk by counting the share of each earnings call devoted to political topics, identified by bigrams more common in political texts than in general financial texts, appearing near words indicating risk or uncertainty. Their measure captured variation in political risk across firms that aggregate measures missed, and it predicted firms' hiring and investment decisions. The same authors, with others, later applied the method to measure firm-level exposure to Brexit, to the coronavirus pandemic and, in work led by Zacharias Sautner published in the Journal of Finance in 2023, to climate change. Topic exposure measures are less about predicting short-term returns and more about building firm-level characteristics that were previously unobservable. They answer questions such as which firms are most exposed to tariffs, or to interest rate risk, or to a particular supply chain disruption, by asking what management and analysts actually discuss. They are among the most productive uses of earnings call text in academic economics and are increasingly used in practice for risk management and thematic portfolio construction. Who is speaking, and to whom The incentives of each participant shape what their words mean, and a careful system models them. Chief executives speak to sell a strategy and maintain confidence. Their language is typically the most positive and the most managed. Chief financial officers speak about numbers and guidance and are typically more cautious and precise; a shift in the chief financial officer's tone may be more informative than one in the chief executive's, because the chief financial officer's baseline is less inflated. Analysts speak to extract information and, sometimes, to signal their own views or curry favour. Research has found that analysts with closer relationships with management tend to ask softer questions, and that firms sometimes favour friendly analysts in the order of questions. A call in which the questions are unusually soft may be a managed call. A call in which an analyst with a known bearish view is not called upon, or is cut off, may be telling. The audience is also strategic. Management knows that its call will be parsed by machines. Sean Cao, Wei Jiang, Baozhong Yang and Alan Zhang, in a paper published in the Review of Financial Studies in 2023 titled "How to Talk When a Machine Is Listening," found that firms with more machine readership of their filings, measured by automated downloads from the SEC's servers, subsequently avoided words that the Loughran-McDonald dictionary considered negative. They found evidence that some managers also adjusted their vocal tone on calls. This is the strategic speaker problem in its sharpest form: once a measure is widely used, the people being measured adapt to it, and the measure degrades. It is a strong argument for moving from fixed dictionaries to context-sensitive models, which are harder to game, and for measuring changes and evasions, which are harder to disguise than word choice. Calls beyond the American market Most of the research cited in this chapter uses calls by American companies, because the American transcript archive is the deepest and the market the most studied. Applying the same methods elsewhere raises issues that a careful system must handle. The first is language. Many companies outside the English-speaking world hold their calls in English for an international investor audience, with executives speaking a second language. Second-language speech differs systematically from first-language speech: vocabulary is narrower, idiom is rarer, hedging may be expressed differently, and disfluencies may reflect language processing rather than discomfort with the subject. A model calibrated on American chief executives may read a Japanese or German executive's careful, formal English as evasive or uncertain when it is simply careful. This is a strong argument for the speaker-level baselines discussed in the first chapter. Measured against the same executive's own past calls, a change in hedging is informative regardless of the speaker's fluency; measured against an American norm, it is confounded. Other companies hold calls in their own language, with transcripts produced in that language and sometimes translated. Multilingual transformer models can score text in many languages, but their quality varies, and financial vocabulary in less widely represented languages may be poorly covered. Translated transcripts introduce a further layer of interpretation between the speaker and the model. The second issue is format and frequency. In many markets, quarterly reporting is not mandatory or is lighter in content than in the United States, and many firms hold calls only semi-annually. The structure varies too: some markets have a tradition of large in-person results meetings, some of separate briefings for analysts and for media. In mainland China, regulators and exchanges have promoted online investor interaction platforms on which retail investors pose written questions and companies post written answers, creating a large body of question-and-answer text with quite different dynamics from an analyst call. The third issue is institutional context. The incentives that shape what executives say depend on legal liability, the culture of disclosure, the composition of the shareholder base and the role of analysts. Findings that hold in the United States, such as the relative informativeness of the question-and-answer session, may hold elsewhere, but they should be tested rather than assumed. The call as part of an information sequence A last point ties this chapter to the next. An earnings call does not arrive in isolation. It follows the earnings press release, often by minutes or hours, which has already conveyed the headline numbers. It follows months of analyst forecasts and guidance from previous calls. It precedes a burst of analyst reports and revisions. The market's reaction to the call is therefore a reaction to the call's information beyond what the press release, the forecasts and the previous calls contained. This has a direct consequence for signal construction. To measure the information in a call, one must control for the information in the numbers. A call whose tone is gloomy because the quarter was bad conveys nothing beyond the numbers. A call whose tone is gloomy despite a good quarter, or whose question-and-answer session reveals worries the prepared remarks concealed, conveys a great deal. The next chapter turns this principle into a pipeline. CHAPTER 4 From Transcript to Trade: Building the Signal A sentiment score is not a trading signal. Between a model that can read an earnings call and a portfolio that profits from what it reads lies a sequence of decisions, each of which can add or destroy value, and most of which have nothing to do with language. This chapter walks through that sequence as a practitioner would build it, with attention to where the information is actually created and where it tends to leak away. The sequence has six stages: acquiring the text with correct timing, segmenting it, scoring it, converting scores into surprises, combining surprises into a cross-sectional signal, and trading that signal net of costs. A seventh concern, whether the whole thing is real or an artefact of the backtest, is important enough to deserve the next chapter to itself. Stage one: the text and its clock The first and most consequential decision concerns time. For every document in the historical archive, the system needs to know precisely when it became available to a trader who could have acted on it. This is not the date of the call. It is the moment at which the transcript, or the live audio, could have been processed. For earnings calls, there are at least three relevant moments. The call itself takes place at a known time, and a trader listening live, or running real-time speech recognition, could act during it. The first transcript from a commercial provider arrives later, sometimes within an hour, sometimes the next day, depending on the provider and the period. Corrected or final versions may arrive later still, with speaker attributions fixed and errors removed. A backtest that uses the final, clean transcript but assumes it was available at the moment the call ended is giving itself information it would not have had. The problem is worse in older data. Vendors have improved their speed over time, so the lag between call and transcript was longer in the 2000s than it is now. Archives may record only the call date, not the transcript delivery time. Some archives have been back-filled with transcripts obtained later from other sources. A backtest that assumes uniform availability across a twenty-year sample is likely overstating the early years. The conservative practice is to assume that the signal can only be traded from the open of the trading day after the transcript's delivery, using the delivery timestamp where it is recorded and a generous lag where it is not. If the signal survives that assumption, it is worth investigating further. If it exists only when the trade is assumed to occur within minutes of the call, it is a latency game, and the relevant question is whether the system can compete with participants who are listening live. The text itself also needs attention. Transcripts contain errors, especially in speaker attribution and in names and technical terms. Different vendors transcribe differently. A model trained on one vendor's transcripts may score another's differently, simply because of stylistic conventions in how hesitations, crosstalk or numbers are rendered. Switching vendors mid-sample, or mixing them, can introduce artificial breaks in a signal's history. Stage two: segmentation As the previous chapter argued, a call should be split into its parts before it is scored. The minimum useful segmentation separates the safe harbour statement, the prepared remarks, the analyst questions and the management answers, and attributes each management utterance to a role: chief executive, chief financial officer, or other executive. Transcripts usually label speakers by name and sometimes by title, and an operator's announcement typically marks the start of the question-and-answer session. Rule-based parsing handles most of this, supplemented by a lookup of executive names and titles from a corporate officer database. Cases where parsing fails, such as transcripts without clear section markers or with misattributed speakers, should be flagged and either fixed or dropped, not silently scored as if they were clean. Segmentation also enables a more targeted kind of scoring. Pairing each analyst question with the management answer that follows allows question-answer alignment to be measured, which is the basis for detecting evasion. Grouping utterances by topic, such as revenue, margins, capital expenditure, demand, pricing and competition, allows the system to track how management's language on each topic changes over time. This topic-level resolution is where modern models earn their keep: a general sentiment score might show no change between quarters, while the tone on margins has darkened and the tone on revenue brightened by an offsetting amount. Stage three: scoring The choice of model, discussed in the second chapter, comes in here. In practice, a robust system usually computes several scores in parallel rather than relying on one. A dictionary score using the Loughran-McDonald lists provides a transparent, contamination-free baseline. Any more sophisticated score should be compared to it; if a transformer model adds nothing beyond the dictionary in out-of-sample tests, the transformer's complexity is not earning its keep. A fine-tuned transformer such as FinBERT, applied at the sentence level and aggregated, provides context-sensitive sentiment. Sentence-level scoring followed by aggregation is generally more reliable than feeding long passages into a model with a limited input length and letting it truncate. A large language model, given a structured instruction, can produce scores for more nuanced attributes: the degree of confidence in guidance, whether each answer addressed its question, whether new risks were introduced, whether management's explanation of a result is consistent with the explanation given previously. The output should be constrained to a fixed format, such as a number on a defined scale with a short justification, so that it can be parsed reliably and audited. Every score should be computed on the segment level and stored with full provenance: which model, which version, which prompt, which date. Models change. A commercial large language model accessed through an interface may be updated without notice, so that the same prompt on the same transcript produces a different answer next month. For a production system, this is unacceptable unless the model version can be pinned, and the safer course is often to use the large model to generate labels and then train a smaller, fixed model that the firm controls. Stage four: from level to surprise This is the stage where the surprise principle, set out in the first chapter, becomes concrete. The raw scores are levels. The signal needs to be a deviation from what was expected. The simplest transformation is the change from the same firm's previous call. If the question-and-answer tone this quarter is lower than last quarter's, that is a negative surprise. The change can be scaled by the firm's historical variability in tone, so that a modest drop from a habitually stable speaker counts for more than the same drop from a volatile one. A richer transformation compares the firm to its peers on the same reporting cycle. If every retailer's tone dropped this quarter because consumer spending weakened, a retailer whose tone dropped by less is conveying relative strength. Subtracting the industry average change isolates the firm-specific component. The most important transformation removes the part of tone that is explained by the numbers. On any given call, tone is strongly driven by the earnings surprise, the revenue surprise and any change in guidance that the press release has already announced. Regressing tone on these quantities, across firms in the same period, and taking the residual produces a measure of what the call's language says beyond the numbers. This residual tone is the piece that the market is most likely to underreact to, because it is soft information that requires reading rather than comparison against a forecast. Consider a stylised illustration. A mid-sized industrial company reports earnings slightly above consensus and reaffirms its annual guidance. The press release is mildly positive and the stock rises modestly in after-hours trading. On the call, the prepared remarks are upbeat. In the question-and-answer session, three analysts ask about order backlog, and the chief financial officer's answers become progressively less specific, moving from a figure to a range to a statement that the company does not provide backlog by segment. The tone of management's answers is lower than on any of the previous eight calls, and the alignment between questions and answers on the backlog topic is poor. None of this was in the press release. A system that measures tone level for the whole call might score this call as neutral or slightly positive. A system that measures residual answer tone, topic-specific change and evasion would flag it clearly. The hypothesis behind the signal is that the market, anchored on the numbers and the upbeat prepared remarks, will be slow to absorb what the question-and-answer session revealed. Stage five: from surprise to signal A set of firm-level surprises becomes a trading signal when it is converted into portfolio weights. The usual approach in equity markets is cross-sectional: at each point in time, rank all firms that have recently reported by their text surprise, and go long the most positive and short the most negative. Several practical choices matter here. Because earnings calls cluster in reporting seasons, the number of firms with fresh signals varies dramatically through the year, from hundreds per day at the peak to almost none between seasons. A portfolio that holds positions for a fixed period after each call will therefore be heavily invested during and just after reporting season and nearly empty between them. Practitioners handle this by holding positions for longer, by combining the text signal with other signals that are available continuously, or by accepting the seasonality and sizing accordingly. The text signal should also be neutralised against known risk factors. A text signal that is merely tilted toward small stocks, or value stocks, or a particular industry, is reproducing exposures that can be obtained more cheaply in other ways. Neutralising the portfolio against market beta, size, value, momentum and industry, as is standard in quantitative equity, isolates the component of returns attributable to the text. Finally, text signals are usually more valuable as a component of a broader model than in isolation. A large quantitative manager will combine earnings call text with price momentum, analyst revisions, accounting quality measures and many other inputs. The question is not whether the text signal is profitable on its own but whether it adds predictive power beyond everything else the model already uses. Many text signals that look strong alone turn out to overlap substantially with analyst revisions, because analysts read the same calls and revise their forecasts accordingly. The value of the text then lies in its speed: it captures the information the analysts will act on, but days earlier. Listening live The pipeline described so far works on transcripts. A different and more competitive approach works on the call as it happens. Modern automatic speech recognition, such as the open-source Whisper models released by OpenAI in 2022 and their many successors, can transcribe English financial speech with accuracy that approaches commercial human transcription, and can do so in near real time. A system that listens to the webcast, transcribes it continuously, and scores each utterance as it is spoken can act during the call rather than after it. The economics of live listening are different from those of transcript-based signals. The edge, if any, lies in minutes rather than days, and it is contested by every other participant with the same capability, which by now includes many large quantitative firms and data vendors. The information that can be extracted in real time is also narrower: a live system can detect a change in guidance, an unexpected admission, or a shift in tone as it occurs, but the more considered measures, such as comparison with the full history of a speaker's calls or careful alignment of questions and answers, are easier to compute once the call is complete. There is a further complication. Commercial providers now distribute machine-generated call summaries, highlights and sentiment scores within minutes of a call's end, and many investors consume these rather than reading or listening to the call. When a large share of the market reads the same summary, produced by similar models and emphasising the same points, the information in those points is priced quickly and uniformly. The remaining edge lies in what the standard summaries omit: the subtle evasion, the topic-specific shift, the inconsistency with a statement made three quarters ago. A practitioner should ask, of any signal, whether it measures something that the widely distributed summaries already capture. If it does, its value is likely to be small and short-lived. The sensible division for most firms is to treat live listening as a risk management tool, alerting portfolio managers to significant developments in names they hold, and to build alpha signals on the fuller analysis that is possible after the call, where the competitive advantage of careful design is greatest and the race for latency is irrelevant. Stage six: costs, capacity and decay A signal that predicts returns is profitable only if the predicted returns exceed the cost of trading. Text signals, as the first chapter noted, tend to be strongest in smaller, less liquid, thinly covered firms, which is exactly where bid-ask spreads and market impact are greatest. A backtest that ignores trading costs, or assumes costs typical of large liquid stocks, can show substantial profits that vanish when realistic costs are applied. Capacity follows from costs. A signal concentrated in small firms can be traded profitably at small scale but not at large scale, because the manager's own trades move prices. A text signal might support a modest dedicated strategy but be immaterial for a large fund. Decay follows from competition. Once a signal is known, and especially once it is published, more participants trade on it, and its profitability falls. The evidence for this is general across published anomalies: David McLean and Jeffrey Pontiff, in a 2016 study in the Journal of Finance, found that the returns of published cross-sectional predictors fell substantially after publication. Text signals are not exempt. Lopez-Lira and Tang, in later versions of their study of language model predictions from news, reported that the profitability of the strategy declined as adoption of such models increased, which is what one would expect from a market that learns. Decay has a further implication specific to text. As the previous chapter noted, firms adapt their language when they know machines are reading. Dictionary-based negativity measures have been partially gamed already. The signals likeliest to retain value are those that are hardest to manipulate: evasion, which is visible in the mismatch between question and answer; change, which requires a firm to alter its language to avoid it; and inconsistency between what management says now and what it said before, which a firm cannot fix retroactively. The honest summary Stated plainly, the evidence supports the following. Earnings call text contains information about future returns, analyst revisions and fundamentals that is not fully captured in the accompanying numbers. The information is concentrated in the question-and-answer session, in changes rather than levels, and in firms that receive less attention. Modern transformer and language models extract it better than dictionaries, especially for nuanced attributes such as evasion and hedging. The returns available from trading on it are real but modest after costs, concentrated in less liquid stocks, and declining as more participants exploit them. That is a more sober picture than vendors of text analytics tend to present. It is also, for a careful practitioner, an encouraging one. It says that the edge lies not in having a language model, which everyone now has, but in the discipline of the pipeline: getting the timing right, measuring surprises rather than levels, asking the questions that are hard to game, and being honest about costs. These are things that cannot be bought with a subscription. Whether one has actually achieved them, rather than appearing to in a backtest, is the subject of the next chapter. CHAPTER 5 Backtests That Lie Every text signal described in the published literature, and every one sold by a data vendor, comes with a backtest: a simulation of how a strategy based on the signal would have performed historically. Backtests are indispensable, because there is no other way to evaluate a signal before risking money on it. They are also the most dangerous tool in quantitative finance, because they reward exactly the errors that are hardest to see. A backtest that is wrong in the optimistic direction looks like a discovery. One that is wrong in the pessimistic direction looks like a dead end and is abandoned. The selection is ruthless and one-sided. Text signals are especially exposed to backtest error, for three reasons. Their data have complicated timing, so look-ahead errors are easy to commit and hard to detect. Their construction involves many discretionary choices, so the scope for overfitting is large. And the most powerful modern tools, large language models, introduce a form of contamination that did not exist before 2020 and that standard backtesting practice was not designed to catch. This chapter takes each in turn. Timing errors The previous chapter introduced the problem of knowing when a document became available. It is worth being precise about the forms the error takes, because each requires a different defence. The first form is timestamp error: the archive records a date or time for a document that is earlier than when it could actually have been acted upon. News archives are notorious for this. A story first published in the evening may be recorded with the date of the following morning's print edition, or vice versa. A story that was updated several times may be recorded with the timestamp of its first version but the text of its last. Transcript archives may record the call date rather than the delivery time. The defence is to use vendor-recorded delivery timestamps wherever possible, to verify a sample manually against independent sources, and to impose a conservative lag where timestamps are uncertain. The second form is revision error: the text in the archive is not the text that was available at the time. Transcripts are corrected after first release. News stories are updated with new information. Filings are amended. Even company identifiers change: a firm that was acquired, renamed or delisted may be recorded in the archive under its later identifier, which implicitly encodes knowledge of what happened to it. The defence is to obtain point-in-time versions of documents and identifiers, which some vendors now offer, and to treat any archive that provides only final versions with suspicion. The third form is universe error, better known as survivorship bias. If the historical archive of transcripts contains only firms that still exist, or is weighted toward them, then the backtest is run on a population that excludes firms that went bankrupt or were delisted. Those firms are disproportionately the ones whose calls had negative signals, and they are the ones on which a short position would have been most profitable or, if the firm was acquired at a premium, most painful. Excluding them distorts both sides. The defence is to build the universe from point-in-time constituent lists that include firms that later disappeared. These three errors are not new and are well understood in quantitative finance. They become more acute with text because text archives are less mature than price databases and because text vendors have historically paid less attention to point-in-time accuracy. They are mostly a matter of careful engineering, and a team that takes them seriously can largely eliminate them. Overfitting and the garden of forking paths The second category of backtest error is harder to eliminate, because it arises from the researcher's own choices. Building a text signal requires dozens of decisions. Which dictionary or model? Which section of the call? Which speakers? Sentence-level or document-level scoring? Level, change, or residual? Change from last quarter or from the average of the last four? Relative to industry or to the market? Over what holding period? With what universe filters? With what neutralisations? Each decision is defensible, and each slightly changes the backtest. A researcher who tries many combinations and reports the best one has, without any dishonesty, produced a result that overstates the signal's true strength. The more combinations tried, the more the best one is likely to owe its performance to chance. This problem has been studied extensively. Campbell Harvey, Yan Liu and Heqing Zhu, in a 2016 paper in the Review of Financial Studies titled "... and the Cross-Section of Expected Returns," catalogued hundreds of published return predictors and argued that, given the number of hypotheses that had been tested across the profession, the conventional threshold for statistical significance was far too low. They proposed that a newly discovered factor should clear a t-statistic of about three, rather than the conventional two, to be taken seriously. David Bailey and Marcos López de Prado proposed the deflated Sharpe ratio, which adjusts a strategy's risk-adjusted return for the number of trials that preceded its selection, the non-normality of its returns and the length of its track record. For text signals, the scope for forking paths is wider than for most quantitative signals because text is so rich. With large language models it becomes wider still, because the prompt itself is a free parameter. Asking a model whether a call is "positive or negative" yields different scores from asking whether it is "bullish or bearish," or whether management is "confident or uncertain," or whether "the stock is likely to go up or down." Each prompt can be phrased many ways, and the model's output is sensitive to phrasing. A researcher who tries twenty prompts and reports the one with the best backtest has run twenty experiments, not one. The defences are procedural. The most effective is to commit to a specification before looking at its performance: to decide, on economic or linguistic grounds, which section, which model, which transformation and which holding period make sense, and to test that specification once. The second is to reserve a genuine holdout sample, a period of data that is not touched during development and is used only for the final evaluation. The third is to record every specification tried, so that the multiple-testing adjustment can be honest. The fourth is to require that the signal exhibit the pattern its economic mechanism predicts, as described in the first chapter: stronger in less-attended firms, stronger for negative news, decaying at a plausible rate. A signal that works but shows none of the predicted patterns is more likely to be an accident. The contamination problem The third category of error is specific to pretrained language models, and it is the most important development in financial backtesting methodology in recent years. A large language model is trained on an enormous corpus of text gathered up to some cutoff date. That corpus includes news articles, analyst commentary, Wikipedia entries, forum discussions and books, which together describe in considerable detail what happened to most large companies over most of the period covered. When such a model is asked to read an earnings call from 2015 and judge whether management's tone was optimistic, it is not a reader in 2015. It is a reader that has, in some compressed and imperfectly accessible form, read about everything that happened to that company from 2015 to its training cutoff. This creates two distinct problems, which Paul Glasserman and Caden Lin distinguished clearly in a 2023 paper titled "Assessing Look-Ahead Bias in Stock Return Predictions Generated by GPT Sentiment Analysis." The first is look-ahead bias in the strict sense. The model may have specific knowledge of what happened after the text: that the stock fell, that the chief executive resigned, that the company restated its earnings. That knowledge can leak into its judgement of the text's sentiment, consciously or not. A model asked to rate the tone of Enron's 2001 calls knows what happened to Enron. The second is what Glasserman and Lin called the distraction effect. The model's general knowledge of a company, even without specific knowledge of subsequent events, may interfere with its reading of a particular text. A model with strong prior associations about a company may rate a neutral headline about it in light of those associations rather than on the text alone. To separate these effects, Glasserman and Lin compared strategies based on GPT sentiment scores of original news headlines with strategies based on the same headlines with company identifiers removed. Within the model's training window, they found, surprisingly, that the anonymised headlines performed better than the originals, suggesting that the distraction effect outweighed look-ahead bias in their setting. The effect was stronger for larger companies, about which the model would have more general knowledge. Out of sample, after the training window, look-ahead bias by definition cannot arise, but distraction can. The anonymisation result does not mean look-ahead bias is unimportant. It means that the two biases can work in opposite directions in some settings, and that the net effect in any particular application is an empirical question. Other researchers have found clearer evidence of leakage. Suproteem Sarkar and Keyon Vafa, in work presented at the International Conference on Machine Learning in 2025, examined how a pretrained model analysing earnings calls could reflect knowledge of events after the calls, and found evidence that model outputs on pre-pandemic transcripts drew on risks whose salience only became apparent later. Their broader argument was that the standard practice of instructing a model to "use only information available as of" a given date does not work: the model cannot reliably forget what it knows. Defences against contamination Several defences are available, none complete. The cleanest is to evaluate only out of sample: to backtest only on data from after the model's training cutoff. This removes look-ahead bias entirely. Its cost is that, for recent frontier models, the post-cutoff period may be only months long, too short for a meaningful evaluation of a quarterly signal. And because each new model has a later cutoff, the clean evaluation window must be reset each time the model is upgraded. The second is anonymisation: removing company names, ticker symbols, executive names, product names and other identifiers before the model reads the text. This reduces both look-ahead bias and distraction, as Glasserman and Lin showed, though it cannot remove everything. A sufficiently capable model may recognise a company from its description, its industry and its numbers. Anonymising an earnings call fully is harder than anonymising a headline, because calls are long and full of identifying detail. The third is to use models with known and early training cutoffs for historical backtests. A BERT-family model pretrained on data up to 2018, and fine-tuned only on data from before the backtest period, is far less exposed than a frontier model trained on recent data. Researchers have begun building language models specifically for this purpose: families of models each trained only on text available up to a particular date, so that a backtest for any year can use a model that genuinely did not know the future. Songrun He, Linying Lv, Asaf Manela and Jimmy Wu have described such "chronologically consistent" models, and the approach is likely to become standard practice for serious research. The fourth is to use the large model only for tasks that do not depend on outcome knowledge, such as extracting structured facts from text, and to perform the predictive step with a separate model trained point-in-time on those facts. Extracting the stated revenue guidance range from a transcript is a task that knowledge of the future cannot much contaminate. Judging whether the tone was optimistic is a task that it can. A taxonomy of backtest errors The errors described in this chapter differ in where they arise, how they inflate results and how they can be detected, as summarised in Table 3. Table 3. Common backtest errors in financial text signals. Error Source Effect on results Main defence Timestamp Archive dates too early Inflates short horizons Delivery timestamps, lags Revision Final text, not first Inflates all horizons Point-in-time versions Survivorship Missing failed firms Distorts long and short Point-in-time universe Forking paths Many specifications tried Inflates significance Pre-commitment, holdout Look-ahead Model knows the future Inflates in-sample Post-cutoff testing Distraction Model priors about firm Noise, either direction Anonymisation Costs ignored Frictionless trading Inflates net returns Realistic cost model The list is not exhaustive, but it covers the errors most commonly found when a promising text signal fails in live trading. Several of them compound. A signal concentrated in small firms, with short holding periods, using news timestamps of uncertain reliability and scored by a large language model on in-sample data, is exposed to nearly every error on the list at once. That combination is, unfortunately, also a natural description of many of the most exciting results reported in recent years. Auditing a claimed signal Most investors encounter text signals not by building them but by being offered them, by data vendors, brokers or research providers, each with a backtest. The errors described in this chapter translate into a set of questions that any buyer should ask, and the quality of the answers is usually more revealing than the backtest itself. The first questions concern timing. What timestamp does the backtest use for each document, and is it the time of the event, the time of the vendor's first delivery, or something else? Was the vendor's delivery latency in early years the same as now, and if not, how is the difference handled? At what point is the trade assumed to execute? A vendor who cannot answer these precisely has probably not considered them. The next concern versions and universe. Are the documents in the backtest the versions that were available at the time, or later corrected ones? Does the universe include companies that were subsequently delisted, acquired or went bankrupt? Were any documents back-filled into the archive after the fact? Then come questions about construction. How many variants of the signal were tested before this one was chosen? When was the specification fixed, and how much data has accumulated since? Is there a period of genuine live or out-of-sample performance, and how does it compare with the backtest? A signal whose live performance is a fraction of its backtest is the norm rather than the exception; the question is whether the fraction is large enough to be useful. For any signal using a pretrained language model, the buyer should ask about contamination. Which model, with what training cutoff? How much of the backtest falls before the cutoff? Were inputs anonymised? Has the model been changed during the backtest period, and if so, was the history rescored? Finally, there are questions about economics. What is the signal's correlation with standard factors and with analyst revisions? Where in the capitalisation spectrum is its return concentrated? What trading cost assumptions were used, and what happens to net performance under more conservative ones? At what level of capital does the signal's own market impact consume its returns? A useful habit is to request the signal's performance separately on the largest and most liquid stocks in the universe. If the signal is strong there, it is likely to be robust and tradable. If its performance is concentrated entirely in the smallest, least liquid names, it may be real but is unlikely to be exploitable at scale, and the headline backtest overstates its practical value. The live test Ultimately, the only backtest that cannot lie is the one run with real money, in real time, on data that did not exist when the signal was designed. This is sometimes called paper trading or incubation when done without capital, and live trading when done with it. Every serious quantitative manager requires a period of incubation before committing significant capital to a new signal, and the period is longer for signals with more scope for backtest error. For text signals, incubation also tests the operational pipeline: whether transcripts arrive when expected, whether the parsing handles the inevitable irregularities, whether the model's outputs are stable, and whether the signal's behaviour in live data resembles its behaviour in the backtest. Many signals fail at this stage not because the economic idea was wrong but because the historical data were cleaner than the live feed, so that the backtest measured a signal that could not be reproduced in practice. The lesson of this chapter is not that text signals are illusory. The academic evidence, much of it carefully constructed and replicated, is that earnings call text, central bank language and other financial documents contain real information. The lesson is that the gap between a backtest and a realisable return is wider for text than for most signals, and that the most capable modern tools widen it further unless they are used with specific precautions. A team that understands this, and builds its process around it, is in a much stronger position than one that simply has a better model. CHAPTER 6 Reading the Central Bank When a company's chief executive speaks, the market listens for information about the company. When a central bank speaks, the market listens for information about the price of money itself, which enters the valuation of every asset. This alone would make central bank communication the most consequential text in finance. But there is a further feature that sets it apart. A company's words describe what the company is doing. A central bank's words are, to a significant degree, what the central bank is doing. Over the past three decades, communication has become a policy instrument in its own right, used deliberately to shape the expectations that drive long-term interest rates, exchange rates and asset prices. To read a central bank is to read policy as it is being made. This chapter sets out what central bank communication is, why its words move markets, and what features of it make it both well suited and treacherous for machine reading. The next chapter turns to the methods. From secrecy to guidance For most of its history, the Federal Reserve said very little about its decisions. Until 1994, the Federal Open Market Committee did not announce changes in its policy stance; markets inferred them from the Fed's operations in the money markets. The view, widely held among central bankers, was that a degree of mystery enhanced the central bank's effectiveness, or at least protected its flexibility. That view was abandoned in stages. In February 1994, the committee issued its first statement announcing a policy decision immediately after a meeting. Over the following years it began to issue statements after every meeting, to include an assessment of the balance of risks, and to name an explicit target for the federal funds rate. Minutes of each meeting, once released only after the following meeting, were from December 2004 published three weeks after the meeting at which the decision was made. Full transcripts are released with a five-year lag. The decisive turn came in 2003, when the committee, worried about deflation, stated that policy accommodation could be maintained "for a considerable period." This was a statement not about the current decision but about future ones, and it was designed to lower long-term interest rates by lowering expected short-term rates. The phrase became an object of intense scrutiny. When the committee eventually replaced it in early 2004, first with language indicating it could be "patient" and then with a statement that accommodation could be removed "at a pace that is likely to be measured," markets treated each change as a policy action. After policy rates reached zero in December 2008, words became nearly the only instrument available beyond asset purchases. The committee said rates would remain exceptionally low "for an extended period," then in August 2011 named a date, "at least through mid-2013," then extended that date, and in December 2012 tied its guidance to economic thresholds, saying rates would stay low at least as long as unemployment remained above six and a half percent and inflation projections stayed contained. Each formulation was crafted with care and debated within the committee, and each was analysed word by word by the markets. Other central banks followed similar paths. The European Central Bank adopted explicit forward guidance in July 2013. The Bank of England adopted threshold-based guidance in August 2013. The most famous single sentence in modern central banking was spoken by Mario Draghi, then president of the European Central Bank, in London on 26 July 2012: that within its mandate the bank was ready to do "whatever it takes" to preserve the euro. The bond yields of Spain and Italy fell sharply over the following weeks, before the bank had bought a single bond under the programme it subsequently announced. Words had done the work. Why the words move markets The economic logic is that long-term interest rates are, to a first approximation, averages of expected future short-term rates plus a premium for risk. A central bank that sets the short-term rate today can influence long-term rates, and hence mortgage rates, corporate borrowing costs and equity valuations, only by changing what markets expect it to do in the future. Communication is how it does that. The empirical evidence for the power of words is strong and was established before any machine reading was involved. Refet Gürkaynak, Brian Sack and Eric Swanson, in a 2005 paper in the International Journal of Central Banking titled "Do Actions Speak Louder Than Words?", studied the high-frequency reaction of financial markets in narrow windows around FOMC announcements. They showed that the market response could not be explained by a single factor, the surprise in the current rate decision. A second factor, which they called the path factor and associated with the committee's statements about the future course of policy, was needed. They found that this second factor accounted for most of the movement in longer-term Treasury yields around announcements. In other words, for long-term rates, what the Fed said mattered more than what it did. Eric Swanson extended this analysis in a 2021 paper in the Journal of Monetary Economics, identifying three dimensions of FOMC announcements over the period 2009 to 2015: changes in the federal funds rate, forward guidance, and large-scale asset purchases. Forward guidance and asset purchase announcements each had large and distinct effects on Treasury yields, corporate bond yields, stock prices and exchange rates. There is also evidence of anticipation. David Lucca and Emanuel Moench, in a 2015 study in the Journal of Finance, documented that American equities earned unusually large excess returns in the twenty-four hours before scheduled FOMC announcements over a long sample period, a pattern they termed the pre-FOMC announcement drift. The phenomenon shows how much attention and risk the market concentrates on these few hours each year. The communication calendar Central bank communication is not a single document but a structured sequence of documents, each with its own timing, author and function. A machine reading system must treat them separately, because they differ greatly in what they reveal and when. The Federal Reserve's main channels, which are broadly paralleled at other major central banks, are set out in Table 4. Table 4. Main channels of Federal Reserve communication. Channel Timing Author What it reveals Statement Decision day, 2:00 pm ET Committee, by vote Decision and guidance Projections Quarterly, with statement Each participant Rate and economic paths Press conference Decision day, 2:30 pm ET Chair Reasoning, emphasis Minutes Three weeks later Staff, approved Range of views Speeches Between meetings Individual officials Personal views, debate Transcripts Five years later Verbatim record Full deliberation The statement is the most compressed and most carefully drafted text. It typically runs to a few hundred words, follows a standard structure, and changes incrementally from meeting to meeting. Each word is voted on. Changes are deliberate, and the market reads the redline against the previous statement before anything else. The statement is the ideal object for difference-based analysis. The Summary of Economic Projections, published four times a year, includes each participant's projection for the policy rate, displayed anonymously as a scatter of points that markets call the dot plot. It is numerical rather than textual, but it provides a quantitative check on the statement's language. The press conference, held after every meeting since 2019 and after some meetings from April 2011, is where the chair explains the decision and answers journalists' questions. Like the earnings call's question-and-answer session, it is partly spontaneous, and like that session it is often where the market learns most. The chair's choice of emphasis, the readiness or reluctance to rule out options, and responses to unexpected questions can move markets substantially. The press conference transcript is longer and less controlled than the statement, and its analysis calls for the same segmentation by question and answer described in the third chapter. The minutes describe the discussion at the meeting, including the range of views. They are drafted by staff and approved by the committee, and they reveal the balance of opinion in ways the statement does not, for example by reporting that "many" or "several" or "a few" participants held a particular view. Those quantifiers have a conventional ordering that markets read carefully. Speeches and testimony by individual officials, outside the blackout period that surrounds each meeting, convey personal views and are often used to prepare markets for a shift that the committee is moving toward. Because individual officials differ in their views, speeches must be attributed and weighted: a speech by the chair or by a voting member signals more than one by a non-voting participant with known outlier views. Transcripts of meetings, released after five years, are of no use for trading but are invaluable for research into how the committee deliberates. When the market hears something else Central bank communication does not always produce the reaction its authors intended, and the episodes where it fails are as instructive as those where it succeeds. The clearest example is the so-called taper tantrum of 2013. On 22 May 2013, in testimony before the Joint Economic Committee of Congress, Chairman Ben Bernanke said in response to a question that if the economy continued to improve, the committee could take a step down in its pace of asset purchases within the next few meetings. The committee had discussed reducing purchases and the remark was consistent with its internal deliberations; it was not a commitment. But markets heard a signal that the extraordinary support of the post-crisis years was ending sooner than they had assumed. Long-term Treasury yields rose sharply over the following months, by roughly a full percentage point, and emerging market currencies and bonds sold off heavily. The committee did not in fact begin to reduce purchases until December of that year. The episode carries a lesson for machine reading. The meaning of a central bank's words, for market purposes, is not what the institution intended but what the market inferred, and the inference depends on what the market previously believed. Bernanke's remark was conditional and modest in its literal content. A model scoring its hawkishness in isolation might have rated it only mildly hawkish. Its impact came from its contrast with an expectation that the market had formed, perhaps wrongly, that purchases would continue for longer. This is the surprise principle again, in its starkest form: no reading of the text alone could have predicted the scale of the reaction without a model of what the market expected. The episode also shaped subsequent communication. Central banks learned that significant changes needed to be prepared over many months, through speeches, minutes and testimony, so that no single statement would carry an unexpected shift. That preparation is part of why the content of speeches between meetings has become so informative, and why monitoring them systematically, as the next chapter describes, matters. Transparency changes the speech The release of transcripts provided one of the most elegant studies in the machine reading of central bank text, and an important lesson about strategic speakers. Until 1993, most FOMC members did not know that verbatim transcripts of their meetings were being kept. In late 1993, following congressional pressure, it became known that they were, and the Fed subsequently decided to publish them with a lag. Stephen Hansen, Michael McMahon and Andrea Prat, in a 2018 study in the Quarterly Journal of Economics titled "Transparency and Deliberation Within the FOMC," used this as a natural experiment. They applied topic modelling, a machine learning method that identifies themes in a body of text, to transcripts from before and after the change. They found that after members knew their words would become public, less experienced members came to meetings better prepared, citing more data, which the authors interpreted as a discipline effect. They also found that those members became more likely to conform to the chair's views and less likely to challenge them, a conformity effect. Transparency improved the quality of individual contributions and reduced the diversity of debate. The lesson for machine reading is that central bank language, like corporate language, is shaped by the speaker's awareness of being read. As central banks have become more transparent and more conscious that their words are parsed by machines, their language has become more standardised, more cautious and more carefully coordinated. Changes in it have become rarer and more deliberate, which makes each change more informative and the baseline more stable. What makes central bank text special for machines Several features distinguish central bank communication from corporate text as an object of machine reading. The first is extreme incrementalism. The FOMC statement changes by small amounts from meeting to meeting, and the changes carry most of the information. A model that reads each statement independently and scores its tone will produce nearly identical scores for consecutive statements and miss the difference that matters. The natural unit of analysis is the change. The second is a specialised and stable vocabulary. Central banks use phrases with established meanings: "accommodative," "restrictive," "data-dependent," "gradual," "patient," "some further policy firming." These terms carry a precise significance within the institution's discourse that general-purpose sentiment models do not capture. "Patient" is not a sentiment word; in a 2014 or 2019 FOMC statement, it signalled the committee's intention not to raise rates soon. Models need either training on central bank text specifically or careful instruction. The third is that the relevant sentiment dimension is not positive versus negative but hawkish versus dovish: whether the language implies tighter or looser policy. A statement can be pessimistic about the economy and dovish, or optimistic and hawkish, or, confusingly, optimistic and dovish if the committee is emphasising that it will not tighten even as growth improves. General sentiment scores conflate these dimensions. The fourth is conditionality. Central bank guidance is almost always conditional: rates will stay low as long as certain conditions hold, or the committee will adjust policy as appropriate if risks emerge. The market's interpretation depends on its view of how likely the conditions are, which the text itself does not supply. A sentence promising patience "as long as inflation remains well anchored" means different things depending on whether inflation is two percent or five. The fifth is multiple speakers with different weights. A committee speaks with one voice in its statement but with many in its speeches and minutes. Measuring the committee's direction from individual speeches requires knowing who is a voter, who has historically been hawkish or dovish, and how close each speaker is to the chair. The sixth, and most important, is the size and speed of the market's reaction. Because so much capital is positioned around central bank announcements, the information in a statement is reflected in prices within seconds. There is no slow diffusion of the kind that makes earnings call signals profitable over weeks. The value of machine reading of central bank text therefore lies less in predicting returns over days than in three other things: reading faster and more accurately than other participants in the seconds after release, building measures of the policy stance that can be used as inputs to macroeconomic and asset allocation models, and tracking the gradual evolution of views through speeches and minutes that markets may digest less efficiently than statements. The next chapter turns to how transformer models address these features, and how their outputs can be tied to the market's own reading of the words. CHAPTER 7 Detecting Subtle Shifts with Transformers The previous chapter described central bank communication as a sequence of carefully drafted documents in which the information lies in small, deliberate changes. This chapter is about how to detect those changes by machine, how to measure their direction and size, and how to verify that what the machine detects is what the market reacts to. It is the part of financial NLP where transformer models have made the clearest difference, because the task is almost entirely one of context: the same word can signal tightening in one sentence and nothing in another, and the significance of a phrase depends on what it replaced. Why level-based sentiment fails The first generation of automated analysis of central bank text applied general sentiment dictionaries or simple word lists to statements and minutes. The results were weak, for the reasons the previous chapter set out. Central bank language is neither positive nor negative in the ordinary sense; it is hawkish or dovish. A description of strong employment growth is positive in tone and, in most circumstances, hawkish in implication. A statement that the committee "will be patient" contains a mildly positive word and is dovish. A general sentiment model scores the surface and misses the policy content. Early efforts to fix this built specialised word lists. David Lucca and Francesco Trebbi, in a 2009 working paper, measured the content of FOMC statements by comparing their phrases to the way those phrases were used in news coverage associated with hawkish or dovish interpretations, and found that changes in the statement's language explained movements in Treasury yields at longer maturities more than the rate decisions did. Other researchers built lists of hawkish and dovish terms by hand. These approaches captured the vocabulary but not the syntax: they could not distinguish "inflation has risen" from "inflation has not risen," or "the committee is prepared to adjust" from "the committee sees no need to adjust." Clause-level stance classification The transformer approach replaces word lists with models that read whole sentences and classify their stance. The most thorough public effort is a 2023 study by Agam Shah, Suvan Paturi and Sudheer Chava at the Georgia Institute of Technology, published at the annual meeting of the Association for Computational Linguistics under the title "Trillion Dollar Words." They assembled FOMC meeting minutes, press conference transcripts and speeches from 1996 to late 2022 and had annotators label a sample of about two and a half thousand sentences as hawkish, dovish or neutral. One design decision in that study illustrates the character of central bank language particularly well. The authors observed that Fed communication routinely pairs a statement of policy direction with a qualifying counterpoint in the same sentence: inflation has moderated, but remains elevated; the labour market is strong, although some indicators suggest cooling. A single label for such a sentence loses information, because its two halves point in opposite directions. They therefore split sentences at contrasting connectives such as "but," "however" and "although," and labelled each part separately. Among the models they tested, a fine-tuned RoBERTa-large, a transformer of the BERT family, performed best by a clear margin, outperforming both rule-based dictionary methods and a zero-shot general-purpose chat model of the time. They then used the classifier to build a document-level hawkishness measure across their whole corpus and validated it against economic data: the measure was positively correlated with consumer and producer price inflation and had a statistically significant relationship with Treasury yields across maturities. They also reported that a simple trading strategy based on the measure outperformed a buy-and-hold position in a technology-heavy equity index over their sample, though, for the reasons set out in the fifth chapter, such a result is best treated as an illustration of the measure's information content rather than as a deployable strategy. The broader lesson is that a modest amount of carefully labelled, domain-specific data, combined with a pretrained transformer, produces a stance classifier markedly better than general-purpose tools. Frontier large language models have since improved on zero-shot performance considerably, and a well-instructed large model with a handful of labelled examples in its prompt can now approach the performance of a fine-tuned classifier. But the fine-tuned model has advantages in a production setting: it is fixed, cheap to run, fast, and its behaviour does not change when a vendor updates its service. Measuring change rather than level Stance classification produces a hawkishness score for each sentence or document. The information, as argued throughout this booklet, lies in the change. There are several ways to measure it, and they capture different things. Redlining and edit classification The most direct method mirrors what human analysts do. Align each new statement with its predecessor sentence by sentence, identify every insertion, deletion and substitution, and classify each edit by its direction. A word added to the description of inflation that intensifies it ("elevated" becoming "remains elevated") is a hawkish edit. A conditional phrase inserted before a commitment ("the committee anticipates" becoming "the committee will assess incoming data to determine") is a softening edit. A paragraph removed entirely may be the most significant change of all. Sentence alignment is straightforward for FOMC statements, whose structure is stable, and harder for press conferences and speeches, whose structure varies. For the statement, it can be done almost mechanically. The judgement lies in classifying the edits. A transformer that sees both the old and new sentence, with an instruction or training to judge which implies tighter policy, is well suited to the task. Asking a model to compare two passages tends to be more reliable than asking it to score each in isolation, because comparison removes the need for the model to hold an absolute scale in mind. Miguel Acosta and Ellen Meade, in a 2015 note published by the Federal Reserve Board, measured the semantic similarity of consecutive FOMC statements and documented how the statements had become longer, more complex and more similar to one another over time, particularly after 2008. A high degree of similarity between consecutive statements makes each change more conspicuous and raises the information content of edits. Embedding drift A second method represents each document, or each section of it, as a point in an embedding space produced by a transformer, and measures how far and in what direction the point moves between consecutive documents. The direction can be given meaning by defining an axis: for example, by taking the embedding of a set of clearly hawkish reference passages and a set of clearly dovish ones, and projecting each new document onto the line between them. Embedding drift is useful for longer and less structured texts, such as speeches and minutes, where sentence-level alignment is impractical. It can also capture changes in emphasis that do not involve any single decisive edit: a speech that devotes more of its length to upside inflation risks than the previous speech did, without changing any explicit statement of policy, will move in the embedding space. The weakness of embedding methods is interpretability. A movement in an embedding space is hard to explain to a portfolio manager or a risk committee. It is good practice to pair any embedding measure with an extraction of the specific passages that contributed most to the movement, so that a human can check the reading. Emphasis and topic share A third method measures how much of a document is devoted to each subject: inflation, employment, financial stability, global conditions, the balance sheet. Shifts in emphasis are often an early signal of a shift in policy. A committee preparing to tighten tends to say more about inflation; one preparing to ease tends to say more about downside risks to growth and employment. Topic shares can be measured with topic models, with zero-shot classification of each sentence by subject, or by instructing a large language model to allocate passages to categories. Commitment and conditionality A fourth method measures how firmly the central bank commits itself. Forward guidance ranges from open-ended statements of intent to commitments tied to dates or thresholds. Shifts along that range are significant: moving from a calendar date to an economic threshold, as the Fed did in December 2012, or from a firm commitment to "data dependence," as many central banks did in 2022 and 2023, changes how the market should interpret every subsequent data release. Extracting the degree of commitment is a natural task for a large language model instructed to identify conditional clauses, quantify the strength of modal verbs and locate any explicit dates or thresholds. Weighting the voices Statements speak for the committee. Speeches speak for individuals, and building a picture of the committee's direction from individual speeches requires decisions about whose words count, and how much. Three considerations dominate. The first is role. The chair's views carry the most weight, since the chair sets the agenda and the committee's consensus usually forms around the chair's position. The Board's vice chair and the president of the Federal Reserve Bank of New York, who holds a permanent vote and serves as the committee's vice chair, typically speak close to the leadership's view. Other governors vote at every meeting. Presidents of the other regional banks vote on a rotating basis, and a speech by a current voter matters more than one by a non-voter for the next decision, though non-voters participate in the deliberation and their views can shift the consensus over time. The second is the speaker's own baseline. Some officials are habitually hawkish and some habitually dovish. A hawkish speech from a known hawk conveys little; a hawkish speech from a known dove, or from an official near the centre of the committee, conveys a great deal. Building a stance history for each official from their past speeches, and measuring each new speech against that history, is the speaker-level equivalent of the baseline-relative measurement this booklet has recommended for executives. It also identifies the officials whose views move with the committee's centre of gravity, whose shifts tend to precede the committee's own. The third is timing. Officials do not speak publicly about monetary policy during the blackout period that begins on the second Saturday before each meeting and ends at midnight on the day after it. The speeches in the weeks before the blackout begins are therefore the last public signals before a decision, and are often used deliberately to prepare the market. Speeches immediately after a meeting may explain the decision or, in the case of dissenters, register disagreement. Combining these, a committee-level speech index weights each speech by the speaker's role and voting status, scores its stance relative to the speaker's own history, and tracks the weighted average over time. Such an index, updated after every speech, provides a continuous estimate of the committee's direction between the eight scheduled statements each year. Its value lies precisely in the intervals between statements, where the market's attention is more diffuse and individual speeches may not be fully absorbed. A case: retiring a word The 2021 episode surrounding the word "transitory" shows how a single term can carry a policy stance and how a shift in its use can be tracked. As inflation rose through 2021, the Federal Reserve repeatedly characterised the increase as largely reflecting transitory factors. The word appeared in statements, press conferences and speeches, and it carried a clear policy implication: if inflation was transitory, there was no need to tighten quickly. Over the autumn, the committee's descriptions became more qualified, and the word was increasingly hedged. Then, in testimony before the Senate Banking Committee on 30 November 2021, Chair Jerome Powell said that it was probably a good time to retire the word. Markets read this as a signal of faster tightening, and the subsequent statement in December accelerated the reduction of asset purchases. A system tracking only the statement would have seen the change at the December meeting. A system tracking speeches and testimony, measuring the frequency of the word, the degree of hedging around it and the stance of the sentences in which it appeared, would have seen a gradual shift over several months culminating in the testimony. The value of the broader reading lies in that lead. The case also illustrates a danger. Any large language model trained on text from after late 2021 knows how this episode ended. Asked to read a September 2021 speech and judge whether it signalled faster tightening, such a model may draw on what it knows of the outcome. The episode is so widely discussed that contamination is almost certain. Historical evaluation of central bank stance models with modern language models is therefore exposed to look-ahead bias even more than evaluation of corporate signals, because central bank episodes are among the most extensively analysed events in financial journalism and economics. Tying words to prices The test of a stance measure is whether it explains how markets respond. The cleanest way to establish that, following the tradition of Gürkaynak, Sack and Swanson described in the previous chapter, is to use high-frequency market data in narrow windows around each communication. The procedure works like this. For each statement or press conference, measure the change in interest rate futures and Treasury yields at several maturities in a window of perhaps thirty minutes around the release, capturing the market's own reading of the news while excluding most other influences. Decompose those changes into components: a surprise in the current decision, inferred from the nearest futures contract; and a surprise in the expected path, inferred from longer-dated contracts. Then ask whether the text measures, constructed without reference to prices, explain the path surprise. If a stance change measure explains a substantial share of the path surprise, it is capturing what the market cares about. If it does not, either the measure is missing the relevant content or the market's reaction is driven by something other than the text, such as the projections released at the same time. This approach also allows the separate contributions of the statement and the press conference to be assessed, since they are released half an hour apart and the market's reaction to each can be distinguished in the high-frequency data. High-frequency validation has a further use: it provides a source of labels. Instead of having annotators judge whether each sentence is hawkish, one can train a model to predict the market's reaction from the text. This approach, analogous to learning word weights from returns in the corporate setting, has the advantage of measuring precisely what moves prices. It has the disadvantage of a very small sample: the FOMC meets eight times a year, so even a twenty-five year history provides only around two hundred statements. Training a large model on two hundred examples invites overfitting. In practice, the most robust approach combines the two: a stance classifier trained on human labels from a large set of sentences, validated against the high-frequency reaction to a smaller set of events. Beyond the Federal Reserve The same methods apply to other central banks, with adjustments. The European Central Bank holds a press conference after each monetary policy meeting in which the president reads an introductory statement and then answers questions; the statement is long and relatively stable, which suits difference-based analysis. The Bank of England publishes a Monetary Policy Summary and minutes simultaneously with each decision, along with individual members' votes, so that dissent is visible. The Bank of Japan has historically been more opaque, and shifts in its communication have occasionally caused severe market moves precisely because they were rare. Language adds a complication. Central bank communication in languages other than English is often available in translation, and translation can alter nuance. A model reading the official English version of a Bank of Japan statement is reading a translation chosen by the bank, which may itself be strategic. Multilingual transformer models allow the original text to be analysed directly, and comparing the reading of the original with that of the translation can itself be informative. There is also the growing use of these methods by central banks themselves. Several central banks and the Bank for International Settlements have published research applying NLP to their own communications and to those of their peers, to measure how clearly they are understood and how their messages are received. When the speaker is also measuring how its words are read, the strategic loop described in earlier chapters closes completely, and the stability of any text-based measure of policy stance should be expected to erode as institutions calibrate their language to the measures. What the methods can and cannot do Transformer models have made it practical to detect shifts in central bank communication that previously required an experienced human reader: a qualifier added, an emphasis moved, a commitment softened. They can do it across many central banks and many communication channels at once, faster than a human team, and with a consistency no human can maintain over thousands of documents. That is a substantial gain for anyone who must monitor global monetary policy. What they cannot do is resolve the inherent ambiguity of conditional guidance, predict how the economic data that the guidance depends on will turn out, or know which shifts the market has already anticipated from other sources. The text tells you what the central bank is signalling. The market reaction tells you how much of that signal was news. The economy decides whether the signal was right. A complete system needs all three, and the language model supplies only the first. CHAPTER 8 Automating ESG Reporting and Reading It Critically Sustainability reporting is the newest large body of financial text and the one whose form is changing fastest. In little more than a decade it has moved from voluntary glossy brochures, written mainly for public relations, to regulated disclosures subject in some jurisdictions to external assurance and digital tagging. It has also become a site of intense political contest, with regulatory regimes being built, scaled back and in some cases abandoned within a few years. For anyone applying language technology to finance, it presents two distinct problems that share the same documents. Companies must produce these reports, which increasingly requires gathering, structuring and drafting large amounts of information. Investors must read them, which requires separating substance from boilerplate and commitments from aspirations. Language models are now central to both tasks. This chapter takes them in turn, after setting out the regulatory landscape that shapes what the documents contain. The regulatory landscape in 2026 For most of the 2010s, sustainability reporting was governed by voluntary frameworks: the Global Reporting Initiative's standards, the industry-specific standards of the Sustainability Accounting Standards Board, the questionnaires of CDP, and from 2017 the recommendations of the Task Force on Climate-related Financial Disclosures, known as the TCFD, which organised climate disclosure around governance, strategy, risk management and metrics. The result was a proliferation of formats and a large volume of text that was difficult to compare across firms. The past several years have seen a move toward mandatory standards, unevenly across jurisdictions. The main regimes, and their status at the time of writing, are compared in Table 5. Table 5. Main sustainability disclosure regimes, status as of late 2026. Regime Origin Scope Status ISSB IFRS Foundation, 2023 Set by each adopting jurisdiction Adopted or in progress in many jurisdictions CSRD European Union, 2022 Narrowed by Omnibus I to the largest firms Revised ESRS apply from 2027 periods SEC United States, 2024 Registrants Stayed; rescission proposed 2026 California State laws, 2023 Large firms doing business in state Emissions reporting proceeding; risk reports litigated The International Sustainability Standards Board, established by the IFRS Foundation, issued its first two standards in June 2023: IFRS S1, on general requirements for disclosure of sustainability-related financial information, and IFRS S2, on climate-related disclosures. They build on the TCFD structure, which the ISSB formally took over in 2023, and focus on information material to investors, sometimes called single materiality. The standards apply only where a jurisdiction adopts them, and a substantial and growing number of jurisdictions have done so or announced plans to, including the United Kingdom, Australia, Japan, Singapore, Hong Kong, Canada and Brazil, often with modifications and phased timelines. The European Union's Corporate Sustainability Reporting Directive, adopted in 2022, took a more expansive approach. It required reporting according to European Sustainability Reporting Standards, the ESRS, whose first set was adopted in 2023. The ESRS were built on double materiality, requiring firms to report both how sustainability matters affect the company and how the company affects people and the environment. The original standards contained around a thousand potential data points, of which each firm reported those deemed material, and initially applied to tens of thousands of companies in phases. The reporting burden prompted a political reaction. In February 2025 the European Commission proposed a simplification package known as Omnibus I. After an initial "stop-the-clock" measure postponing later waves of reporting, the final package, agreed at the end of 2025 and enacted in early 2026, restricted mandatory reporting to companies with more than one thousand employees and more than four hundred and fifty million euros of net turnover. A revised and substantially shortened set of ESRS, with far fewer mandatory data points, applies to reporting periods beginning in 2027, with companies already reporting permitted to adopt it earlier. In the United States, the Securities and Exchange Commission adopted climate disclosure rules in March 2024. They were challenged in court and stayed. In March 2025, the Commission voted to stop defending them. The Eighth Circuit Court of Appeals declined to rule on the merits in the absence of a defence, holding the case in abeyance pending agency action, and in 2026 the Commission moved toward a formal rescission. Federal mandatory climate disclosure in the United States is therefore effectively dead for the present. California, however, enacted two laws in 2023 requiring large companies doing business in the state to report greenhouse gas emissions and climate-related financial risks. The emissions reporting regime has proceeded, with first reports of direct and energy-related emissions due in 2026. The climate risk reporting law was paused by a Ninth Circuit injunction in November 2025 pending appeal. The practical consequence is fragmentation. A multinational company may report under the revised ESRS in Europe, under an ISSB-based standard in several Asian and Commonwealth jurisdictions, under California law for its emissions, and under voluntary frameworks for investors and raters elsewhere. The texts that result overlap but are not identical, and they change as the standards change. Automating production On the production side, the problem is one of information logistics. A sustainability report requires data from across a company and often beyond it: energy consumption from thousands of utility bills, fuel use from fleet records, emissions factors from published databases, workforce data from human resources systems, supplier information from procurement, policies from legal and compliance, and narrative descriptions of strategy, governance and risk from management. Much of this information exists only in unstructured documents. Language models contribute at several points in this chain. Extraction from documents. Invoices, utility bills, supplier questionnaires, certificates and contracts arrive in many formats. Models that combine optical character recognition with language understanding can extract the relevant quantities, units, dates and entities and map them to structured records. This was possible with earlier machine learning tools, but required separate training for each document type. Modern multimodal language models handle a wide variety of formats with instruction alone, which has made extraction practical for long-tail documents that were previously processed by hand. Supply chain and value chain data. Emissions in a company's value chain, known as Scope 3 emissions, are typically the largest part of its footprint and the hardest to measure. Firms rely on supplier-provided data, where available, and on estimates based on spending and industry averages, where not. Language models help by reading supplier sustainability reports and disclosures to extract their emissions figures and targets, by classifying purchases into the categories used for spending-based estimates, and by flagging inconsistencies. Mapping to standards. Each standard defines its disclosure requirements in its own terms, and a company reporting under several must map its information to each. A model can take a disclosure requirement, such as a particular ESRS data point or ISSB paragraph, and identify which internal data and which existing narrative text address it, highlighting gaps. This is a retrieval and classification task well suited to modern language models, provided their outputs are checked. Drafting narrative. Large portions of a sustainability report are narrative: descriptions of governance arrangements, strategy, risk processes, policies and actions. Language models can produce first drafts from internal documents and structured data. This is the most visible and most hazardous application. A model that drafts fluent prose about a company's climate strategy can easily produce statements that are more confident, specific or favourable than the underlying facts support, or can invent details that sound plausible. In a regulated disclosure subject to liability and, in Europe, to limited assurance by an auditor, any such statement is a legal risk. Digital tagging. Both the ESRS and the ISSB standards are accompanied by digital taxonomies for tagging disclosures in machine-readable form using XBRL, the format already used for financial statements. Tagging narrative disclosures requires judgement about which taxonomy element each passage corresponds to, a classification task that language models can assist with. The governing principle for production use is that language models should accelerate the work of preparers, not replace their judgement. Every extracted figure needs a traceable link to its source document. Every drafted sentence needs to be reviewed by someone accountable for its truth. The models are best deployed as tools that find, organise and propose, with human sign-off at each point where a figure or claim enters the published report. The ninth chapter returns to the governance questions this raises. Reading critically On the consumption side, investors face a different problem. Sustainability reports are long, numerous and, by the nature of their history, often promotional. The relevant questions are not about sentiment, since nearly all such reports are positive in tone, but about substance. Does the report contain specific, quantified commitments, or only aspirations? Are the commitments consistent with what the company reported last year, and with its capital spending? Does the report address the risks that are material to its industry, or does it emphasise peripheral matters? Is the company exposed to climate risk in ways its report does not acknowledge? Cheap talk and cherry-picking The most influential early application of transformer models to these questions was ClimateBERT, developed by Nicolas Webersinke, Mathias Kraus, Julia Anna Bingler and Markus Leippold and released in 2021. It was a language model further pretrained on a large body of climate-related text, then fine-tuned for tasks such as identifying climate-related passages, classifying their sentiment and judging their specificity. The same team applied it, in a 2022 paper in Finance Research Letters titled "Cheap Talk and Cherry-Picking: What ClimateBERT Has to Say on Corporate Climate Risk Disclosures," to the annual reports of firms that had publicly declared their support for the TCFD recommendations. They found that support for the TCFD was largely cheap talk: declaring support did not substantially increase the quantity or quality of firms' climate disclosure, and firms tended to cherry-pick, reporting mainly on the TCFD categories where disclosure was least demanding and least material. The study illustrated both the value of the tool and the nature of the material. A model trained to distinguish specific from vague statements can measure what human readers suspected but could not quantify at scale: that much sustainability disclosure is designed to signal engagement without committing to anything measurable. Specificity, consistency and alignment Building on this work, the most useful critical reading of sustainability text measures three properties. Specificity is the degree to which a statement commits the company to something verifiable: a quantity, a date, a baseline, a scope. "We are committed to reducing our environmental footprint" is unspecific. "We will reduce absolute Scope 1 and 2 emissions by forty-two percent by 2030 from a 2020 baseline" is specific. Transformer models can classify statements along this dimension reliably, and large language models can extract the components of specific commitments into structured fields. Consistency is whether commitments and figures are stable over time and across documents. A company that quietly changes its baseline year, narrows the scope of a target, moves a target date, or restates a previous year's emissions without explanation is doing something an investor should know about. Detecting this requires comparing each year's report with previous years', exactly the difference-based analysis that "Lazy Prices" applied to annual reports and that the previous chapter applied to central bank statements. The same principle holds: the information is in the change. Alignment is whether what a company says matches what it does. A company with an ambitious transition plan whose capital spending remains concentrated in high-emitting assets, or whose lobbying, as reported by third parties, opposes the policies its report endorses, displays a gap between word and deed. Language models can extract the claims; testing them requires linking the claims to financial data, emissions data and other evidence, which is where the analysis moves from language processing to investment research. A worked reading: one target, three reports A stylised example shows how these properties are measured in practice. Consider a European consumer goods company and the climate target it reports across three consecutive annual sustainability reports. The first step is extraction into a fixed schema. For each target, a language model is instructed to extract: the type of target (absolute emissions, emissions intensity, renewable energy share, or other); the scopes covered (Scope 1, 2 and which categories of Scope 3); the baseline year and baseline value; the target year; the reduction percentage; whether the target relies on carbon offsets; whether it has been validated by an external body such as the Science Based Targets initiative; and the exact passage from which each field was taken. The fields are checked automatically against the quoted passages, and any field that cannot be grounded in a passage is left empty rather than inferred. In the first report, the extraction yields an absolute target for Scope 1 and 2 emissions, a reduction of a stated percentage by 2030 against a 2019 baseline, externally validated, and a separate intensity target for Scope 3. In the second report, the Scope 1 and 2 target is unchanged, but the Scope 3 target now applies only to purchased goods and services, a narrowing of scope from the previous year's reference to the full value chain. The narrative describes the change as a "refinement of methodology." In the third report, the baseline year for the Scope 1 and 2 target has moved from 2019 to 2021, a year in which the company's emissions were higher because of an acquisition, so that the same percentage reduction now implies a higher 2030 emissions level. The report explains the change in a footnote as a "rebaselining to reflect structural changes." Each change may be legitimate. Rebaselining after an acquisition is accepted practice under the main greenhouse gas accounting standards, and narrowing a Scope 3 target to categories the company can actually measure may improve its credibility. But each change also weakens the commitment in absolute terms, and neither is prominent in the report. A human analyst covering hundreds of companies would be unlikely to catch both. An extraction system that compares structured fields year on year flags them immediately, and the analyst's time is spent judging whether they are justified rather than finding them. The same structured history supports alignment checks. If the company's capital expenditure, as reported in its financial statements, shows continued investment in high-emitting capacity while its targets are being quietly loosened, the combination is a stronger signal than either alone. Exposure from the outside A complementary approach measures a company's sustainability exposure not from its own sustainability report but from other texts in which it is discussed less defensively. Zacharias Sautner, Laurence van Lent, Grigory Vilkov and Ruishen Zhang, in a 2023 paper in the Journal of Finance titled "Firm-Level Climate Change Exposure," measured how much attention management and analysts devoted to climate-related topics in earnings calls, separately for opportunities, regulatory shocks and physical threats. Because earnings calls are focused on financial performance and involve sceptical analysts, climate discussion there is more likely to reflect material financial concern than the sustainability report's managed narrative. Their measures captured variation across firms and over time that was not visible in standard ESG ratings and were associated with real outcomes such as green-technology hiring and patenting. Robert Engle, Stefano Giglio, Bryan Kelly, Heebum Lee and Johannes Stroebel, in a 2020 paper in the Review of Financial Studies titled "Hedging Climate Change News," took a market-level approach, building indices of climate news from newspaper text and constructing portfolios designed to hedge the risk represented by those news shocks. Their work illustrated how text could be used to define and price a risk factor that had no obvious numerical measure. The ratings problem Much investor use of ESG information runs through ratings produced by specialist providers, and those ratings are themselves largely produced by reading company documents. Florian Berg, Julian Kölbel and Roberto Rigobon, in a 2022 paper in the Review of Finance titled "Aggregate Confusion: The Divergence of ESG Ratings," compared ratings from six major providers and found that the correlations between them ranged from about 0.38 to 0.71, far lower than the near-perfect agreement among credit rating agencies. They traced the divergence mainly to differences in what the raters measured and how they measured it, rather than to differences in how they weighted the categories. For the practitioner building a text system, this is both a warning and an opportunity. It is a warning because a model trained to replicate an ESG rating will learn one provider's idiosyncratic method, not an objective truth. It is an opportunity because a firm that builds its own transparent extraction of specific, consistent, verifiable information from primary documents is not dependent on any provider's black box and can construct measures tailored to the questions it actually cares about. The convergence ahead The regulatory fragmentation of recent years is likely to persist, but the direction of travel in most major jurisdictions outside the United States is toward digitally tagged, standardised disclosure that is designed to be read by machines. When reports are tagged at the level of individual data points, much of the extraction work that language models currently perform on unstructured text will become a matter of reading structured data. The role of language models will then shift toward the parts of disclosure that remain narrative and judgemental: the description of strategy, the assessment of risk, the explanation of changes. Those are precisely the parts where substance and boilerplate are hardest to separate and where careful critical reading, measured against the company's own history, adds most value. CHAPTER 9 Governing the Models: NLP Inside a Regulated Firm A research prototype that reads earnings calls and produces a signal is a piece of software. The same system running inside a bank, an asset manager or an insurer, informing investment decisions with client money or producing figures that enter a regulated disclosure, is something else: a model, in the specific sense that financial regulators use the word, subject to expectations about validation, documentation, monitoring, accountability and the treatment of the people affected by its outputs. Language models fit awkwardly into the frameworks built for older kinds of model, and the awkwardness is not merely bureaucratic. It points to real weaknesses that a firm deploying these systems must manage. This chapter sets out the main governance problems that financial NLP raises, and how well-run firms address them. What regulators mean by model risk The reference point for model governance in banking is the supervisory guidance on model risk management issued by the Federal Reserve and the Office of the Comptroller of the Currency in April 2011, commonly known as SR 11-7. It defines a model broadly, as a quantitative method that processes input data into estimates, and identifies two sources of model risk: a model may have fundamental errors and produce inaccurate outputs, or it may be used incorrectly or outside the circumstances for which it was designed. It requires firms to maintain an inventory of models, to validate them independently before use and periodically thereafter, to document their design, assumptions and limitations, and to monitor their performance on an ongoing basis. In the United Kingdom, the Prudential Regulation Authority's supervisory statement SS1/23 on model risk management principles for banks, which took effect in May 2024, applies similar principles and explicitly anticipates that firms will use models with greater complexity and less transparency, including machine learning models. Asset managers that are not banks are not directly subject to these banking frameworks, but most large managers apply comparable principles, and their regulators expect them to understand and control the tools that drive their investment decisions. For any firm, the question a supervisor will ask is the same: do you know what this system does, how do you know it works, and what happens when it does not? Why language models strain the framework Traditional quantitative models have properties that make validation tractable. Their logic can be written down. Their inputs are defined. Given the same inputs, they produce the same outputs. Their behaviour can be tested across the range of conditions in which they will be used. Language models, and especially large generative models, depart from each of these properties. Opacity. A fine-tuned transformer has hundreds of millions of parameters; a frontier generative model has many more. Neither has logic that can be written down in a form a validator can inspect. Techniques exist to identify which parts of an input most influenced an output, and they are useful for diagnosis, but they do not provide the kind of structural understanding a validator has of a regression model or a pricing formula. Non-determinism. Generative models produce outputs by sampling, and even when the sampling is set to be as deterministic as possible, the same prompt can produce different outputs across runs or across minor infrastructure changes. A model that gives different scores to the same transcript on different days is, from a validation standpoint, not one model but a distribution of them. Silent change. A model accessed through a commercial interface may be updated by its provider. Providers increasingly offer pinned versions that remain fixed for a period, and a well-governed firm uses them, but pinned versions are retired after some time, forcing migration. Each migration is, in effect, the introduction of a new model, requiring revalidation. The prompt as part of the model. In a generative system, the instruction given to the model is as much a part of the model's specification as its weights. A change of wording in the prompt can change outputs materially. Prompts must therefore be version-controlled, documented and validated in the same way as code, and changes to them must go through the same approval process as changes to any model component. Firms that treat prompts as informal configuration that analysts can adjust at will have, in governance terms, an uncontrolled model. Open-ended outputs. A classification model produces one of a fixed set of outputs. A generative model can produce anything, including outputs that are fluent, confident and wrong. This failure, usually called hallucination, is the most discussed weakness of generative models, and it matters acutely in finance, where a fabricated figure or a misattributed statement can lead to a bad investment or a false disclosure. Containing hallucination The most effective containment strategies share a principle: constrain the model to work from evidence that can be checked, and check it. Grounding in source documents. Rather than asking a model what it knows about a company, a well-designed system gives it the specific document to analyse and instructs it to answer only from that document. Retrieval-augmented generation, in which relevant passages are retrieved from a controlled document store and provided to the model as context, extends this to larger collections. Grounding sharply reduces hallucination but does not eliminate it; models can still misread or embellish what they are given. Required citation. The model can be required to cite, for every extracted figure or judgement, the exact passage that supports it. The citation can then be checked automatically: does the quoted passage exist in the source document, and does it contain the figure claimed? An output whose citation fails verification is rejected or flagged for review. This one practice converts a hard problem, deciding whether a model's statement is true, into a much easier one, deciding whether a quoted string appears in a document. Constrained output formats. Requiring output in a defined structure, such as a fixed set of fields with defined types and allowable values, limits the space in which the model can go wrong and makes outputs machine-checkable. Cross-checks. Extracted figures can be checked against other sources: an emissions figure against the previous year's report and the company's own reported trend, a revenue guidance range against the press release, a quoted statement against the transcript. Discrepancies are routed to human review. Separation of extraction from judgement. As the fifth chapter recommended for backtesting reasons, it is often wise to use a large model only to extract structured facts and to perform predictive or evaluative judgement with a separate, simpler and more transparent model trained on those facts. This also helps governance: the extraction step can be validated by checking its accuracy against human-labelled samples, and the judgement step can be validated with conventional quantitative methods. Validation in practice Validating an NLP system in a financial firm draws on both machine learning practice and quantitative finance practice. On the language side, validation means measuring accuracy on representative labelled samples, separately for each task and each document type, and investigating systematic errors. Does the stance classifier perform equally well on speeches and minutes? Does the evasion detector perform worse for executives whose first language is not English, or for transcripts from a particular vendor? Does extraction accuracy fall for scanned documents? Error analysis of this kind often reveals that a model with high average accuracy has pockets of severe failure that matter disproportionately. On the financial side, validation means testing that the signal or output behaves as intended in the use for which it was designed: that a trading signal's out-of-sample performance is consistent with its backtest, that it is not unduly exposed to known risk factors, that its turnover and capacity are understood, and that the backtest errors catalogued in the fifth chapter have been addressed. For disclosure uses, it means testing that extracted and drafted content is accurate against source evidence to the standard required by auditors. Validation also means stress testing: deliberately feeding the system unusual inputs to see how it fails. A transcript with garbled speaker labels. A central bank statement in an unexpected format. A sustainability report that uses unusual terminology. An adversarial passage designed to mislead. A system that fails gracefully, flagging inputs it cannot handle, is far safer than one that produces confident nonsense. Ongoing monitoring completes the cycle. The distribution of inputs changes over time, as companies adapt their language, as new topics emerge, as regulations change the structure of disclosures. A model's accuracy should be measured periodically on fresh labelled samples, and the distribution of its outputs tracked for unexplained shifts. A sudden change in the average score assigned to earnings calls, without a corresponding change in economic conditions, is a warning that something in the inputs or the model has changed. Information, data rights and market conduct Several legal and conduct issues arise specifically from the use of text in finance. Data rights. Transcripts, news archives and analyst reports are licensed products, and the licences often restrict how they may be used, including whether they may be used to train models, whether derived data may be redistributed, and whether they may be sent to third-party model providers. A firm that sends licensed transcripts to an external language model service may be in breach of its licence terms. Data governance must therefore track not only the content of each dataset but the terms under which it was obtained. Confidentiality. Many potentially valuable texts are confidential: internal research, client communications, unpublished deal documents. Sending them to an external model service raises questions of data protection, client confidentiality and, in some cases, material non-public information. Firms typically address this through contractual commitments from providers about data retention and use, through deployment of models within their own infrastructure, or through policies prohibiting certain categories of data from leaving the firm. Market abuse. Text systems that ingest information quickly can, in principle, ingest information that should not be traded on: a leaked document, a mistakenly published filing, a social media post containing confidential information. The legal question of whether trading on such information is permissible depends on how it was obtained and its status, and a fully automated system cannot make that judgement. Firms need controls over the sources their systems ingest. Build, buy or borrow A firm deploying financial NLP faces a basic sourcing choice, and each option carries a distinct risk profile. It can buy finished outputs: sentiment scores, call summaries, ESG extractions or central bank stance indices produced by a data vendor. This is the fastest route and requires the least internal expertise. Its governance weakness is opacity at one remove: the firm relies on a model it cannot inspect, built on data it may not see, with a methodology that may change without notice. A firm that buys signals must validate them as it would any external model, insist on documentation of methodology and changes, and recognise that any signal sold widely is shared with every other buyer. It can borrow capability by calling a commercial frontier model through an interface and building its own pipeline around it. This gives the firm control over the questions asked, the data provided and the post-processing, while relying on the provider for the model itself. Its weaknesses are the ones described above: non-determinism, version changes, data confidentiality, licence restrictions on sending data to third parties, and dependence on a small number of providers. It also carries the contamination risk for backtesting described in the fifth chapter, since the firm does not control what the model was trained on. It can build, by fine-tuning or training models it hosts itself, usually starting from open-weight models whose parameters are publicly released. This gives the greatest control: the model is fixed, versioned and auditable; data need never leave the firm; and for historical work, a model can be chosen or trained with a known cutoff. Its costs are the expertise and infrastructure required, and the fact that the firm's own model may lag behind the frontier in capability. In practice, mature firms combine the three. A common pattern is to borrow frontier capability for exploration and for generating labelled training data, to build smaller fixed models for production scoring where stability and auditability matter, and to buy vendor data where it provides coverage the firm cannot efficiently produce itself. Whatever the mix, the firm should maintain an exit plan for each dependency: what happens if a vendor changes its methodology, a provider retires a model version, or a licence is not renewed. A model that cannot be replaced is a concentration risk for the firm, just as widespread reliance on a few providers is a concentration risk for the system. The regulatory horizon Regulatory attention to artificial intelligence in finance has grown, though its direction has varied by jurisdiction. In the European Union, the AI Act, which entered into force in August 2024, takes a risk-based approach. It classifies certain uses as high-risk, including the evaluation of individuals' creditworthiness and risk assessment and pricing in life and health insurance, subjecting them to requirements for risk management, data governance, transparency, human oversight and accuracy. Investment uses of NLP, such as trading signals, are not on the high-risk list, but obligations for providers of general-purpose AI models began to apply from August 2025, and the Act's broader expectations about transparency and oversight shape industry practice. The timing of some high-risk obligations has itself been the subject of further simplification proposals. In the United States, the Securities and Exchange Commission proposed in 2023 a rule addressing conflicts of interest arising from broker-dealers' and advisers' use of predictive data analytics with investors. The proposal drew heavy criticism and was withdrawn in June 2025. Existing obligations, including fiduciary duty, best execution, recordkeeping and the prohibition on misleading statements, continue to apply to AI-assisted activities, and the Commission has brought enforcement actions against firms that made false claims about their use of AI, a practice that came to be called AI washing. At the international level, the Financial Stability Board, in a November 2024 report on the financial stability implications of artificial intelligence, and the International Monetary Fund, in a chapter of its October 2024 Global Financial Stability Report on AI in capital markets, identified concerns that bear directly on financial NLP. Chief among them were third-party dependencies and concentration among a small number of model and cloud providers, market correlations arising from many firms using similar models on similar data, and the potential for faster and more synchronised market reactions to news. If many participants read the same central bank statement with similar models and trade on similar outputs, the result could be sharper and more correlated price moves. This is the system-level counterpart of the decay described in the fourth chapter: signals that everyone shares not only stop being profitable but can amplify volatility. Accountability Underlying all of this is a principle that financial regulation has always applied and that does not change for language models: a person is accountable for every decision. A model can propose a trade, draft a disclosure, flag a risk or score a document. It cannot be accountable for any of them. The firm must be able to identify who approved the model's use, who reviewed its output, and who made the decision it informed. In practice, this means designing NLP systems with clear points of human decision. A trading signal derived from text is approved for use by an investment committee after validation, monitored by a named owner, and subject to limits. A sustainability report drafted with model assistance is reviewed and signed off by the officers responsible for its accuracy. A central bank stance measure used in an asset allocation process is understood by the people who rely on it, including its limitations. The firms that use these tools well tend to share a characteristic. They treat language models as powerful but fallible instruments that extend the reach of skilled people, not as oracles that replace them. They invest at least as much in data quality, timing, validation and review as in model capability. And they are honest, internally, about what their systems cannot do. That disposition is not a regulatory burden layered on top of good practice. It is good practice, and it is what separates a durable capability from an expensive experiment. Conclusion: The Edge Is in the Measurement The capability to read financial text by machine is no longer scarce. Any analyst with a laptop can download an open-source financial sentiment model, and any firm with a commercial subscription can ask a frontier language model to summarise an earnings call, classify a central bank statement or extract the targets from a sustainability report. The capability that was once a source of advantage is becoming a utility. What follows from that, and from the arguments of the preceding chapters, is a set of conclusions about where value now lies, what remains unsettled, and what practitioners should do differently. Where the value has moved When everyone has the same model, the advantage moves to the parts of the problem the model does not solve. The chapters of this booklet have identified those parts repeatedly, in three quite different settings. The first is timing. A signal is worth what it adds to what the market already knows at the moment a trade can be made. Knowing exactly when each document became available, in what version, and what the market had already absorbed from other sources, is engineering work of an unglamorous kind. It does not improve with a better model. It is also where the largest errors in published and vendor backtests have tended to lie. The second is the baseline. The same words mean different things from different speakers, in different sections of a document, in different economic conditions. An executive's cautious answer, a committee's new qualifier and a company's revised emissions target are all informative relative to what that speaker said before. Building the history to measure those deviations, for every speaker, every document type and every topic, is an investment that accumulates over time and is hard for a newcomer to replicate. The third is the question asked. Early text analysis asked how positive a document was, because that was what dictionaries could measure. Modern models can be asked much better questions: whether an answer addressed the question, whether a commitment has weakened, whether a risk has been newly emphasised, whether a figure is consistent with last year's. These questions are harder to game than word choice, and the evidence suggests they carry more information. Choosing them well requires understanding the economics and incentives of the documents, which is domain knowledge, not machine learning. The fourth is honesty about evidence. The contamination of large language models by knowledge of the future, the breadth of choices available in building any text signal, and the concentration of text signals in illiquid securities together mean that the gap between backtested and realisable performance is unusually wide in this field. A team that closes that gap with disciplined process will make better decisions than one with better models but looser habits, and will avoid committing capital to signals that were never real. Three settings, one principle It is worth noticing how consistently the same principle applied across the three arenas this booklet examined, despite their differences. For earnings calls, the evidence pointed to changes in tone rather than levels, to the question-and-answer session rather than the prepared remarks, and to evasion and inconsistency rather than vocabulary. The information was in what management could not fully control. For central banks, the information lay in the edits between consecutive statements, in shifts of emphasis across speeches, and in the parts of guidance the market had not anticipated, measured by the high-frequency reaction of prices. The information was in what had changed. For sustainability reports, the information lay in specificity rather than tone, in consistency over time, and in the alignment between claims and evidence. The information was in what could be verified. What unites these is that financial speakers are strategic and know they are being read, increasingly by machines. Any measure that captures only the surface of their language will be gamed, and the evidence that firms have already shifted their vocabulary in response to widely used dictionaries shows that this happens quickly. The durable measures are those that capture what is costly or impossible to disguise: change relative to one's own past, the gap between question and answer, the presence or absence of verifiable specifics. This is not a temporary feature of the current state of technology. It follows from the nature of the documents, and it will remain true as models improve. What remains unsettled Several questions are open, and honest practitioners should hold their views on them loosely. It is not settled how much of the apparent skill of large language models in financial prediction will survive clean out-of-sample testing over long periods. The early results are encouraging, and the post-cutoff evidence that exists is consistent with genuine skill in reading text. But the clean evaluation windows are short, each new model generation resets them, and the decay of profitability with adoption is already visible. Chronologically consistent models, trained only on data available up to each point in history, will make this question more tractable, and their results over the next few years deserve close attention. It is not settled whether widespread machine reading will make markets more efficient or more fragile. In one view, faster and more complete processing of public information is exactly what efficient markets require, and the profits of early adopters are the price of that improvement. In another, many participants reading the same documents with similar models and acting simultaneously will produce correlated trading and sharper price moves, with the system-level risks that financial stability authorities have begun to flag. Both effects are probably real, and their balance may differ between ordinary times and stressed ones. It is not settled what sustainability disclosure will look like in five years. The regulatory reversals of 2025 and 2026 show that its scope and form depend heavily on politics, and the divergence between European, Asian and American approaches may widen or narrow. Systems built to produce or read these disclosures need to be designed for change rather than for any particular standard. And it is not settled how central banks themselves will adapt. As they become more aware that their words are measured by machines, and as they use the same tools to measure their own communication, their language may become more standardised and predictable, reducing the information in each change, or they may deliberately vary it to retain flexibility. Either response would change what machine reading of their communication can achieve. What to do differently For a portfolio manager or researcher building text signals, the practical implication is to spend less effort choosing models and more on data and design. Insist on point-in-time data with verified timestamps. Build baselines for every speaker and document type. Measure surprises, not levels. Ask questions that are hard to game. Pre-commit to specifications and hold out data. Test large language models only on periods after their training cutoffs, or anonymise inputs, or use chronologically consistent models for historical work. Assume that any signal that works will decay, and plan for its replacement. For a policy economist or macro strategist reading central banks, the implication is to treat the text as one of three necessary inputs, alongside the market's high-frequency reaction and the economic data on which guidance is conditioned. Track the full communication sequence, including speeches and minutes, not only the statement. Be especially sceptical of historical evaluations using modern models, because central bank episodes are among the most discussed events in the training data of every such model. For a sustainability or reporting lead, the implication is to use language models to accelerate the gathering and organising of information while keeping a traceable chain from every published figure and claim back to its source and to an accountable person. Design for changing standards. Read peers' and suppliers' disclosures critically for specificity and consistency, not tone. For a risk officer or model validator, the implication is to treat the prompt, the model version and the data licence as components of the model, to require citation and verification of model outputs wherever they feed decisions or disclosures, and to measure performance continuously on fresh samples, because the language being read will shift in response to being read. The underlying lesson is old, and it is the one that every generation of quantitative finance has had to learn again with its new tools. Information is valuable only relative to what is already known, at the moment it can be acted upon, and only if one is honest about the evidence that it is real. Language models have made it vastly easier to read. They have not made it any easier to know what one has read is news. That remains the work, and it remains where the advantage lies. Notes Chapter 1 1. Paul C. Tetlock, "Giving Content to Investor Sentiment: The Role of Media in the Stock Market," Journal of Finance 62, no. 3 (2007): 1139–1168. 2. Werner Antweiler and Murray Z. Frank, "Is All That Talk Just Noise? The Information Content of Internet Stock Message Boards," Journal of Finance 59, no. 3 (2004): 1259–1294. 3. Victor L. Bernard and Jacob K. Thomas, "Post-Earnings-Announcement Drift: Delayed Price Response or Risk Premium?" Journal of Accounting Research 27 (1989): 1–36. 4. Feng Li, "Annual Report Readability, Current Earnings, and Earnings Persistence," Journal of Accounting and Economics 45, nos. 2–3 (2008): 221–247. 5. Scott R. Baker, Nicholas Bloom and Steven J. Davis, "Measuring Economic Policy Uncertainty," Quarterly Journal of Economics 131, no. 4 (2016): 1593–1636. 6. Asaf Manela and Alan Moreira, "News Implied Volatility and Disaster Concerns," Journal of Financial Economics 123, no. 1 (2017): 137–162. 7. Leland Bybee, Bryan Kelly, Asaf Manela and Dacheng Xiu, "Business News and Business Cycles," Journal of Finance 79, no. 5 (2024). 8. Lauren Cohen, Christopher Malloy and Quoc Nguyen, "Lazy Prices," Journal of Finance 75, no. 3 (2020): 1371–1415. Chapter 2 1. Tim Loughran and Bill McDonald, "When Is a Liability Not a Liability? Textual Analysis, Dictionaries, and 10-Ks," Journal of Finance 66, no. 1 (2011): 35–65. 2. Pekka Malo, Ankur Sinha, Pekka Korhonen, Jyrki Wallenius and Pyry Takala, "Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts," Journal of the Association for Information Science and Technology 65, no. 4 (2014): 782–796. 3. Narasimhan Jegadeesh and Di Wu, "Word Power: A New Approach for Content Analysis," Journal of Financial Economics 110, no. 3 (2013): 712–729. 4. Zheng Tracy Ke, Bryan T. Kelly and Dacheng Xiu, "Predicting Returns with Text Data," NBER Working Paper 26186 (2019). 5. Tomas Mikolov, Kai Chen, Greg Corrado and Jeffrey Dean, "Efficient Estimation of Word Representations in Vector Space," arXiv:1301.3781 (2013). 6. Ashish Vaswani et al., "Attention Is All You Need," Advances in Neural Information Processing Systems 30 (2017). 7. Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," Proceedings of NAACL-HLT (2019). 8. Dogu Araci, "FinBERT: Financial Sentiment Analysis with Pre-trained Language Models," arXiv:1908.10063 (2019). 9. Allen H. Huang, Hui Wang and Yi Yang, "FinBERT: A Large Language Model for Extracting Information from Financial Text," Contemporary Accounting Research 40, no. 2 (2023): 806–841. 10. Shijie Wu et al., "BloombergGPT: A Large Language Model for Finance," arXiv:2303.17564 (2023). 11. Alejandro Lopez-Lira and Yuehua Tang, "Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models," arXiv:2304.07619 (2023, revised 2025). 12. Alex Kim, Maximilian Muhn and Valeri V. Nikolaev, "Financial Statement Analysis with Large Language Models," Chicago Booth Research Paper, arXiv:2407.17866 (2024). 13. Yifei Chen, Bryan T. Kelly and Dacheng Xiu, "Expected Returns and Large Language Models," working paper (2022, revised 2024). Chapter 3 1. Dawn Matsumoto, Maarten Pronk and Erik Roelofsen, "What Makes Conference Calls Useful? The Information Content of Managers' Presentations and Analysts' Discussion Sessions," The Accounting Review 86, no. 4 (2011): 1383–1414. 2. S. McKay Price, James S. Doran, David R. Peterson and Barbara A. Bliss, "Earnings Conference Calls and Stock Returns: The Incremental Informativeness of Textual Tone," Journal of Banking and Finance 36, no. 4 (2012): 992–1011. 3. Stephan Hollander, Maarten Pronk and Erik Roelofsen, "Does Silence Speak? An Empirical Analysis of Disclosure Choices During Conference Calls," Journal of Accounting Research 48, no. 3 (2010): 531–563. 4. David F. Larcker and Anastasia A. Zakolyukina, "Detecting Deceptive Discussions in Conference Calls," Journal of Accounting Research 50, no. 2 (2012): 495–540. 5. William J. Mayew and Mohan Venkatachalam, "The Power of Voice: Managerial Affective States and Future Firm Performance," Journal of Finance 67, no. 1 (2012): 1–43. 6. Tarek A. Hassan, Stephan Hollander, Laurence van Lent and Ahmed Tahoun, "Firm-Level Political Risk: Measurement and Effects," Quarterly Journal of Economics 134, no. 4 (2019): 2135–2202. 7. Sean Cao, Wei Jiang, Baozhong Yang and Alan L. Zhang, "How to Talk When a Machine Is Listening: Corporate Disclosure in the Age of AI," Review of Financial Studies 36, no. 9 (2023): 3603–3642. Chapter 4 1. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey and Ilya Sutskever, "Robust Speech Recognition via Large-Scale Weak Supervision," arXiv:2212.04356 (2022). 2. R. David McLean and Jeffrey Pontiff, "Does Academic Research Destroy Stock Return Predictability?" Journal of Finance 71, no. 1 (2016): 5–32. Chapter 5 1. Campbell R. Harvey, Yan Liu and Heqing Zhu, "... and the Cross-Section of Expected Returns," Review of Financial Studies 29, no. 1 (2016): 5–68. 2. David H. Bailey and Marcos López de Prado, "The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting, and Non-Normality," Journal of Portfolio Management 40, no. 5 (2014): 94–107. 3. Paul Glasserman and Caden Lin, "Assessing Look-Ahead Bias in Stock Return Predictions Generated by GPT Sentiment Analysis," arXiv:2309.17322 (2023). 4. Suproteem K. Sarkar and Keyon Vafa, "Lookahead Bias in Pretrained Language Models," working paper (2024), presented at the International Conference on Machine Learning (2025). 5. Songrun He, Linying Lv, Asaf Manela and Jimmy Wu, "Chronologically Consistent Large Language Models," working paper (2025). Chapter 6 1. Refet S. Gürkaynak, Brian Sack and Eric T. Swanson, "Do Actions Speak Louder Than Words? The Response of Asset Prices to Monetary Policy Actions and Statements," International Journal of Central Banking 1, no. 1 (2005): 55–93. 2. Eric T. Swanson, "Measuring the Effects of Federal Reserve Forward Guidance and Asset Purchases on Financial Markets," Journal of Monetary Economics 118 (2021): 32–53. 3. David O. Lucca and Emanuel Moench, "The Pre-FOMC Announcement Drift," Journal of Finance 70, no. 1 (2015): 329–371. 4. Stephen Hansen, Michael McMahon and Andrea Prat, "Transparency and Deliberation Within the FOMC: A Computational Linguistics Approach," Quarterly Journal of Economics 133, no. 2 (2018): 801–870. Chapter 7 1. David O. Lucca and Francesco Trebbi, "Measuring Central Bank Communication: An Automated Approach with Application to FOMC Statements," NBER Working Paper 15367 (2009). 2. Agam Shah, Suvan Paturi and Sudheer Chava, "Trillion Dollar Words: A New Financial Dataset, Task and Market Analysis," Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (2023). 3. Miguel Acosta and Ellen E. Meade, "Hanging on Every Word: Semantic Analysis of the FOMC's Postmeeting Statement," FEDS Notes, Board of Governors of the Federal Reserve System (2015). Chapter 8 1. IFRS Foundation, IFRS S1 General Requirements for Disclosure of Sustainability-related Financial Information and IFRS S2 Climate-related Disclosures (June 2023). 2. Directive (EU) 2022/2464 (Corporate Sustainability Reporting Directive), and the Omnibus I simplification package proposed by the European Commission in February 2025 and finalised in early 2026. 3. US Securities and Exchange Commission, "The Enhancement and Standardization of Climate-Related Disclosures for Investors," Release No. 33-11275 (March 2024), and subsequent proceedings in Iowa v. SEC before the Eighth Circuit. 4. Nicolas Webersinke, Mathias Kraus, Julia Anna Bingler and Markus Leippold, "ClimateBERT: A Pretrained Language Model for Climate-Related Text," arXiv:2110.12010 (2021). 5. Julia Anna Bingler, Mathias Kraus, Markus Leippold and Nicolas Webersinke, "Cheap Talk and Cherry-Picking: What ClimateBERT Has to Say on Corporate Climate Risk Disclosures," Finance Research Letters 47 (2022): 102776. 6. Zacharias Sautner, Laurence van Lent, Grigory Vilkov and Ruishen Zhang, "Firm-Level Climate Change Exposure," Journal of Finance 78, no. 3 (2023): 1449–1498. 7. Robert F. Engle, Stefano Giglio, Bryan Kelly, Heebum Lee and Johannes Stroebel, "Hedging Climate Change News," Review of Financial Studies 33, no. 3 (2020): 1184–1216. 8. Florian Berg, Julian F. Kölbel and Roberto Rigobon, "Aggregate Confusion: The Divergence of ESG Ratings," Review of Finance 26, no. 6 (2022): 1315–1344. Chapter 9 1. Board of Governors of the Federal Reserve System and Office of the Comptroller of the Currency, "Supervisory Guidance on Model Risk Management," SR Letter 11-7 (April 2011). 2. Prudential Regulation Authority, Supervisory Statement SS1/23, "Model Risk Management Principles for Banks" (May 2023, effective May 2024). 3. Regulation (EU) 2024/1689 (Artificial Intelligence Act). 4. Financial Stability Board, "The Financial Stability Implications of Artificial Intelligence" (November 2024). 5. International Monetary Fund, "Advances in Artificial Intelligence: Implications for Capital Market Activities," Global Financial Stability Report, chapter 3 (October 2024). Further Reading Tim Loughran and Bill McDonald, "Textual Analysis in Accounting and Finance: A Survey," Journal of Accounting Research (2016).** The clearest single survey of the dictionary-era literature by the authors of the field's standard word lists, with candid warnings about the pitfalls of text measures. Matthew Gentzkow, Bryan Kelly and Matt Taddy, "Text as Data," Journal of Economic Literature (2019).** An accessible and rigorous overview of statistical methods for turning text into economic data, with attention to the problems of high dimensionality that underlie overfitting. Justin Grimmer, Margaret E. Roberts and Brandon M. Stewart, Text as Data: A New Framework for Machine Learning and the Social Sciences (Princeton University Press, 2022).** A book-length treatment of text analysis as a research method, strong on the discipline of defining what is being measured before choosing a model. Lewis Tunstall, Leandro von Werra and Thomas Wolf, Natural Language Processing with Transformers (O'Reilly, 2022).** A practical guide to building, fine-tuning and deploying transformer models, written by engineers at Hugging Face, for readers who want to build the systems this booklet describes. Marcos López de Prado, Advances in Financial Machine Learning (Wiley, 2018).** A demanding but essential book on why financial backtests fail and how to construct them honestly, applicable directly to text signals. Alan S. Blinder, Michael Ehrmann, Marcel Fratzscher, Jakob De Haan and David-Jan Jansen, "Central Bank Communication and Monetary Policy: A Survey of Theory and Evidence," Journal of Economic Literature (2008).** The standard survey of why and how central bank words affect markets, written as forward guidance was becoming a central tool. Ben S. Bernanke, 21st Century Monetary Policy: The Federal Reserve from the Great Inflation to COVID-19 (W. W. Norton, 2022).** A former chair's account of how communication became an instrument of policy, with detailed discussion of forward guidance episodes that text models are now used to analyse. Florian Berg, Julian F. Kölbel and Roberto Rigobon, "Aggregate Confusion: The Divergence of ESG Ratings," Review of Finance (2022).** The definitive study of why sustainability ratings disagree, and required reading before training any model on them. Hashtags: #NaturalLanguageProcessingInFinance #FinancialNLP #NLPInFinance #FinancialSentimentAnalysis #MarketPrediction #StockMarketPrediction #FinancialTextAnalytics #FinancialArtificialIntelligence #AIInFinance #FinTech #FinancialMachineLearning #FinancialDataScience #FinancialAnalytics #QuantitativeFinance #ComputationalFinance #AlgorithmicTrading #QuantitativeTrading #SentimentDrivenTrading #TextBasedTradingSignals #FinancialMarketForecasting #StockReturnPrediction #AssetPricePrediction #PredictiveAnalytics #FinancialNewsAnalysis #EarningsCallAnalysis #EarningsCallSentiment #EarningsCallTranscripts #ManagementToneAnalysis #CorporateDisclosureAnalysis #FinancialStatementAnalysis #SECFilings #AnnualReportAnalysis #TenKFilings #QuarterlyEarnings #FinancialTextMining #TextualSentiment #SentimentScoring #SentimentSurprises #ResidualTone #ManagementGuidance #ManagementUncertainty #EarningsSurprises #EarningsAnnouncementDrift #AnalystQuestions #QuestionAnswerAnalysis #ManagementEvasion #LinguisticUncertainty #FinancialRiskSignals #TopicModeling #FinancialTopicModeling #FinancialEmbeddings #SemanticAnalysis #TransformerModels #FinBERT #BERT #BloombergGPT #LargeLanguageModels #LLMsInFinance #FinancialLanguageModels #LoughranMcDonaldDictionary #FinancialPhraseBank #TextClassification #NaturalLanguageUnderstanding #ContextualEmbeddings #CentralBankCommunication #FederalReserveStatements #MonetaryPolicyAnalysis #MonetaryPolicySentiment #HawkishDovishSentiment #CentralBankTone #MarketExpectations #PolicySurprises #InterestRateExpectations #MacroeconomicSentiment #EconomicPolicyUncertainty #SustainabilityReporting #ESGTextAnalytics #ESGDisclosureAnalysis #ClimateRiskDisclosure #SustainabilityDataExtraction #ISSBStandards #FinancialModelRisk #ModelValidation #FinancialAIExplainability #ModelGovernance #PointInTimeData #LookAheadBias #DataLeakage #Backtesting #BacktestOverfitting #OutOfSampleTesting #WalkForwardValidation #TradingTransactionCosts #SignalDecay #AlphaSignals #PortfolioConstruction #InvestmentResearch #MarketEfficiency #BehavioralFinance
- IoT Security (Securing Embedded Devices and Over-the-Air Updates)
Download the Book (PDF): Introduction On the evening of 20 September 2016, the website of the security journalist Brian Krebs went dark under a flood of traffic measured at roughly 620 gigabits per second, among the largest denial-of-service attacks anyone had then recorded. The traffic did not come from a rented army of servers or from a nation's cyber command. It came from home routers, digital video recorders and, above all, cheap internet-connected cameras: devices bought to watch a front door or a sleeping baby, plugged in, and forgotten. A month later the same family of malware, by then called Mirai, knocked over the managed DNS provider Dyn and with it, for much of a working day on the east coast of the United States, access to services that ordinary people thought of as the internet itself. What made Mirai frightening was not its sophistication. It was the opposite. The malware broke into devices by trying a short list of factory usernames and passwords over a remote login service that most owners did not know was running. It exploited no clever memory corruption bug. It did not need to. Hundreds of thousands of devices had been designed, manufactured, certified for sale and shipped with a door that opened to anyone who knocked with the words written in the manual. A decade on, the technology has moved and the lesson has not entirely landed. Botnets assembled from routers, cameras and television boxes still set records for attack volume; in 2025 one of them, known as Aisuru, was linked by Cloudflare to attacks peaking near thirty terabits per second, about fifty times the attack that took Krebs offline. State-linked groups have been caught building covert networks out of tens or hundreds of thousands of small-office routers and cameras, using them not for noisy floods but as anonymous relays for espionage. Millions of inexpensive streaming boxes have been found to arrive from the factory with malware already installed. The attack surface has not shrunk. It has grown with every product category that acquired a radio and an app. This booklet is written for the people who can change that: the engineers, architects and product leaders who design connected devices, and the security engineers who have to make those devices defensible for years after they leave the factory. It is a defensive book. It describes how attacks work at the level a designer needs in order to stop them, and spends its effort on prevention. There is no exploit code here, and no step-by-step recipe for compromising anything. The argument The controlling idea of the booklet is simple to state and hard to execute. The security of a connected device is decided in two places: in the chain of trust that runs from silicon to application at boot, and in the update channel that has to keep that chain intact for the device's entire working life. Almost every serious failure in consumer and industrial IoT can be traced to a weakness in one of the two. Default passwords, the most notorious failure of all, are best understood not as a configuration mistake by users but as a symptom of products that treat security as something the owner is supposed to add, rather than a property the maker builds in. This framing matters because device security is otherwise easy to approach as an endless checklist. Encrypt traffic. Disable debug ports. Sanitise input. Use strong cryptography. Each item is correct, and a team that works through such a list will produce a better product than one that does not. But checklists do not tell you which items are load-bearing. A device with perfect transport encryption and an unsigned firmware image is not secure: anyone who can get code onto it owns the keys that encryption depends on. A device with a beautifully verified boot chain and no way to ship a fix is secure only until the first vulnerability is found in the code that chain so carefully verifies. A device with both, and a factory-set password shared by every unit, can still be conscripted into a botnet in the minutes after it is plugged in. The two foundations reinforce each other. A hardware root of trust is what lets a device distinguish its maker's code from an attacker's. A secure update system is what allows the maker to keep supplying code worth trusting. Without the first, updates are an attack vector; without the second, the root of trust protects a body of software that is decaying in public. Most of what follows is an elaboration of how to build each well and how to keep them working together. What the booklet covers The first chapter reconstructs Mirai and its successors conceptually, because the way those botnets spread reveals precisely which design decisions made them possible. The second maps the attack surface of a connected appliance as a whole, from debug pins on the circuit board to the cloud service and phone app that sit behind it, and argues for threat modelling as the discipline that keeps that map honest. The third deals squarely with credentials: why universal default passwords persist, what has replaced them in well-designed products, and how regulation in several jurisdictions has finally made them illegal. The next two chapters build the chain of trust. Chapter 4 explains hardware roots of trust, from one-time-programmable fuses and boot ROM to secure elements, trusted execution environments and identity schemes such as DICE, and the unglamorous but critical work of provisioning keys in a factory. Chapter 5 follows the boot process itself, distinguishing verified from measured boot, explaining anti-rollback protection, and cataloguing the ways secure boot has failed in practice. The following two chapters are about updates. Chapter 6 treats the over-the-air update as a reliability problem as much as a security one: how to make updates atomic, recoverable and safe on devices that lose power at the worst moment. Chapter 7 turns to the update pipeline behind the device, the build systems, signing keys and distribution servers whose compromise would turn a vendor's own update mechanism into the most efficient malware distribution network it could ever build, and to the frameworks, such as The Update Framework and Uptane, designed to survive such compromises. The final chapter considers the device across its lifetime: vulnerability disclosure, support periods, end of life, and the regulatory regimes, notably the United Kingdom's product security law, the European Union's Cyber Resilience Act and the United States' labelling programme, that are turning good practice into legal obligation. What it leaves out A short book is defined by what it declines to cover, and this one declines a good deal. It does not survey the security of particular wireless protocols in depth, though it touches on them where they bear on the argument. It says little about privacy, which is a large and important subject in its own right. It does not attempt to cover the specialised world of industrial control systems and safety-critical operational technology, whose constraints and standards differ enough to need their own treatment, though many of the principles transfer directly. And it does not teach cryptography: it assumes the reader knows, or can quickly learn, what a digital signature is and why a private key must stay private, and focuses on how such primitives are assembled into systems that survive contact with real attackers, real factories and real customers. A word on audience and tone The intended reader is technically literate but not necessarily a specialist in embedded security. A firmware engineer who has never provisioned a key, a cloud architect suddenly responsible for a fleet of devices, a product manager who must sign off on a support period, a security engineer coming from web applications: each should find the material accessible and, I hope, occasionally uncomfortable. Where the text uses a term of art it defines it in place. The tone is deliberately practical. Device security has a literature of alarm, full of numbers about how many billions of devices are connected and how many are vulnerable. Those numbers are real enough, but they are not useful to someone deciding on Tuesday whether to fuse the debug port shut or how to rotate a signing key. The useful questions are narrower. What has to be true at the moment of manufacture for this device to be trustworthy? What has to be true of the update system for that trust to last ten years? What will happen to this product when the company stops caring about it, and what should happen instead? The chapters that follow try to answer those questions in enough detail to act on. Chapter 1: What Mirai Taught The most useful way to read a famous botnet is as a design review conducted by an adversary. Every stage of its life cycle, from finding victims to holding on to them, depends on some property of the devices it infects. Those properties were chosen by people: by an engineer who left a service enabled, a product manager who decided a unique password per unit would complicate the production line, a company that never planned to ship a second firmware version. Reconstructing how Mirai worked, at the conceptual level a defender needs, tells us which of those choices mattered most. The life cycle of an IoT botnet Mirai's operation can be described in four stages: discovery, entry, installation and command. None of them is unique to Mirai, which is why the pattern is worth learning. Discovery was brute and indiscriminate. Infected devices scanned random addresses across the IPv4 internet, looking for hosts that answered on the ports used by Telnet, the decades-old unencrypted remote login protocol. Mirai looked at the standard Telnet port and at an alternative port that many embedded vendors used for the same service. Scanning the entire IPv4 space is cheap; a fast scanner can do it in under an hour, and a botnet of hundreds of thousands of scanners does it continuously. The defensive lesson is that obscurity of address provides no protection whatsoever. A device that exposes a service to the internet will be found, usually within minutes of being connected. Entry relied on a list of factory credentials. The original Mirai source code, which its authors published on a hacking forum at the end of September 2016, contained a little over sixty username and password pairs. They were not secret. Many were printed in user manuals or support pages; others were the kind of combinations that anyone who has configured a cheap router would guess. Some belonged to accounts that the user could not change because they were not exposed in the device's interface at all: they were maintenance or vendor accounts, baked into the firmware for convenience and left there. The botnet did not need to know which brand of device it was talking to. It simply tried each pair in turn until one worked. Installation was minimal. Once logged in, the compromised device was instructed to fetch and run a small binary built for its processor architecture. Mirai ran in memory and did not attempt to modify the device's persistent storage, which meant a reboot cleared the infection. In practice that hardly mattered, because the device came back up with the same exposed service and the same default credentials, and the botnet's scanners would find and reinfect it again quickly. Mirai also took steps to defend its territory: it tried to kill other processes bound to the remote-access ports and to remove competing malware, effectively closing the door behind itself so that rival botnets could not take its victims. Command was centralised. Infected devices reported to command-and-control servers, which issued instructions to launch floods of various kinds against chosen targets. The attacks themselves were not novel. What was novel was the sheer number of sources and their distribution across residential and business networks worldwide, which made them hard to filter without also blocking legitimate users. A detailed academic reconstruction of the botnet, published at the USENIX Security Symposium in 2017 by a team led by Manos Antonakakis, tracked its growth and found that it peaked at roughly 600,000 infected devices. The same study found that the bulk of identifiable victims were cameras, digital video recorders and routers, with a long tail of other product types, and that a relatively small number of manufacturers accounted for a large share of the population. That concentration is itself a lesson. A single design decision, replicated across a product line and sold through white-label resellers under many brand names, can put hundreds of thousands of insecure devices into the world. The attacks that followed The attack on Krebs's site in September 2016 was followed within days by a larger one against the French hosting company OVH, which reported traffic in excess of a terabit per second. Then, on 21 October 2016, Mirai-based botnets were turned on Dyn, a provider of managed DNS. Because many large web services depended on Dyn to translate their domain names into addresses, the attack made those services unreachable for many users in waves throughout the day. Nothing had been breached in the conventional sense. Websites were simply impossible to find. In November 2016, a Mirai variant equipped with a different entry technique, exploiting a remote management interface exposed on certain broadband routers, caused roughly 900,000 customers of Deutsche Telekom to lose connectivity. The routers were not successfully recruited; the infection attempts crashed them. This episode deserves attention because it shows a second pathway into the same devices. Default credentials were Mirai's signature, but the code was modular, and its successors added exploits for known vulnerabilities in exposed web interfaces and management protocols. A device that has eliminated default passwords but cannot be patched is only one public vulnerability away from the same fate. The people behind the original botnet turned out to be young Americans who had been running a business selling protection against denial-of-service attacks, including to operators of Minecraft game servers, while also launching such attacks. Three of them pleaded guilty to federal charges in late 2017. Their prosecution did not end the problem. The publication of the source code had already turned Mirai into a template, and variants multiplied. Security researchers have catalogued many dozens of them over the years, each adding new credential lists, new exploits for specific device models, or new processor architectures. Successors and variations Mirai's descendants and contemporaries illustrate how the basic pattern evolves, and each evolution points to a design weakness in devices. Some botnets abandoned central command servers in favour of peer-to-peer communication, so that infected devices coordinate among themselves and there is no single server to take down. Hajime, which appeared in late 2016, worked this way, and so did Mozi, which became one of the most widespread IoT botnets of the late 2010s and early 2020s before an apparent kill switch disabled much of it in 2023. Peer-to-peer designs make takedowns harder, which shifts the burden of defence even further onto the devices themselves. Others turned destructive. BrickerBot, which surfaced in 2017, broke into devices using techniques similar to Mirai's and then deliberately corrupted their storage, rendering them useless. Its author presented the campaign as a form of vigilante hygiene: better a dead camera than a weaponised one. Whatever one thinks of the ethics, BrickerBot demonstrated that the same access that allows a device to be conscripted also allows it to be destroyed, and that a product with no recovery mechanism can be turned into electronic waste remotely. Later generations moved from noise to stealth. In January 2024 the United States Department of Justice announced a court-authorised operation to disrupt a botnet of small-office and home-office routers, many of them models that had reached end of life and no longer received security updates, which had been used by a state-sponsored group to conceal its activities against critical infrastructure. In September 2024, a joint advisory from American and allied agencies described a botnet called Raptor Train, attributed to a Chinese company working with a group known as Flax Typhoon, which had compromised more than 260,000 devices, including routers, cameras, video recorders and network storage appliances. These networks were not built to flood websites. They were built to provide a large, constantly changing pool of residential and business addresses through which espionage traffic could be relayed, making it look like ordinary users. At the same time, the flooding capability has grown. Throughout 2025, the botnet known as Aisuru, built largely from compromised routers and other consumer devices, was linked to a succession of record-setting attacks, with Cloudflare reporting peaks of around thirty terabits per second. The arithmetic behind those numbers is a reminder that consumer broadband has become much faster since 2016. A device on a gigabit fibre connection that is turned against a target contributes far more traffic than a camera on a 2016 cable line. Finally, there is the supply-chain variant, in which devices do not need to be broken into at all. In 2025 the FBI warned that a campaign it called BadBox 2.0 had compromised more than a million inexpensive Android-based streaming boxes, projectors and tablets, many of them sold with malicious software already present when they left the factory, or delivered through the first update after setup. Here the owner did nothing wrong, and no network-facing vulnerability was required. The compromise lay in the firmware supply chain itself. Why devices make such good recruits It is worth pausing on why connected appliances, rather than laptops or servers, became the raw material of choice for botnet operators. The answer lies in a combination of properties that each seem harmless in isolation. Connected devices are always on. A laptop is closed at night and carried between networks; a camera or router runs continuously, on the same connection, for years. That makes it a reliable, predictable resource. They are rarely monitored. Nobody installs endpoint security software on a video recorder, and few owners would notice if its processor were busy or its upload bandwidth consumed, because there is no screen on which to notice it and no habit of looking. They usually run a general-purpose operating system, most often a stripped-down Linux, which means an attacker can write malware once and compile it for the handful of processor architectures that cover most of the market. And they sit on residential and small-business connections, whose addresses are hard to block because they are shared with legitimate users. That last property has become more valuable over time. Beyond flooding, there is now a substantial commercial market for residential proxy services, which route customers' traffic through addresses that look like ordinary households, for purposes ranging from legitimate price monitoring to fraud, credential stuffing and evasion of anti-abuse systems. Some of the capacity behind those services comes from consenting participants; some, according to repeated findings by researchers and law enforcement, comes from compromised routers, cameras and television boxes whose owners have no idea their connections are being resold. A compromised device is thus not merely a weapon for a single attack but an asset that can be rented out continuously. Finally, devices are often owned by people who are not in a position to defend them. A household that buys a camera has no security team, no inventory of its connected equipment and no process for applying patches. The maker is the only party with the knowledge and access to defend the device, which is the practical reason this booklet addresses itself to makers rather than to users. What the pattern reveals Set side by side, these episodes reveal a small set of recurring design failures. Each is addressed by a later chapter of this booklet, and together they amount to a specification for what a device must not be. The first is shared, known credentials on an exposed service. Mirai's original attack required both conditions: a remote login service reachable from the internet, and an account whose password was the same on every unit. Remove either and the original technique fails. Chapter 3 treats this problem in detail. The second is unnecessary exposure. Many of the devices Mirai infected did not need Telnet at all for their intended function. The service was there for development or factory testing and was never removed. Every listening service is a commitment to defend that code for the device's lifetime, and most products make that commitment without noticing. Chapter 2 deals with mapping and minimising the attack surface. The third is the absence of a working update path. The Deutsche Telekom routers could be patched remotely by the operator, and were, within days. Most of the cameras Mirai conscripted could not, either because the vendor never released an update or because the update process required the owner to download a file from a website and upload it through a web interface they had never opened. The routers in the 2024 state-sponsored botnet had been declared end of life by their makers. The cameras and recorders in Raptor Train ran firmware with long-known vulnerabilities. A device that cannot be fixed will eventually be broken. Chapters 6 and 7 treat updates. The fourth is the absence of integrity guarantees. Mirai did not persist across reboots, but many later families do, by modifying firmware or startup scripts. A device that verifies its own software at boot, and refuses to run anything its maker did not sign, makes persistence far harder and makes recovery possible: a reboot returns it to a known-good state. A device that does not verify itself can be permanently altered, and in the supply-chain case may never have been in a known-good state at all. Chapters 4 and 5 treat the hardware root of trust and secure boot. The fifth is the lack of any end-of-life plan. Products are sold for a few years and used for many more. The older a device gets, the more likely it is to run code with published vulnerabilities, and the less likely its maker is to care. Chapter 8 treats the device over its lifetime and the regulation now addressing this gap. Why the market did not fix it It is worth asking why these failures persisted for so long after Mirai made them famous. The answer is not ignorance. By 2017, every competent security team in the industry understood the problem. The answer is economics. The cost of an insecure device falls mainly on people other than its buyer and maker. When a camera is conscripted into a botnet, its owner usually notices nothing; the picture still works. The victim is a website on the other side of the world, or an internet user who cannot reach a service, or an organisation whose network is being probed by espionage traffic relayed through someone's living room. Economists call this an externality, and markets handle externalities badly. Buyers cannot see security when they choose between two cameras, and would not pay much for it if they could, because the harm it prevents does not fall on them. Meanwhile, the costs of security fall squarely on the maker. Unique credentials require changes to the production line. A hardware root of trust adds components and provisioning steps. A secure update system needs servers, signing infrastructure, and engineers to maintain both, for years after the product has stopped generating revenue. In a market where devices compete on price and margins are thin, each of these is a cost a competitor may choose not to bear. This is why the past few years have seen a shift from guidance to law. The United Kingdom's product security regime, which took effect in April 2024, bans universal default passwords outright. The European Union's Cyber Resilience Act, which entered into force in December 2024, will make security requirements, vulnerability handling and update support a condition of selling connected products across the single market. These developments, discussed in Chapter 8, do not change the engineering. They change the economics, by making insecure design a legal liability rather than a cost saving. For a device maker, the practical consequence is that the defensive measures in this booklet are no longer optional virtues. They are becoming the price of market access. A defender's reading There is one more lesson in Mirai, which is easy to miss because it is so obvious. The botnet's authors did not find anything that the makers of those devices could not have found. Every weakness they exploited was visible to anyone who unboxed the product, plugged it in and ran a port scan, or read its manual. The vendors did not lack the information needed to prevent the attack. They lacked a process that would have caused someone to look, and a set of incentives that would have caused someone to act on what they saw. That is encouraging, because process and incentives are things organisations can change. The rest of this booklet is about what the process should contain. It begins, in the next chapter, with the question every device team should answer before choosing a single component: what, exactly, is exposed, and to whom? Chapter 2: The Attack Surface of a Connected Appliance A connected appliance is not a single thing to secure. It is a small distributed system that happens to be sold in a box. The camera on the shelf has a processor running an operating system, a radio or Ethernet port, a set of network services, a companion phone application, a cloud backend that brokers access when the owner is away from home, a firmware update server, and a manufacturing history that determines what code and keys were placed on it before it was packaged. Each of these is a way in. The attack surface of the product is the sum of all of them, and an attacker needs only one. This chapter maps that surface layer by layer, then argues that the only reliable way to keep the map accurate is to make threat modelling a standing part of the engineering process rather than a document produced once for an audit. Five layers of exposure It helps to think of a device's attack surface as five layers, ordered roughly by how close an attacker must be to use them. Table 1 summarises them; the discussion that follows explains why each matters. Table 1. The five layers of a connected device's attack surface. Layer Typical entry points Attacker position Primary defences Physical hardware Debug ports, exposed flash, test pads, buses Hands on the device Debug lockdown, encrypted storage, tamper resistance Firmware and OS Bootloader, kernel, services, parsers Code execution or crafted input Secure boot, memory safety, least privilege Local network and radio Wi-Fi, Bluetooth, Zigbee, Thread, LAN services Same network or radio range Authenticated pairing, encryption, minimal services Internet-facing services Remote login, web UI, port forwarding, P2P relays Anywhere online No inbound exposure, strong authentication Cloud, app and supply chain Accounts, APIs, update servers, factory Anywhere, or inside the vendor Account security, API authorisation, signed builds Physical hardware The innermost layer is the circuit board itself. Almost every embedded design includes interfaces used during development and manufacturing: a serial console, a JTAG or SWD debug port that allows a connected probe to halt the processor and read or write memory, test pads used by factory fixtures, and flash memory chips that can be desoldered and read in a programmer. On many consumer devices these interfaces remain fully functional in the shipped product. A serial console may drop straight into a root shell. A debug port may allow the entire contents of memory, including keys, to be extracted in seconds. Physical attacks are sometimes dismissed on the grounds that the attacker needs the device in hand. That reasoning is incomplete in two ways. First, many devices are physically accessible to people other than their owner: a doorbell camera is mounted on the outside of a house, a smart meter sits in a shared basement, a parking sensor is in the street. Second, and more important, an attacker who extracts a secret from one unit may be able to use it against every unit. If a firmware image is encrypted with a key that is the same on every device, or if a cloud service accepts a credential that is shared across a product line, then one purchase and an afternoon with a soldering iron yields an attack that scales to the whole fleet. The defensive principle is that physical access to one device must never compromise others. That principle, more than any attempt to make a device impossible to open, is what hardware security design is mostly about. Firmware and operating system The second layer is the software running on the device. On a microcontroller this may be a real-time operating system and a single application; on a camera or router it is often a Linux distribution assembled from hundreds of open-source packages, many of them old versions, together with the vendor's own code and binary drivers supplied by the chip maker. Every component that parses input from outside is a potential vulnerability: a web server handling configuration requests, a media library decoding video streams, a network stack processing packets, a firmware update routine unpacking an archive. Embedded firmware has historically been written in C, without the exploit mitigations that desktop operating systems adopted long ago. Address space layout randomisation, stack canaries and non-executable memory are inconsistently enabled on embedded Linux and frequently absent on microcontrollers. Processes commonly run as root. Command injection, in which input supplied through a web form is passed to a shell, remains one of the most common vulnerability classes in consumer routers and cameras precisely because firmware developers often implement configuration changes by assembling shell commands. Two developments are improving this picture. The first is broader use of memory-safe languages, particularly Rust, in new embedded code, which removes whole classes of memory corruption bugs. The second is the slow adoption of hardware features such as memory protection units and privilege separation on microcontrollers, which allow even small devices to isolate their most sensitive functions. Neither helps a product that is built on a decade-old software stack with no plan to update it, which is why firmware integrity and updatability are treated as foundations later in this booklet. The third-party component problem deserves particular emphasis. In 2020, researchers disclosed two large families of vulnerabilities in widely used embedded TCP/IP stacks: Ripple20, in a commercial stack licensed to many manufacturers, and Amnesia:33, across several open-source stacks. The affected code was buried deep in products from many vendors, often without the vendors themselves being fully aware it was there. A device maker cannot defend code it does not know it ships. That is the practical case for a software bill of materials, an inventory of every component and version in a firmware image, which is discussed in Chapter 7. Local network and radio The third layer comprises the interfaces that face the local network and nearby radios: Wi-Fi, Bluetooth Low Energy, Zigbee, Thread, and whatever services the device offers to other machines on the same LAN. Historically, many devices treated the home network as trusted, offering unauthenticated configuration pages or discovery protocols to anything that could reach them. That assumption fails in practice. Home networks contain laptops that browse malicious websites, phones that install dubious apps, and other IoT devices that have already been compromised. An attacker who controls any one machine on the network can reach every other device that trusts its neighbours. Universal Plug and Play deserves a particular mention. Many routers support a feature that allows devices on the LAN to ask the router to open inbound ports. Cameras and video recorders have used it to make themselves reachable from the internet without the owner configuring anything. This was one way in which devices behind home routers came to have Telnet and web interfaces exposed to Mirai's scanners. A device that opens holes in its owner's firewall without asking has effectively moved itself from the third layer to the fourth. Radio protocols bring their own concerns. Pairing is the critical moment: if the procedure by which a new device joins a network or binds to an owner's phone can be observed or spoofed, the security of every subsequent message may be undermined. Modern protocol designs, including Bluetooth's LE Secure Connections and the commissioning flow in the Matter smart-home standard, use authenticated key exchange and out-of-band information such as a QR code printed on the device to resist this. Products that implement older or proprietary pairing schemes, or that skip authentication to make setup smoother, give that resistance away. Internet-facing services The fourth layer is anything an attacker anywhere in the world can reach. This was Mirai's layer, and it remains the most dangerous, because exposure to the internet means exposure to automated, continuous, indiscriminate scanning. Search engines dedicated to indexing internet-connected devices make it trivial to find every instance of a particular camera model, often with its firmware version visible in the banner it returns. The single most effective architectural decision a consumer device maker can take is to expose nothing to the internet at all. Well-designed modern devices make outbound connections to their maker's cloud service over authenticated, encrypted channels, and receive commands over those connections. They accept no inbound connections from the internet. The owner's phone talks to the cloud, the cloud talks to the device, and there is no listening port for a scanner to find. This model has costs, which the next section discusses, but it eliminates the class of attack that built the original botnets. Where a device must accept inbound connections, for instance an enterprise camera recording to an on-premises server or an industrial gateway, the exposure should be explicit, documented, opt-in, and protected by strong per-device authentication. A service that is reachable from the internet by default, and protected only by a password, is the configuration that every IoT botnet has been built to exploit. Cloud, app and supply chain The outermost layer moves the target away from the device entirely. The cloud service that brokers access to a fleet of devices is itself an enormously attractive target, because compromising it compromises every device it controls. In March 2021, attackers gained access to the systems of Verkada, a vendor of cloud-managed security cameras, using administrative credentials they reported finding exposed on the internet, and were able to view live feeds from many thousands of cameras installed in hospitals, schools, prisons and companies. No individual camera was hacked. The architecture that removed exposure from the device had concentrated it in the cloud. Customer accounts are a related weakness. In late 2019, owners of Ring home cameras reported strangers speaking to them, and to their children, through compromised devices. The attackers had not broken Ring's systems; they had used passwords leaked from other websites and reused by the account holders, an attack known as credential stuffing. The company responded by making two-factor authentication mandatory. The lesson for device makers is that the account is part of the device's attack surface. If an attacker who has a customer's reused password can watch their home, the product is insecure, regardless of how well the camera itself is built. The companion app is part of this layer too, and is often reviewed less carefully than the device. Mobile apps are distributed to anyone who wants them and can be decompiled with freely available tools. Security researchers examining smart-home apps have repeatedly found embedded cloud API keys, hard-coded credentials for vendor services, and cloud endpoints that trusted the app to enforce which devices a user could access rather than checking authorisation on the server. In the worst cases, changing a device identifier in a request was enough to view another customer's camera. The principle is that nothing distributed to every customer, whether firmware or app, can hold a secret, and that every authorisation decision must be made on the server side, per user and per device. Behind all of this sits the supply chain: the build servers where firmware is compiled, the signing keys that authenticate it, the update servers that distribute it, the contract manufacturer that flashes it, and the component suppliers whose code it contains. The BadBox campaigns described in Chapter 1 showed how devices can be compromised before they ever reach a customer. The update pipeline is important enough to have a chapter of its own. The cloud trade-off Moving every device behind a vendor cloud is the dominant pattern in consumer IoT, and on balance it is the right one, but its costs should be faced honestly. It creates the concentration risk illustrated by Verkada: a single breach of the vendor's systems can affect the entire fleet. It creates a dependency on the vendor's continued existence: when a company shuts down a service, devices that rely on it may stop working, as owners of several discontinued smart-home products have discovered. And it raises privacy questions about what data flows to the vendor and what the vendor does with it. These costs do not argue for returning to devices that listen on the internet. They argue for designing the cloud architecture with the same care as the device. Administrative access to the fleet should be protected with strong, hardware-backed authentication and tightly scoped privileges. Commands to devices should be authorised per device and per user, and devices should be able to verify that commands come from the genuine service. Where feasible, sensitive data such as video can be end-to-end encrypted between device and owner, so that the cloud relays it without being able to read it. And products should be designed to retain some local function if the cloud goes away, an obligation that the end-of-life discussion in Chapter 8 returns to. Threat modelling as a habit A map of the attack surface is only useful if it is accurate, and it will not stay accurate on its own. Features are added; a debug service is enabled for a field trial and never removed; a new cloud API is exposed for a partner integration. Threat modelling is the discipline of asking, systematically and repeatedly, what could go wrong with a design, who might make it go wrong, and what the product does to prevent it. There are formal methods for this, of which the best known is STRIDE, a mnemonic developed at Microsoft that prompts designers to consider spoofing, tampering, repudiation, information disclosure, denial of service and elevation of privilege against each element of a system. Adam Shostack's book on threat modelling remains the standard practical introduction. But the method matters less than the habit. A useful threat model for an IoT product answers four questions for every interface on the map. What does it expose? Who can reach it, from where? What happens if it is abused? And what in the design prevents that, or detects it, or allows recovery? Several practices make this habit stick. The threat model should be a living artefact maintained alongside the architecture, not a report written once for a certification. It should be reviewed whenever an interface changes. It should be tested against the shipped product, not the intended design: a port scan of a production unit, an inspection of what services actually start at boot, and an attempt to connect to the debug port on a board pulled from the production line will regularly reveal differences between what the team believes the device exposes and what it actually exposes. And it should explicitly include the factory and the update pipeline, which are the parts of the system that designers most often forget are part of the system at all. Minimising what remains The most powerful outcome of threat modelling is usually deletion. Every service that is not needed should be removed from the production firmware, not merely disabled by a configuration flag that an attacker might flip. Development conveniences, such as a shell on the serial port, a Telnet daemon or a hidden factory account, should be built out of production images entirely. Ports that must exist should listen only on the interfaces that need them. Processes should run with the least privilege required for their task, isolated from each other so that a flaw in a media parser does not grant control of the update mechanism. The European standard for consumer IoT security, ETSI EN 303 645, captures this in one of its thirteen provisions: minimise exposed attack surfaces. The phrase is easy to agree with and hard to live by, because every exposed service exists because someone found it useful. The discipline lies in recognising that usefulness during development is not a reason for exposure in the field, and that every interface shipped is a promise to defend it for as long as the device remains in use. With the surface mapped and minimised, the most notorious single weakness on it can be addressed. The next chapter deals with credentials, and with the long overdue end of the universal default password. Chapter 3: Ending the Default Password Of all the weaknesses that made Mirai possible, the universal default password was the most avoidable and the most persistent. It had been understood as a problem for at least twenty years before 2016. Security guidance had warned against it repeatedly. And yet, at the moment Mirai's authors compiled their list of a few dozen credential pairs, it was enough to conscript the better part of a million devices. Understanding why the practice survived for so long is the key to making sure it does not return in subtler forms. Why default passwords survived A universal default password is a credential that is identical across every unit of a product, or derivable from public information such as the model number or the MAC address. It exists because it makes several things easy. The factory can test every unit with the same script. Support staff can talk a customer through logging in without knowing anything about their particular device. Installers can configure a building full of cameras without keeping a spreadsheet. Documentation can say "log in with admin and admin" and be correct for every reader. Each of these conveniences is real, and each is paid for by the owner and by everyone else on the internet. The owner is supposed to change the password during setup. In practice most do not, often because the device works without them doing so and nothing tells them it matters. Worse, many products had additional accounts the owner could not see: a vendor support account, a factory test account, a debug account left in the firmware by an engineer who needed it once. These hidden credentials could not be changed through any interface the owner had access to, and some were hard-coded into the firmware itself. The deeper reason the practice persisted is that the alternative requires a change to the production process, and production processes are expensive to change. Giving every device a unique credential means generating that credential, programming it into the device, printing it on a label that matches the specific unit, and making sure the two never get separated on the way to the customer. For a manufacturer producing millions of units on thin margins, or for a white-label supplier whose firmware is shipped under dozens of brand names by resellers who never look inside it, that is real work. Avoiding it was rational for the individual firm. It was disastrous for everyone else. What the law now says That calculation has changed because the law has changed. California's SB-327, in force since January 2020, requires connected devices sold in the state to have reasonable security features, and specifies that if a device offers authentication from outside the local network, it must either have a password unique to each device or require the user to set a new credential before first access. Oregon passed a similar law at the same time. The United Kingdom went further. Under the Product Security and Telecommunications Infrastructure Act 2022, a set of security requirements for consumer connectable products came into force on 29 April 2024. The first and most prominent is a ban on universal default passwords: passwords must either be unique per product or set by the user, and must not be easily guessable, for instance by being based on incremental counters or derived from publicly available information such as a serial number. The other two requirements oblige manufacturers to publish a way to report vulnerabilities and to state the minimum period for which the product will receive security updates. These three requirements correspond directly to the first three provisions of ETSI EN 303 645, the European standard for consumer IoT security that the UK government helped to develop. The regulator can impose fines of up to ten million pounds or four per cent of worldwide turnover. The European Union's Cyber Resilience Act takes a broader approach, requiring products to be delivered with a secure default configuration and protected against unauthorised access through appropriate authentication mechanisms, among many other essential requirements. Its main obligations apply from December 2027. In combination, these laws mean that a product with a universal default password can no longer be legally sold in some of the world's largest consumer markets. Patterns that replace the default password The law states what is prohibited. Engineering has to supply what replaces it. There are four broadly successful patterns, and well-designed products often combine them. Unique per-device credentials The most direct replacement is a credential generated uniquely for each unit during manufacture and printed on a label on the device or its packaging. Many routers now ship this way, with a Wi-Fi passphrase and administrative password on a sticker underneath. The pattern is simple and familiar to users, but it has to be done carefully to be worth anything. The credential must be generated randomly, using a cryptographically secure generator, not derived from the serial number, MAC address or any other value an attacker can observe or guess. Several router manufacturers in the past generated "unique" default Wi-Fi passwords by applying a fixed algorithm to the device's MAC address, which the router broadcasts. Once researchers reverse-engineered the algorithm from one unit's firmware, every unit's password could be computed from a distance. The UK's regulations explicitly forbid this. The credential must have enough entropy to resist guessing given whatever rate limiting the device applies, and the device should store only a salted hash of it, so that dumping one unit's storage does not reveal its password in readable form. And the production process must ensure that the label printed for a unit matches the credential programmed into that unit, which is a logistics problem as much as a security one and which tends to be the place where projects stall. User-set credentials at first use The second pattern is to ship with no usable credential at all and require the owner to set one before the device does anything useful. This is common for devices with a screen or a local setup interface. Its strength is that there is no factory secret to leak. Its weakness is the window between power-on and setup, during which the device must be protected by some other means. A device that offers an unauthenticated setup page to anyone on the network, or worse, to the internet, can have its first password set by an attacker before its owner gets there. The setup interface should therefore be reachable only locally, for a limited period after power-on, and ideally only after a physical action such as pressing a button on the device. App-based onboarding without passwords The dominant pattern in modern consumer IoT eliminates the device password altogether. The owner installs the manufacturer's app, scans a QR code on the device or its packaging, and the app and device perform an authenticated exchange over Bluetooth or a temporary Wi-Fi network. The device joins the owner's network, connects outbound to the manufacturer's cloud, and is bound to the owner's account. From then on, the owner controls the device by logging in to their account, and there is no device-level password for anyone to guess. Done well, this is a large improvement, and it is the model adopted by the Matter smart-home standard. In Matter, each device carries a setup code, typically presented as a QR code, that is used in a password-authenticated key exchange during commissioning. Each device also carries a device attestation certificate, issued under a hierarchy of certificate authorities rooted in the standard's governing body, so that the commissioning device can verify that the product is a genuine certified device and not a counterfeit. The setup code is unique to the device and is used only during commissioning; it does not become a long-lived password. Done badly, app-based onboarding simply moves the password problem to the cloud account, where it is no less exploitable. The Ring incidents described in Chapter 2 were a direct consequence: devices with no default password were nonetheless taken over through reused account passwords. App-based onboarding is therefore only as strong as the account security behind it. That means offering, and increasingly requiring, multi-factor authentication; supporting passkeys, which resist phishing and cannot be reused across sites; detecting credential stuffing through rate limits and checks against known breached passwords; and notifying owners when a new device or a new phone gains access to their home. Certificate-based device identity The fourth pattern applies to the device's own identity rather than to any human's login. Rather than authenticating to the cloud service with a shared secret or a password, each device holds a unique private key, ideally generated inside a secure element or other protected hardware so that it never exists outside the device, and a certificate binding the corresponding public key to the device's identity. The device proves its identity by signing a challenge, typically as part of a mutually authenticated TLS connection to the cloud. Compromising one device yields at most that device's key, and the service can revoke it individually. This pattern is the foundation on which the others rest. A device whose cloud connection is authenticated with a credential shared across the product line is one firmware extraction away from an attack on the whole fleet. A device with a unique, hardware-protected key is not. Chapter 4 discusses how such keys are created and provisioned in the factory. Credentials beyond the login screen Eliminating the default password from the login screen is necessary but not sufficient. Several related weaknesses deserve the same treatment. Hard-coded secrets in firmware are the most serious. A firmware image that contains an API key for the vendor's cloud, a private key for a TLS server, or a symmetric key used to encrypt local traffic, and that contains the same value on every device, gives anyone who extracts it the ability to impersonate the vendor or the device. In 2015, the security firm SEC Consult analysed thousands of firmware images from embedded devices and found that a few hundred private keys for HTTPS and SSH were shared across millions of internet-facing devices, because vendors had simply compiled the same keys into every build. Anyone who obtained the key from one firmware image could impersonate or eavesdrop on all of them. Secrets must be generated per device, ideally inside the device, and never stored in the firmware image that is distributed to every unit. Service and debug accounts are the next. Every account on the device should be visible in its documentation, controllable by the owner or removed entirely from production builds. Hidden accounts for vendor support, whatever their convenience, are indistinguishable from backdoors once someone finds them, and someone always does. Rate limiting and lockout matter even for strong credentials. A device that accepts unlimited login attempts at network speed invites brute-force attack, and some devices have been found that accept short numeric PINs over the network with no limit at all. Exponential back-off after failed attempts, combined with credentials of adequate length, makes guessing impractical without making the device trivially lockable by an attacker who wants to deny service to its owner. Recovery mechanisms deserve scrutiny. The reset button that returns a device to factory settings must return it to a state that is as secure as the out-of-box state, not to a state with a known universal password. And any remote password recovery flow must be designed with the same care as the login it bypasses, since an attacker will use whichever is weaker. Professional installation without shared secrets One argument often raised in defence of default credentials concerns professional deployments. An installer commissioning two hundred cameras across a warehouse, or an integrator configuring a building's worth of access controllers, cannot reasonably peel and transcribe two hundred labels, and has historically relied on every unit accepting the same login so that a configuration tool could sweep across the network and set them all up at once. Any replacement has to serve that workflow, or installers will find ways around it. Several patterns serve it well. The most common is to ship devices that require a credential to be set before first use, and to provide a commissioning tool that discovers new, unconfigured units on the local network and sets a strong, unique credential on each, recording them in the customer's management system. The unconfigured state should be limited in time and restricted to the local network, so that a device left uncommissioned does not wait indefinitely for anyone to claim it. A second pattern uses the per-device identity keys described below: the customer's management system enrols devices by scanning their certificates or QR codes, often in bulk from the shipping manifest, so that no human-chosen password exists at all. A third, increasingly common in enterprise products, is zero-touch provisioning, in which each device, on first connection, contacts the maker's cloud service, proves its identity with its factory-provisioned key, and is directed to the customer's own management server according to an ownership record created at the time of purchase. Each of these preserves the installer's ability to configure many devices quickly, and none of them relies on a secret that is the same across units. The design principle is that the convenience of bulk configuration should come from tooling, not from shared credentials. Designing the owner's side A related point concerns how products present credentials to owners. Many devices that technically allow a password to be changed bury the option deep in a settings menu, accept trivially weak replacements, and give no indication of why it matters. If the product's security depends on the owner taking an action, the product should make that action unavoidable at the moment it is needed and easy to get right, by requiring it during setup, enforcing a reasonable minimum strength, checking against lists of commonly used and previously breached passwords, and explaining in a sentence why the step exists. The better approach, wherever the product allows it, is not to depend on the owner at all. The factory side of the problem It is worth returning to the reason universal default passwords persisted, because the replacements described above all depend on doing something per unit in the factory: printing a unique label, programming a unique setup code, generating a unique key. These operations are where the security design meets the production line, and where good intentions most often fail. A practical factory flow for unique credentials has a few essential properties. Secrets are generated either on the device itself during provisioning, or by a hardened provisioning server, never by a script on a technician's workstation. Each unit's identity and credential are recorded in a manufacturing database only to the extent the business genuinely needs them; for a device that generates its own key pair, only the public key and certificate need ever leave the device. Labels are printed from the same record that was programmed into the device, at the same station, and verified by scanning before packaging. Units that fail provisioning are quarantined rather than reworked by hand. And the whole process is audited, because a contract manufacturer with access to every unit's secrets is a single point of compromise for the entire fleet. None of this is exotic. Payment card manufacturers, SIM card vendors and smartphone makers have run such processes at enormous scale for years. The barrier for consumer IoT has been cost and habit rather than technology. Law has now removed the option of avoiding the cost, and the habit is changing accordingly. What the end of the default password does not solve It is important not to overstate what eliminating default credentials achieves. It closes the specific door Mirai walked through. It does nothing about vulnerabilities in exposed services that can be exploited without logging in at all, which is how many of Mirai's successors spread. It does nothing to stop firmware from being modified once an attacker has found another way in. And it does nothing to ensure the device can be fixed when a vulnerability is found. Those problems require foundations deeper than a login prompt. The device must be able to tell whether the code it runs is its maker's, and to refuse anything else. That capability starts in hardware, with a root of trust, which is the subject of the next chapter. Chapter 4: The Hardware Root of Trust Every security property a device claims ultimately rests on something it cannot verify. Software can check other software, but the checking code must itself be trusted, and so must whatever checked it, and so on down. The chain has to end somewhere. A root of trust is the place where it ends: a component whose correct behaviour is assumed rather than verified, because it is built to be immutable, isolated or both. Everything above it inherits its trustworthiness, and inherits any flaw it has. The phrase "hardware root of trust" is used loosely in marketing, so it is worth being precise. The root of trust provides a small number of functions on which the rest of the system depends. It holds secrets, such as a device's unique private key, in a form that cannot be read out by software running on the main processor. It performs cryptographic operations with those secrets on request. It verifies the first piece of mutable code that runs at boot, or measures it so that its identity can be reported. And it stores a small amount of state that cannot be rolled back, such as a counter of the minimum acceptable firmware version. A component that does these things reliably, in a way that software compromise cannot undo, is a root of trust. A component that merely accelerates cryptography is not. Why it must be hardware It might seem that a carefully written bootloader could serve as the root of trust. The problem is that anything stored in rewritable memory can be rewritten. If the code that verifies firmware signatures is itself stored in flash, an attacker who gains the ability to write flash, through a software vulnerability, a misused update mechanism or a programmer clipped onto the chip, can replace the verifier with one that approves everything. Secure boot implemented entirely in mutable storage is a lock whose key is hanging on a hook beside the door. The same logic applies to secrets. A private key stored in ordinary flash, however obscurely, can be read by anyone who can run code on the device or read its memory chip. If the key is the device's identity, extracting it allows the attacker to impersonate the device to its cloud service indefinitely, even after the device itself has been repaired or replaced. If the key is used to decrypt firmware, extracting it from one unit decrypts every unit's firmware. A hardware root of trust breaks this circularity by anchoring trust in something that software cannot modify: code in mask ROM, written into the silicon at fabrication; values in one-time-programmable fuses, which can be set once and never cleared; and keys held in a dedicated security processor or protected memory that the main processor cannot read. An attacker who controls every byte of flash still cannot change the ROM, unblow a fuse or read a key out of a secure element through its software interface. The building blocks There is no single hardware root of trust design. Device makers assemble one from a small set of building blocks, chosen according to cost, threat model and the capabilities of the chosen processor. Boot ROM and fuses Almost every modern microcontroller and application processor intended for connected products includes immutable boot ROM and a bank of one-time-programmable fuses. The ROM contains the first code the processor executes after reset. When secure boot is enabled, this code reads a hash of the manufacturer's public key, or the key itself, from the fuses, loads the next-stage bootloader from flash, and verifies its signature before running it. Further fuses typically control whether secure boot is enforced, whether debug interfaces are enabled, and the minimum allowed version of the bootloader. This combination is the minimum viable root of trust and is available at very low marginal cost. Its weaknesses are that the ROM can contain bugs that can never be fixed, a point the next chapter returns to, and that fuses provide only very limited protected storage: they can hold a key hash and a few configuration bits, but cannot perform cryptographic operations or store a device private key in a way that resists extraction by software running on the processor. Secure elements A secure element is a separate, small, tamper-resistant chip, of the same lineage as the chips in payment cards and SIM cards, attached to the main processor over a simple bus. It generates and stores private keys internally and performs signing and key agreement operations without the keys ever leaving the chip. It is typically designed and certified to resist physical attack: probing, fault injection, power and electromagnetic analysis. A device that keeps its identity key in a secure element can be fully compromised at the software level without that key being extracted. The attacker can use the key while they control the device, but cannot copy it to use elsewhere, and loses access when the device is repaired. Secure elements cost a little more and add a component to the bill of materials, but they are now inexpensive enough to be standard in many consumer products, and several chip vendors offer them pre-provisioned with keys and certificates so that the device maker does not need to handle secrets in its own factory. Trusted platform modules A trusted platform module is a standardised security chip, specified by the Trusted Computing Group, with a defined command interface for key storage, signing, measured boot and attestation. The current version, TPM 2.0, is ubiquitous in PCs and increasingly used in industrial gateways, networking equipment and other embedded Linux devices where its standardised software support is valuable. TPMs are less common in small consumer devices, where their cost and power consumption are harder to justify and a simpler secure element may suffice. Trusted execution environments and integrated security subsystems Many processors provide hardware isolation between a secure world and a normal world on the same chip. Arm's TrustZone, available on both application processors and on the Armv8-M microcontroller architecture, is the most widespread example. Code in the secure world can hold keys and perform sensitive operations, and the hardware prevents the normal world, where the main operating system and applications run, from reading secure-world memory. On microcontrollers, the open-source Trusted Firmware-M project provides a reference implementation of secure-world services for this purpose. Increasingly, chip makers also integrate a dedicated security subsystem, sometimes called a secure enclave or hardware security module, directly on the main die: a separate small processor with its own memory, cryptographic accelerators and key storage, isolated from the application cores. These integrated designs can approach the protection of a separate secure element without the extra component. Their resistance to physical attack varies widely, and a device maker should look for independent certification rather than relying on data-sheet language. Open-source roots of trust A notable development of recent years is the emergence of open-source hardware roots of trust whose designs can be inspected. OpenTitan, a project hosted by the non-profit lowRISC with participation from Google and others, publishes a complete silicon root-of-trust design and has reached production silicon. Caliptra, launched in 2022 by a group including AMD, Google, Microsoft and NVIDIA, defines an open-source root of trust intended to be integrated into data-centre processors and other system-on-chip designs. These projects matter for IoT less because device makers will adopt them directly in the near term than because they establish an expectation that the most trusted component in a system should be the most open to scrutiny. Table 2 compares these options on the dimensions that typically drive the choice. The judgements are general; specific products vary. Table 2. Common hardware root-of-trust options compared. Option Protects device keys from software compromise Physical attack resistance Relative cost Typical use Boot ROM with fuses only No Low Lowest Basic secure boot on low-cost devices On-chip TEE (e.g. TrustZone) Partly, if secure world is sound Low to moderate Low Isolating keys and services on one chip Integrated security subsystem Yes Moderate, varies by vendor Low to moderate Modern connected MCUs and SoCs Discrete secure element Yes High, often certified Moderate Device identity, consumer and industrial Discrete TPM 2.0 Yes Moderate to high Moderate to higher Gateways, Linux-class devices Identity: one device, one key The most important job of a hardware root of trust in a connected product is usually not secure boot but identity. Every device that talks to a cloud service, joins a network or receives updates needs a way to prove which device it is. The strong form of that proof is a unique asymmetric key pair per device, with the private key protected by the root of trust and the public key certified by the manufacturer or by a certificate authority it trusts. The certificate turns the key into a verifiable claim. A device attestation certificate, of the kind used in Matter, says in effect: this public key belongs to a genuine device of this model, made by this manufacturer. A cloud service presented with such a certificate, and a signature proving possession of the private key, can be confident it is talking to a real device and not to a script running on an attacker's laptop. It can also maintain a per-device record and revoke any individual device's access if that device is reported stolen or shows signs of compromise. A refinement of this model is the Device Identifier Composition Engine, or DICE, a specification from the Trusted Computing Group designed for devices too small or inexpensive for a TPM. In DICE, the hardware holds a unique device secret that is accessible only at the very start of boot. Before handing control to the first mutable code, the hardware combines that secret with a cryptographic hash of the code to derive a new value, the compound device identifier, and then locks the original secret away until the next reset. Each subsequent boot stage can repeat the process, deriving keys for the next stage from its own secret and a measurement of the code it is about to run. The effect is elegant. The keys a device can use depend on the exact firmware it is running. If an attacker modifies the firmware, the derived keys change, and the device can no longer produce signatures that the cloud service recognises as belonging to the genuine software. The device's identity and its software integrity become the same thing, and a compromised device cannot lie about what it is running without losing the ability to prove who it is. DICE requires very little hardware, which makes it attractive for constrained devices, and it is increasingly supported by microcontroller vendors. Provisioning in the factory A root of trust is only as good as the process that puts secrets into it. Provisioning, the step in manufacturing where each device receives its keys, certificates and configuration, is where hardware security design meets the reality of a production line, and it deserves as much engineering attention as the silicon. The preferred model is for each device to generate its own key pair internally, inside the secure element or security subsystem, so that the private key never exists anywhere else. The device exports only its public key, which a provisioning system signs into a certificate and writes back. A compromise of the factory then yields, at worst, the ability to issue certificates for fake devices, which is serious but bounded, rather than a copy of every genuine device's private key. Where keys must be injected rather than generated, for example because a symmetric key is needed or the hardware cannot generate keys itself, they should be generated in a hardware security module and delivered to the device over an encrypted channel, so that they are never visible in plaintext to the provisioning station or the people operating it. Several secure element vendors offer pre-provisioning, in which keys are generated and certificates installed in the chip vendor's own secure facility before the parts are shipped to the device maker. This shifts trust to the chip vendor but removes secrets from the device maker's contract manufacturer entirely, which is often a good trade. After provisioning, the device must be locked. Secure boot is enforced by blowing the appropriate fuse, so that the ROM will refuse to run unsigned code. Debug interfaces are disabled or placed in an authenticated mode where they can be reopened only with a cryptographic challenge from the manufacturer. Any provisioning-only commands are removed or permanently disabled. A device that leaves the factory with secure boot configured but not enforced, or with JTAG enabled, has a root of trust in name only. This is one of the most common findings when security researchers examine production devices, and it typically results from a decision to defer lockdown until a later stage of manufacturing that never quite happens, or from concern that locked devices cannot be reworked if a defect is found. The right way to address that concern is to design the rework path deliberately, typically as an authenticated debug unlock that requires a per-device challenge signed by a key held only by the manufacturer, rather than leaving every device permanently open in case one needs repair. Choosing for the threat model The comparison above can make the choice look like a matter of buying the most protection one can afford. In practice the right root of trust depends on what an attacker would gain from a successful physical attack, and the threat modelling described in Chapter 2 should drive the decision. For a low-cost smart plug, the assets on the device are modest: its identity with the cloud service and its Wi-Fi credentials. If the design ensures that each plug's identity key is unique, an attacker who spends a day extracting one key gains control of one plug, which they already owned. Boot ROM with fuses for secure boot, combined with an integrated security subsystem or a TrustZone-isolated key store, is usually proportionate. For a doorbell camera, the calculation changes. The device is mounted outside, where anyone can remove it; it stores the household's Wi-Fi password; and it may hold keys that protect recorded video. A discrete or certified integrated secure element, so that the Wi-Fi credentials and identity key cannot be read out of a stolen unit, is a reasonable investment, as is a design in which removing the device from its mount triggers a notification and the owner can revoke it from the app. For a smart meter or an industrial controller, the device may hold keys that authenticate it to infrastructure on which many others depend, and it may sit for years in locations where an attacker can work on it undisturbed. Here certified tamper resistance, attestation and careful separation of keys across functions are justified, and the cost of a stronger root of trust is small relative to the cost of the system it protects. The general rule is that the root of trust should make the attack on one device cost more than that device is worth to an attacker, and should ensure that the result of a successful attack does not transfer to any other device. Certification and assurance Device makers rarely have the resources to evaluate the physical security of the components they buy, and data sheets are not a reliable guide. Independent certification fills that gap. Secure elements are commonly evaluated under Common Criteria, the international scheme for security evaluation, at assurance levels that include penetration testing by accredited laboratories. For the broader IoT market, Arm and a group of partners launched PSA Certified in 2019, a scheme with three levels covering a chip's root of trust and its firmware, with the higher levels including laboratory testing of resistance to software and then physical attacks. The SESIP methodology, developed by GlobalPlatform and published as European standard EN 17927, offers a lighter-weight evaluation approach intended for IoT platforms. Certification does not make a product secure, and a certified component can be integrated insecurely. But it provides evidence, from someone other than the vendor, that the component does what it claims under attack. When choosing a root of trust, asking which certification it holds and at what level is a reasonable first filter. The root is not the house A hardware root of trust is the foundation of device security, but it is only a foundation. It guarantees that a small number of things are true: that a secret cannot be extracted, that a key hash cannot be changed, that the first code checked was the code expected. Everything else depends on how those guarantees are used. The next chapter follows the chain upward, from the ROM through each stage of boot to the running application, and examines how that chain is built, how it fails, and how a well-designed device recovers when it does. Chapter 5: Secure Boot and the Chain of Trust When a connected device powers on, it runs a sequence of programs, each loading the next: the boot ROM, one or more bootloaders, an operating system kernel or real-time executive, and finally the application. Secure boot is the practice of making each stage verify the next before handing over control, so that the device runs only code its maker has signed. The idea is old and simple. Its implementation is where many devices fail, and the failures are instructive because they show how much of security depends on details that look, at first glance, like housekeeping. Verified boot, step by step In a verified boot chain, every stage carries the public key, or a hash of the public key, needed to check the stage after it. The boot ROM reads a key hash from fuses, loads the first-stage bootloader from flash, checks that the bootloader's embedded public key matches the fused hash, and verifies the bootloader's signature with that key. If the signature is valid, it jumps to the bootloader. The bootloader in turn verifies the kernel or application image against keys it trusts, and so on. If any check fails, the stage refuses to proceed and takes some defined recovery action. The essential property is that trust never flows upward. Each stage is trusted only because the stage below it verified it. The bottom of the chain is the immutable ROM and the fused key, which is why Chapter 4 placed so much weight on the root of trust. Above the ROM, nothing is trusted merely because it is present in flash. On embedded Linux devices, the chain usually extends beyond the kernel into the root filesystem. The kernel itself cannot practically verify a signature over an entire filesystem at every boot, so the common approach, used by Android and widely adopted elsewhere, is dm-verity: the filesystem is stored read-only, a tree of hashes over its blocks is computed at build time, and the root of that hash tree is signed or embedded in a signed image. As each block is read during operation, the kernel checks it against the tree, so that any modification is detected on access. This extends verification to every executable and configuration file on the system partition without the cost of checking everything at boot. A short sketch of the decision logic at each stage makes the order of operations explicit. It is not tied to any particular platform. image = load_candidate_image(slot) if not signature_valid(image, trusted_key): reject(slot); try_other_slot_or_recover() if image.security_version < rollback_counter(): reject(slot); try_other_slot_or_recover() copy_to_protected_memory(image) # verify what you run, run what you verified lock_down_before_jump() # disable debug, drop privileges, lock keys jump(image.entry) Each line encodes a lesson learned from a real failure, and the rest of this chapter explains them. How secure boot fails Secure boot is conceptually straightforward, which tempts teams to treat it as a feature to be ticked off. In practice there is a long history of verified boot chains that verified nothing, or verified the wrong thing, or could be bypassed by someone with a cheap piece of hardware. Enabled but not enforced The most common failure is not a subtle bug but a configuration that was never finished. Many chips support secure boot, and many device makers sign their firmware, but the fuse that makes the ROM enforce verification is never blown, or debug interfaces are left open so that an attacker can simply halt the processor and write whatever they like into memory. Security researchers examining consumer devices regularly find firmware that is signed, with signatures that are never checked. The fix is procedural: lockdown must be a required, verified step in the manufacturing process, tested on units pulled from the production line, not an option left to the factory's discretion. Bugs in the verifier The code that verifies signatures is the most security-critical code on the device and has historically been some of the buggiest. Parsers for image headers, certificate formats and signature encodings have been found to accept malformed inputs that cause them to skip verification, verify the wrong region of memory, or overflow buffers. Each such bug is serious in a bootloader and catastrophic in boot ROM, because ROM cannot be patched. Two widely reported cases illustrate the stakes. In 2018, researchers disclosed a vulnerability, dubbed Fusée Gelée, in the boot ROM of NVIDIA's Tegra X1 processor, used in the original Nintendo Switch, which allowed arbitrary code execution through a USB recovery mode before any signature check. In 2019, a researcher published checkm8, a boot ROM vulnerability affecting Apple devices with processors from the A5 through the A11, also exploitable over USB. In both cases the affected silicon could never be fixed; the manufacturers could only change the ROM in new chip revisions. The lesson is that ROM should be as small as possible, should do as little parsing as possible, and should be subjected to the most rigorous review and testing a vendor can afford, including formal verification where practical. Every feature added to boot ROM, such as a recovery protocol over USB, is a permanent addition to the attack surface. Verifying one thing, running another A subtle class of failure arises when the verifier checks an image in one place and the processor executes it from another, or when the image can be changed between check and use. This is known as a time-of-check to time-of-use problem. If a bootloader verifies a kernel image while it sits in external flash, then reads it again from flash to execute it, an attacker with a device that can intercept or alter the flash bus can present the genuine image during verification and a modified one during loading. The defence is to copy the image into internal memory that the attacker cannot modify, verify it there, and execute exactly the bytes that were verified. That is the meaning of the comment in the sketch above. Similar problems arise when only part of an image is signed. If a bootloader verifies the kernel but not the device tree, the command line or the initial filesystem the kernel loads, an attacker may be able to achieve the same effect as modifying the kernel by modifying the unverified parts. Everything that influences what code runs or how it behaves must be within the signed envelope. Fault injection Physical attackers have a further option: inducing a fault in the processor at the exact moment it evaluates a security decision. Brief disturbances in supply voltage, clock signal or electromagnetic environment can cause a processor to skip an instruction or compute a wrong result. If the skipped instruction is the branch that rejects an invalid signature, the device boots the attacker's code. Fault injection was once the preserve of well-equipped laboratories; inexpensive open-source tools have made it accessible to hobbyists, and it has been used to bypass protection on a range of popular microcontrollers. Defending against fault injection is partly a matter of choosing hardware with built-in countermeasures, such as voltage and clock monitors, and partly a matter of coding the verification logic defensively. Common techniques include checking critical conditions more than once, using values for success and failure that differ in many bits rather than a single zero or one, adding random delays so that an attacker cannot easily time a glitch, and verifying that control flow reached the expected points. These measures raise the cost of an attack rather than eliminating it, which is why the principle from Chapter 2 matters: a successful fault attack on one unit should not yield a secret that compromises others. Rollback protection Signature verification answers the question of whether an image was produced by the manufacturer. It does not answer whether it is the right image. Every old firmware version, including those with known vulnerabilities, carries a valid signature. An attacker who can install a signed but outdated image can reintroduce a vulnerability the manufacturer has already fixed, and then exploit it. This is a downgrade or rollback attack, and a verified boot chain without rollback protection is incomplete. The defence is a security version number, embedded in each signed image and compared at boot against a monotonic counter stored somewhere the attacker cannot reset. Fuses are the strongest option: each increment blows another fuse, and the ROM or bootloader refuses any image whose security version is below the number of blown fuses. Fuses are a limited resource, typically allowing a few dozen increments over a device's life, so the security version should be incremented only when an update fixes a vulnerability serious enough to make returning to the old version dangerous, not with every release. Other options include replay-protected memory blocks, a feature of eMMC and UFS storage that provides authenticated writes to a small area, and counters maintained by a secure element or TPM. Rollback protection interacts with update reliability, discussed in Chapter 6. A device should not advance its rollback counter until a new image has booted successfully and confirmed that it works, because once the counter advances, the previous image can no longer be used as a fallback. Getting this sequence wrong can turn a failed update into a device that will not boot at all. Measured boot and attestation Verified boot enforces a policy at the device: only signed code runs. Measured boot does something complementary. At each stage, the device computes a cryptographic hash of the code it is about to run and records it in a way that cannot be undone or forged, for example by extending a register in a TPM, or by deriving keys from it as DICE does. The device may run code that has not been signed; the point is that it cannot hide what it ran. The value of measurement appears when combined with remote attestation. A remote party, such as the manufacturer's cloud service or an enterprise's network access controller, sends the device a fresh random challenge. The device returns a signed report of its boot measurements, signed by a key that only the root of trust can use. The remote party compares the measurements with known-good values and decides whether to trust the device: whether to allow it onto the network, send it secrets, or accept its data. Attestation allows policy decisions to move from the device to a party with more context. A cloud service can refuse to issue credentials to devices running an outdated or unknown firmware version, quarantine devices whose measurements suddenly change, or require an update before granting access. For fleets of industrial or enterprise devices, this is a powerful tool. For consumer devices, it is less widely used but increasingly practical, particularly with DICE-based designs where measurement is built into the key derivation and comes at almost no extra cost. Encrypting firmware Signing proves where firmware came from. Encryption hides what it contains. The two are often confused, and it is worth being clear about what encryption can and cannot achieve. Encrypting firmware images, whether in update packages or in the device's own storage, makes it harder for an attacker to obtain and study the code, which raises the cost of finding vulnerabilities and of building attacks against them. It also protects any proprietary algorithms the firmware contains. These are legitimate goals. But encryption is not a substitute for signing, because an attacker who can modify ciphertext may be able to cause controlled changes in the decrypted image unless the encryption is authenticated. And the protection encryption offers depends entirely on the decryption key. If every device holds the same key, as is common, extracting it from one unit decrypts the firmware for all, and the encryption becomes a modest speed bump. The more robust view is to treat firmware secrecy as a defence in depth, never as a primary control. A device should be secure even if its firmware is public. Several manufacturers of security-critical products publish firmware openly, on the reasoning that scrutiny finds bugs faster than obscurity hides them. Where encryption is used, it should be authenticated, the key should be protected by the root of trust, and the design should assume that the key will eventually be extracted. Locking down at the hand-off The final line of the sketch earlier in this chapter, lock down before jumping, captures a principle that is easy to neglect. Each stage of boot typically has privileges that later stages should not: access to a key used to decrypt the next image, the ability to write rollback counters, debug access, or the device's unique secret in a DICE design. Before handing control to the next stage, a careful bootloader revokes these privileges, by locking key slots, disabling hardware peripherals, and configuring memory protection so that later code cannot reach what it does not need. A vulnerability in the application then cannot be used to reach back into the boot process and steal what the bootloader used. A worked example: one camera's boot It helps to see these principles assembled in one device. Consider a mid-range home security camera built on an application processor with a boot ROM, fuse bank, an integrated security subsystem and external flash, running embedded Linux. At the factory, after provisioning, three things are fused: the hash of the manufacturer's root public key, a bit that makes the ROM enforce signature verification, and a bit that places the debug port in authenticated mode. The rollback counter starts at zero. When the camera powers on, the ROM reads the first-stage bootloader from flash into internal SRAM, checks that the key embedded in its header matches the fused hash, verifies the signature over the whole image in SRAM, compares the image's security version with the fused rollback counter, and only then jumps to it. The ROM does not parse a filesystem, does not offer a network recovery mode and does not support any image format other than one fixed header, because each such feature would be permanent attack surface. The first-stage bootloader initialises external memory, then loads the second-stage bootloader and verifies it against a key certified by the root. The second stage reads a small, signed boot control record to determine which of two firmware slots to try, loads that slot's kernel, device tree and initial filesystem as a single signed bundle, verifies it in RAM and checks its security version. Before handing over, it asks the security subsystem to derive the key that unlocks the encrypted storage partition, passes that key to the kernel, and then locks the subsystem's key derivation interface so that nothing running later can request the bootloader-level keys. It records a measurement of the bundle that the device can later report in an attestation. The kernel mounts the root filesystem through dm-verity, with the root hash taken from the signed bundle, so every block of system code is checked as it is read. The camera application runs as an unprivileged user in a restricted sandbox; the update agent runs as a separate process with permission to write only to the inactive firmware slot. Now consider an attacker who finds a command injection flaw in the camera's web interface and gains code execution as the application user. They cannot write to the system partition, which is read-only and verified. If they escalate privileges and write to the inactive slot, the next boot will reject the unsigned image and return to the active one. If they write to the active slot, the bootloader will reject that too and fall back to the other slot. They cannot extract the device's identity key, which never leaves the security subsystem, or the storage key, which was locked away before the kernel started. What they have achieved, at worst, is a foothold that lasts until reboot, and the manufacturer can close the flaw with a signed update. That outcome is not perfect security, but it is the difference between a vulnerability and a catastrophe. Recovery: resilience as well as protection NIST Special Publication 800-193, published in 2018, frames platform firmware security as three linked goals: protection, detection and recovery. Most of this chapter has been about protection and detection, preventing unauthorised code from running and noticing when something is wrong. Recovery is the part most often neglected and most important in practice. A device that detects corrupted or unauthorised firmware and simply refuses to boot has protected itself, but at the cost of becoming useless. For a consumer camera that may be an acceptable outcome. For a router serving a household, a medical device or an industrial controller, it is not. A resilient design keeps a known-good recovery image, protected from modification, from which the device can restore itself. This is often implemented with two copies of the main firmware, as Chapter 6 describes, together with a minimal recovery partition that can fetch a fresh image over the network or accept one from a local interface. The recovery path must itself be verified, since an attacker who can trigger recovery and supply the recovery image has bypassed everything else. Secure boot, then, is not a single check but a chain of decisions, each of which can be gotten right or wrong. When it is right, a device can be compromised at runtime and still return to a trustworthy state on reboot, which transforms the consequences of most software vulnerabilities. But it only returns to the code it already has. If that code contains a vulnerability, the device will faithfully boot it every time. The chain of trust needs a way to carry fixes, which is the job of the update system. Chapter 6: Designing the Over-the-Air Update An update system is the only security control that improves with age. Everything else on a device is fixed at manufacture or, at best, configured during setup. The update channel is how a maker responds to vulnerabilities that no one had found when the product shipped, and over a device life of five or ten years, there will be many. Without it, every other measure in this booklet is a depreciating asset. It is also the most dangerous feature a device has. An update mechanism is, by design, a way for a remote party to replace all of the code on the device. If anyone other than the maker can use it, it is the perfect backdoor. If it fails partway through, it can destroy the device. Designing an over-the-air update system is therefore a problem of reliability and security at once, and the two concerns are more closely entwined than they first appear. An update system that frequently fails will be used reluctantly, delayed, or disabled by owners, and a device that is not updated is not secure. This chapter treats the device side: how an update is received, verified, installed and confirmed without risking the device. Chapter 7 turns to the pipeline that produces and distributes updates, and to the frameworks designed to keep that pipeline trustworthy when parts of it are compromised. What an update must guarantee A well-designed update process guarantees five things. It installs only images the maker has authorised for this device. It never installs an image older than the device's security floor, as described in Chapter 5. It leaves the device bootable at every moment, including if power fails or the network drops at any point during the process. It confirms that the new image actually works before committing to it, and reverts automatically if it does not. And it completes without requiring the owner to do anything difficult, because owners do not do difficult things. Each of these guarantees has a familiar failure mode. Updates that were accepted without proper verification have been used to install malware. Updates that could be rolled back have been used to reintroduce fixed vulnerabilities. Updates that wrote directly over the only copy of the firmware have bricked devices when interrupted. Updates that booted but broke network connectivity have left devices unable to receive the fix. In August 2017, a smart lock vendor pushed a firmware update intended for one model to another, leaving several hundred locks, many installed on holiday rental properties, unable to operate, and requiring the units to be removed and sent back for repair. Updates that required owners to download a file and upload it through a web page were simply never applied. Partition schemes The core reliability decision is how the device stores the new image while it is being installed. There are a few standard patterns, and Table 3 compares them on the properties that matter most. Table 3. Common firmware update storage schemes. Scheme Survives interruption Automatic fallback Storage cost Typical use Single image, in-place overwrite No No Lowest Legacy and very constrained designs Single image plus recovery partition Yes, via recovery Limited Low to moderate Cost-sensitive Linux devices Dual image (A/B) Yes Yes Roughly double Routers, cameras, gateways, phones Staging area with swap (MCU) Yes, if swap is resumable Yes Extra slot plus scratch Microcontrollers using MCUboot The weakest scheme overwrites the running firmware in place. If anything interrupts the write, the device is left with a partial image that will not boot, and because the verified boot chain will reject it, there is nothing to fall back to. This pattern should not appear in any new product. The dual-image or A/B scheme keeps two complete copies of the firmware in separate partitions. The device runs from one, say A, and writes the update into B while continuing to operate normally. When the write is complete and verified, the bootloader is told to try B on the next boot. If B boots and passes its health checks, it becomes the active image; if not, the bootloader returns to A. At no point is there not a complete, working image on the device. Android adopted this model, under the name seamless updates, from version 7.0, and it is now standard on Linux-class embedded devices with sufficient storage. Its cost is roughly double the firmware storage, which on a device with a few hundred megabytes of flash is usually acceptable. On microcontrollers, where flash is measured in hundreds of kilobytes and doubling it is expensive, a common alternative is to download the new image into a secondary slot and then have the bootloader swap it into the primary slot at boot. The open-source MCUboot bootloader, widely used with Zephyr and other embedded operating systems, implements this with careful bookkeeping so that a swap interrupted by a power cut can be resumed from where it stopped, and so that the old image can be swapped back if the new one fails to confirm. This gives most of the reliability of A/B at lower storage cost, at the price of a longer first boot after update and additional wear on the flash. A third option for cost-sensitive Linux devices keeps one main image and a small, rarely changed recovery partition. The device boots into recovery to install updates and can always return to recovery if the main image fails. It saves storage, but the device is out of service during installation, and the recovery partition becomes critical code that must itself be kept secure and occasionally updated. Verification before, during and after An update image should be verified at least twice: when it arrives, and when it boots. Verification on arrival, by the update agent running on the device, prevents the device from wasting time and flash writes on an image that should never be installed, and allows it to reject images intended for another model or hardware revision. Verification at boot, by the secure boot chain described in Chapter 5, is the check that actually matters, because it is performed by code anchored in the root of trust, and because it catches any modification to the image between installation and execution. Arrival checks should cover more than the signature. The update should be bound to a specific device type and hardware revision, so that an image built for one product cannot be installed on another, which is exactly the error the smart-lock vendor made. It should carry a version number and a security version, so that the device can refuse downgrades. It should list dependencies, such as a minimum bootloader version or a required intermediate step, so that the device does not install an image it cannot run. And its metadata should be signed together with the image, so that an attacker cannot change the target device or version while keeping the image's signature valid. The IETF has standardised these concerns for constrained devices in its work on Software Updates for Internet of Things, known as SUIT. RFC 9019, published in 2021, describes the architecture, and RFC 9124, published in 2022, sets out the information that an update manifest must carry: identifiers for the vendor and device class, sequence numbers to prevent rollback, digests of each payload, conditions that must hold before installation, and instructions for how to install. The manifest is a small signed document that travels with the image and tells the device what it is, whom it is for, and what to do with it. Even teams that do not adopt the SUIT format directly will find its information model a useful checklist. Confirming and committing Booting the new image is not the same as succeeding. A firmware build might boot cleanly and then fail to bring up Wi-Fi, crash when the camera sensor initialises, or be unable to reach the update server. A device in that state is worse than one that never updated, because it may be unable to receive the fix for whatever broke it. The standard defence is a trial boot with explicit confirmation. The bootloader marks the new image as provisional and boots it. The new firmware runs a set of health checks: it brings up its critical functions, establishes contact with the cloud service or update server, and confirms that it can receive a further update. Only when those checks pass does it mark itself as confirmed. If the device reboots before confirmation, whether because of a crash, a watchdog timeout or a power cut, the bootloader treats the new image as failed and returns to the previous one. Health checks should be chosen to reflect what would make a device unrecoverable. The most important check is the ability to update again. Confirmation is also the moment to advance the rollback counter. As Chapter 5 explained, once the counter moves past the old image's security version, the old image can no longer boot. Advancing it before the new image is confirmed risks a device with no bootable firmware at all. Advancing it after confirmation, and only for releases that fix serious vulnerabilities, keeps both safety and the downgrade defence intact. Bandwidth, energy and timing Many connected devices operate under constraints that make full-image downloads expensive. Battery-powered sensors on low-power wide-area networks may have bandwidth measured in kilobits per second and a strict energy budget. Vehicles and industrial equipment may have connectivity that costs money per megabyte. Delta updates, which transmit only the differences between the old and new image, can reduce transfer sizes dramatically, but they add complexity: the device must reconstruct the new image from the delta and the old image, and the result must be verified as a complete image against the maker's signature, not merely trusted because the delta was signed. Timing matters too. A thermostat should not reboot to install an update in the middle of a cold night; a vehicle should not update while being driven; a medical device should not update during therapy. The update agent needs a policy for when installation is permissible, and the policy should be enforced by the device rather than relying on the server to know the device's state. At the same time, a policy that allows owners to postpone updates indefinitely will, in practice, mean that many devices never update. The right compromise for most consumer products is automatic installation by default during idle periods, with owners able to schedule but not permanently refuse security updates. A worked example: the battery-powered sensor The constraints just described are clearest in the smallest devices. Consider a wireless soil-moisture sensor deployed across farmland: a microcontroller with a megabyte of flash, a low-power radio link with tight duty-cycle limits, a battery expected to last five years, and no one nearby to press a reset button. A full image download of several hundred kilobytes might take hours of radio time and a measurable fraction of the battery's life, so the maker ships delta updates computed against the exact version each device runs. The device downloads the delta in small chunks over days, during its normal wake windows, storing each chunk in a staging region of flash and checking its hash against a signed manifest received first. The manifest, in the spirit of SUIT, names the device class, the base version the delta applies to, the resulting version and its security version, and the digest of the complete reconstructed image. When all chunks have arrived, the device checks that its battery is above a threshold sufficient to complete the installation with a margin, because running out of power during a swap is the scenario most likely to leave it stranded. It reconstructs the full image into the secondary slot, computes its digest and compares it with the manifest. Only if they match does it ask the bootloader to perform a swap. MCUboot exchanges the slots in resumable steps, so that if the battery fails in the middle, the swap continues at next power-up. The new firmware boots provisionally, rejoins the network, reports its version to the management system, confirms receipt of a small test message from the server, and only then marks itself as confirmed. Every one of these steps protects against a failure that has happened somewhere in the field. Together, they turn an operation that could destroy a remote device into one that can be performed thousands of times across a fleet with no one visiting a single field. Fleet rollout The device is only half of the update system. The other half is how the maker decides which devices receive an update and when. Releasing a new firmware image to an entire fleet at once means that any defect not caught in testing reaches every device simultaneously. The consequences of that kind of simultaneous failure were demonstrated well outside IoT in July 2024, when a faulty content update to CrowdStrike's endpoint security software crashed millions of Windows machines worldwide within hours. Staged rollout is the standard defence. A new release goes first to internal test devices, then to a small percentage of the fleet, then to progressively larger fractions, with telemetry monitored at each stage for increased crash rates, failed confirmations, lost connectivity or other signs of trouble. Rollout pauses automatically if those signals cross a threshold. The fleet management system should be able to target releases by hardware revision, region and current firmware version, and to halt or reverse a release quickly. Open-source tools such as Eclipse hawkBit for rollout management, and Mender, RAUC and SWUpdate for device-side update handling on embedded Linux, implement many of these capabilities, and commercial platforms offer more. There is a tension here with security. When an update fixes a vulnerability that is being actively exploited, a slow rollout leaves devices exposed for longer. The rollout policy should therefore distinguish between routine releases and urgent security fixes, with a faster, still staged, path for the latter. The time from vulnerability disclosure to fleet-wide deployment is one of the most meaningful security metrics a device maker can track. Regulated industries as a guide Some sectors have already turned update management into a formal requirement, and their approaches are a useful guide for others. Since 2022, new vehicle types sold in the European Union and other countries that apply the relevant UNECE regulations have had to comply with UN Regulation No. 156, which requires manufacturers to operate a certified software update management system. The regulation requires manufacturers to record the software versions installed on each vehicle type, assess the impact of each update on safety and type approval, protect the update process against compromise, ensure that updates can be performed safely, and inform owners about them. These requirements were motivated in part by demonstrations such as the 2015 remote compromise of a Jeep Cherokee by researchers Charlie Miller and Chris Valasek, which led Fiat Chrysler to recall about 1.4 million vehicles for a software fix delivered by USB stick, because the vehicles had no secure over-the-air channel. The point for device makers in other sectors is not to copy automotive bureaucracy but to notice what a regulator considered essential: knowing what software each device runs, controlling what changes, and proving the process is secure. When devices are bricked deliberately A final consideration links the update system back to the attackers of Chapter 1. In October 2023, researchers at Lumen's Black Lotus Labs reported that some 600,000 routers belonging to a single American internet service provider had been rendered permanently inoperable over a 72-hour period by malware that overwrote their firmware, an incident they called Pumpkin Eclipse. The affected units had to be physically replaced. Whatever the attackers' intent, the outcome shows what happens when a device has no protected recovery path: malicious code with write access to firmware can destroy it. A device with a hardware root of trust, verified boot, an A/B or recovery scheme and protected rollback counters is much more resistant to this kind of attack. Even if malicious code overwrites one firmware slot, the verified boot chain will reject the modified image and fall back to the other, or to a protected recovery image. The update system and the boot chain, designed together, make the device not only harder to compromise but able to heal itself. That resilience assumes that the images the device accepts are genuinely its maker's. The device can verify a signature; it cannot know whether the signing key was stolen, whether the build server was tampered with, or whether the update server is distributing something the maker never intended. Those are properties of the pipeline behind the device, to which the next chapter turns. Chapter 7: Protecting the Update Pipeline Every argument in the previous chapters has ended at the same place: the device trusts whatever its maker signs. The root of trust anchors a public key. Secure boot accepts images signed by the matching private key. The update agent installs them. This is exactly as it should be, and it has a corollary that device makers sometimes prefer not to dwell on. The most valuable target in the entire system is not any device. It is the machinery that produces and signs firmware. An attacker who can get their code signed by the maker's key does not need to break a single device. The fleet will download, verify and install the attacker's code on its own, and the security features described so far will help them do it. How update pipelines are attacked The pipeline from source code to installed firmware has several stages, and each can be attacked. Developers' accounts and source repositories can be compromised so that malicious code is committed. Build servers can be tampered with so that the binary they produce differs from the source code reviewers saw. Signing keys can be stolen, or the systems that use them can be tricked into signing something they should not. Update servers can be compromised to serve malicious or outdated images. And the network between server and device can be attacked, which is the threat most teams think of first and, given ordinary transport encryption and signed images, usually the least serious. Two incidents outside the IoT world illustrate the deeper threats. In 2019, Kaspersky disclosed an operation it named ShadowHammer, in which attackers compromised ASUS's software update infrastructure and distributed a backdoored version of the company's Live Update utility, signed with ASUS's legitimate certificates, to a large number of computers. The malicious code activated only on a small list of target machines identified by their network hardware addresses, which is part of why it went undetected for months. In 2020, investigators found that attackers had compromised the build environment of SolarWinds, inserting malicious code into the company's Orion network management software during compilation, so that the resulting signed updates carried a backdoor to thousands of customers, including government agencies. In both cases, verifying the signature on the update would have confirmed that it came from the vendor. It did. That was the problem. IoT device makers are attractive targets for the same reasons and often have weaker defences. Their build systems may be a single server under a desk; their signing key may be a file on a build machine, readable by any process that runs there; their update server may be a cloud storage bucket with credentials embedded in a deployment script. Each of these arrangements is common, and each means that the security of the entire fleet depends on the security of one ordinary computer. Protecting signing keys The first discipline is to treat firmware signing keys as the crown jewels they are. Private signing keys should be generated and stored in hardware security modules, or in cloud key management services backed by such hardware, so that they cannot be copied even by an administrator with full access to the systems around them. The signing operation should be performed by the hardware on request, under policies that require authentication, log every use, and ideally require more than one person to approve signing a production release. A key hierarchy limits the damage of any single compromise. The root key, whose hash is fused into devices and which therefore can never easily be changed, should be used as rarely as possible: ideally only to sign a small number of subordinate keys, and otherwise kept offline in a physically secured hardware module that is brought out only for key ceremonies. Day-to-day firmware signing is done with subordinate keys that the root has certified. If a subordinate key is compromised, the root can be used to revoke it and certify a replacement, and devices can be updated to trust the new key, without anyone needing to touch the fused value. Where hardware supports it, fusing hashes of several root keys, so that one can be revoked in favour of another, provides a further layer of recovery. Separate keys should be used for separate purposes and product lines. A key that signs bootloaders should not sign application images. A key for one product family should not be trusted by another. This is not merely tidiness. It means that a key compromised in a less sensitive context cannot be used to attack the most sensitive one. What a signing ceremony looks like For readers who have not seen one, it may help to describe what careful key handling involves in practice, since the term "key ceremony" can sound grander than it is. When a device maker creates the root key for a new product family, it convenes a small group: typically two or three key custodians, a security officer and a witness, none of whom can act alone. They meet in a controlled room with a hardware security module that has never been connected to a network. The module generates the root key internally, and the key is backed up only in encrypted form, split into shares held on separate smart cards by different custodians, so that a threshold of them, say two of three, must be present to restore or use it. The module then exports the root public key, whose hash will be fused into devices, and signs the certificates for the online subordinate signing keys. Every step is scripted in advance, performed in sequence, recorded and signed off by the witness. The module and the smart cards are then stored in separate secured locations. Day-to-day, production releases are signed by the subordinate key held in an online hardware module, but only through a release system that requires the build to have passed its tests and to carry provenance from the official build environment, and that requires approval from at least two named people. The module refuses requests that do not come from that system. Every signing operation is logged to a store that the release engineers cannot modify. The root is brought out again only to rotate a subordinate key, to revoke one that has been compromised, or to introduce a new algorithm. Each of those events is another ceremony. This may look elaborate for a consumer camera company, but the cost is a few days of effort a year, and the asset it protects is the authority to replace the software on every device the company has ever sold. Surviving compromise: The Update Framework and Uptane Protecting keys reduces the chance of compromise. A well-designed update system also assumes that some compromise will eventually occur and limits what an attacker can do with it. This idea is the basis of The Update Framework, known as TUF, which grew out of research by Justin Samuel, Nick Mathewson, Justin Cappos and Roger Dingledine published in 2010 under the title "Survivable Key Compromise in Software Update Systems," and has since been adopted widely in software distribution and hosted by the Cloud Native Computing Foundation. TUF's central insight is that a single signing key is a single point of failure, and that different aspects of an update's trustworthiness can be separated and signed by different keys with different protection. Table 4 summarises the main roles, and the extension of them in Uptane, the variant designed for vehicles. Table 4. Signing roles in The Update Framework and their purpose. Role What it signs Protects against Typical key protection Root The keys trusted for every other role Wholesale key compromise; enables rotation Offline, threshold of several keys Targets Which images exist, with their hashes Installation of unauthorised images Offline or tightly controlled online Snapshot The current version of all targets metadata Mix-and-match of old and new metadata Online Timestamp Freshness of the snapshot Freeze attacks serving stale updates Online, short expiry Director (Uptane) Which images a specific device should install Wrong or malicious images for a given device Online, per-device instructions The roles divide the attacker's problem. Compromising the online timestamp key, which must be available constantly, lets an attacker delay updates for a short time but not install anything the targets role has not authorised. Compromising the targets key lets an attacker describe malicious images, but clients will not accept that metadata unless it is consistent with the snapshot and timestamp. Compromising the root, which is kept offline and can require signatures from a threshold of several keys held by different people, is hard, and if any other key is compromised, the root can rotate it. Every piece of metadata carries an expiry time, so that an attacker who controls the network cannot freeze a device on an outdated version indefinitely by replaying old metadata, a threat that simple signed images do not address. Uptane, developed from 2016 by a team including researchers from New York University's Tandon School of Engineering, the University of Michigan Transportation Research Institute and the Southwest Research Institute, adapts TUF to vehicles and, by extension, to other complex fleets. It separates an image repository, which holds signed images and is relatively static and well protected, from a director repository, which instructs each individual vehicle which images to install and is necessarily online. A vehicle installs an image only if both repositories agree. An attacker who compromises the online director can, at most, instruct vehicles to install genuine images in combinations the image repository already authorised; it cannot introduce new code. Uptane is now maintained as a standard under the IEEE Industry Standards and Technology Organization, and its design is a useful model for any device maker whose fleet management system must be online to function. Adopting TUF or Uptane wholesale is a large step for a small embedded team, and constrained devices may be unable to process the full metadata. But the principles transfer to any design. Separate the key that says what images exist from the key that says what is current. Keep the most powerful keys offline. Require more than one key for the most powerful operations. Give metadata expiry times. And plan for key rotation from the beginning, because a system that cannot rotate keys will one day face the choice between trusting a compromised key and abandoning its fleet. Trusting the build Signing proves that the maker approved an image. It does not prove that the image corresponds to the source code the maker's engineers wrote and reviewed. SolarWinds showed what happens when that gap is exploited. Closing it requires attention to the build process itself. The most direct technique is the reproducible build, in which the build is made deterministic so that anyone compiling the same source with the same tools obtains a bit-for-bit identical binary. If two independent build systems, ideally operated by different people on different infrastructure, produce the same output, then compromising one build system is not enough to insert malicious code undetected. Reproducibility takes work: timestamps, file orderings, path names and toolchain versions all have to be controlled. The Reproducible Builds project, which has driven this work for Debian and other Linux distributions, documents the common sources of non-determinism and their fixes. Short of full reproducibility, build provenance provides much of the value. A provenance record states which source revision was built, by which build system, with which inputs, and is signed by the build system. The SLSA framework, an industry initiative originating at Google and now hosted by the Open Source Security Foundation, defines levels of build integrity, from simple documentation of the build process up to hardened, isolated builds with non-forgeable provenance. Tools such as Sigstore make it practical to sign artefacts and record those signatures in public, append-only transparency logs, so that a maker, or anyone else, can detect if an image was signed that should not have been. Transparency logs deserve particular emphasis. If every firmware image the maker signs is recorded in a public log, and devices check that an update appears in the log before installing it, then an attacker who steals a signing key cannot sign a targeted malicious image for a few devices without either recording it publicly, where it can be noticed, or failing the check. ShadowHammer, which targeted a small list of machines precisely to avoid notice, is the kind of attack this defends against. Knowing what you ship The update pipeline also determines how quickly a maker can respond to vulnerabilities in third-party code. When Ripple20 and Amnesia:33 were disclosed in 2020, many device makers could not immediately say whether their products were affected, because they did not have an accurate inventory of the components in their firmware. A software bill of materials, recording every component and version in each firmware image, turns that question from an investigation into a database query. Standard formats such as SPDX, maintained by the Linux Foundation, and CycloneDX, maintained by OWASP, make SBOMs machine-readable and exchangeable with customers and regulators. An SBOM is most useful when it is generated automatically by the build system for every release, rather than assembled by hand, and when it is matched continuously against vulnerability databases so that newly disclosed issues in included components raise alerts without anyone needing to remember to check. Regulators have noticed: the EU Cyber Resilience Act requires manufacturers to identify and document the components of their products, including by drawing up a software bill of materials, and United States federal procurement guidance has moved in the same direction. Cryptography for a decade A device sold today with a signing key fused into silicon will still be checking signatures against that key in the mid-2030s. That raises a question most product teams have not had to consider: what if the signature algorithm itself is broken in that time? The most discussed risk is a cryptographically relevant quantum computer, which would break the elliptic-curve and RSA signatures used by almost every device today. Nobody knows when or whether such a machine will exist, but the long life of devices and the immutability of their root keys make firmware signing a case where early preparation is warranted. NIST published standards for post-quantum signatures in August 2024, FIPS 204 for ML-DSA and FIPS 205 for SLH-DSA. Earlier, in 2020, it approved stateful hash-based signature schemes, LMS and XMSS, in Special Publication 800-208. Hash-based signatures rely only on the security of hash functions, are well understood, and are particularly suited to firmware signing, where the number of signatures over a key's life is modest and the signer can be carefully controlled. The NSA's Commercial National Security Algorithm Suite 2.0 recommends them for software and firmware signing and calls for their adoption in that role ahead of other post-quantum transitions. Practical preparation does not require a team to switch algorithms tomorrow. It requires the boot ROM, or at least the first mutable bootloader, to be able to verify more than one algorithm, so that a post-quantum signature can be added alongside or in place of a classical one later; it requires the update metadata format to accommodate larger signatures; and it requires the key hierarchy to allow new root keys to be introduced. Hardware vendors have begun shipping microcontrollers whose boot ROMs support hash-based signature verification for this reason. This is cryptographic agility in its most concrete form: designing so that the algorithm can change without the device being replaced. The pipeline is part of the product The theme running through this chapter is that the update pipeline is not infrastructure that supports the product; it is part of the product's security boundary. A threat model that covers the device but not the build server, the signing ceremony and the update repository is a threat model of the wrong system. The same rigour that goes into choosing a secure element should go into deciding who can trigger a production signing operation, how many people must approve it, and how anyone would know if a signature were issued that should not have been. With the device able to trust what it runs, and the pipeline able to produce things worth trusting, a final question remains. Both have to keep working for as long as the device is in use, which is usually far longer than anyone planned. The last chapter considers the device across its whole life, and the regulation that is beginning to hold makers to that obligation. Chapter 8: The Device Over Its Lifetime A connected product has two lifetimes. The commercial one runs from launch to the day the product is discontinued and its team reassigned, often three or four years. The physical one runs from the day a customer plugs the device in to the day it is finally thrown away, which for a router, a camera, a thermostat or an industrial sensor can easily be ten years or more. Most of the security failures described in this booklet happen in the gap between the two. The routers in the state-sponsored botnet disrupted in January 2024 were end-of-life models, still working, still connected, no longer patched. The cameras and video recorders in the Raptor Train network ran firmware whose vulnerabilities had been public for years. This chapter is about closing that gap. It covers the processes a maker needs to handle vulnerabilities after a product ships, the decisions involved in setting and honouring a support period, what should happen at end of life, and the regulatory regimes that are now turning these decisions from matters of corporate conscience into legal obligations. Hearing about vulnerabilities A maker cannot fix vulnerabilities it does not know about, and the people most likely to find them are outside the company: independent researchers, academic groups, customers' security teams and, unfortunately, attackers. The first lifetime obligation is to make it easy for the well-intentioned among them to report what they find. That means a published vulnerability disclosure policy, stating how to report an issue, what the reporter can expect in response and within what time, and committing not to take legal action against researchers acting in good faith. It means a monitored contact point, conventionally a security page on the maker's website and a security.txt file at a standard location on its domain, so that a researcher does not have to guess whom to email. It means internal processes for triaging reports, reproducing issues, developing fixes and coordinating the timing of public disclosure with the reporter. International standards describe these processes: ISO/IEC 29147 covers vulnerability disclosure and ISO/IEC 30111 covers the handling processes behind it. None of this is expensive compared with the alternative. Researchers who cannot find a way to report a vulnerability, or who are ignored or threatened when they do, will often publish anyway, and the maker learns of its problem at the same moment as every attacker. A disclosure policy is also now a legal requirement in the United Kingdom for consumer connectable products, and the EU Cyber Resilience Act requires manufacturers to have one. Many larger makers go further. Becoming a CVE Numbering Authority allows a maker to assign identifiers to vulnerabilities in its own products, which helps customers and scanners track them. Bug bounty programmes, which pay researchers for valid reports, can attract more attention to a product's security than an internal team alone could provide, though they work best once the basic disclosure process is functioning; a bounty programme bolted onto a company that cannot ship fixes merely accumulates reports. When a vulnerability is already being exploited The disclosure process just described assumes a vulnerability reported privately, with time to prepare a fix. The harder case is a vulnerability that attackers are already using, discovered when a researcher spots it in a botnet's code or a customer's network monitoring flags unusual traffic from a device. Here the maker's response is measured in hours. The European Union has made that timetable explicit. Under the Cyber Resilience Act, from September 2026, a manufacturer that becomes aware of an actively exploited vulnerability in its product must submit an early warning to the authorities within twenty-four hours, a fuller notification within seventy-two hours, and a final report within fourteen days of a corrective measure becoming available. Severe incidents affecting the security of a product carry similar obligations. These reports go to a single reporting platform operated by ENISA, the EU's cybersecurity agency, and to the relevant national authority. Whatever the legal regime, a maker facing active exploitation needs to be able to do several things quickly. It must be able to identify which products and firmware versions are affected, which is where the software bill of materials described in Chapter 7 earns its keep. It must be able to estimate how many devices in the field are exposed, which depends on telemetry about firmware versions. It must be able to offer mitigations before a full fix is ready, such as a cloud-side change that blocks the attack path, a configuration change pushed to devices, or clear guidance to owners. It must be able to build, sign, test and roll out a fix on an accelerated path without bypassing the controls that protect the pipeline. And it must be able to tell customers what is happening in plain language. None of these capabilities can be created in the middle of an incident. They are the product of preparation: an incident response plan that names who decides what, a pre-agreed fast path through the release process, and rehearsals that test both. Committing to a support period The second obligation is to decide, and say, how long the product will receive security updates. This is a harder decision than it sounds, because it commits engineering resources years into the future for a product that may by then generate no revenue. The decision has several inputs. The expected physical life of the product is the starting point: a support period much shorter than the time customers will realistically use the device leaves a long tail of unpatched units in the field. The software architecture matters greatly. A product built on a widely used operating system with long-term-support releases, such as a Linux kernel with an extended maintenance branch, or a real-time operating system with an active security process, is far cheaper to maintain than one built on a vendor-specific fork of an old kernel that the chip maker stopped updating years ago. The supply chain matters: if the chip vendor's board support package, binary drivers or Wi-Fi firmware will not receive fixes for the full period, the device maker will be unable to fix vulnerabilities in those components regardless of its own intentions. Contracts with component suppliers should therefore address security support explicitly. Once chosen, the support period should be stated clearly at the point of sale, in a form a consumer can understand. The UK's regulations require exactly this: a manufacturer must publish the minimum length of time for which security updates will be provided, expressed as a period or an end date, and may extend but not shorten it once published. The EU Cyber Resilience Act goes further and sets a floor: the support period should reflect the time the product is expected to be in use and should, in general, be at least five years, unless the product is expected to be used for less. These requirements turn the support period into a feature that buyers can compare. That is precisely what economists would prescribe for the market failure described in Chapter 1, because it makes a hidden cost visible at the moment of purchase. Honouring the commitment is a matter of organisation as much as engineering. Products that are no longer actively developed still need an owner: a team, however small, with the access, build environment, signing authority and knowledge to produce a fix. The update pipeline described in Chapter 7 has to keep working for every supported product, which argues for a shared pipeline across product lines rather than a separate one built for each launch and abandoned with it. Build environments for old products should be preserved, ideally in a reproducible, containerised form, because a fix that cannot be compiled cannot be shipped. End of life done well Eventually every product reaches the end of its support period. What happens next determines whether it becomes harmless electronic waste or a recruit for the next botnet. There is no single right answer, but there are several better ones than silence. The minimum is honest notice. Owners should be told, in advance and through channels they actually see, such as the companion app and email, that security updates will end on a given date, what risks that creates, and what they can do about it. Many end-of-life routers that ended up in botnets were still in service because their owners had no idea they were no longer supported. Better is to make the end-of-life state safer. A final update can reduce the device's attack surface by disabling remote management features, closing any remaining listening services or restricting functions that depend on code most likely to be attacked. For cloud-connected devices, a final update can allow the device to keep working in a local-only mode once the cloud service is withdrawn, so that owners are not forced to choose between a dead product and an insecure one, and so that the maker's eventual shutdown of its servers does not strand a fleet of devices reaching out to domains that someone else might later register. Better still, where the business can accept it, is to open the device to continued maintenance by others. Some makers have released documentation, signing arrangements or source code for discontinued products, allowing communities to keep them updated. This is not always possible, particularly when third-party licensed components are involved, but it is worth planning for at design time, because it is very difficult to arrange after the fact. In some cases the responsible course is to encourage replacement. For devices with known, serious, unfixable vulnerabilities, a maker can offer discounts or trade-in programmes, and some internet service providers have proactively replaced end-of-life customer routers. Regulators and law enforcement have increasingly been willing to name end-of-life devices in public advisories, which provides makers with both pressure and justification for such programmes. The regulatory landscape Several of the lifetime obligations described above have already appeared as legal requirements in earlier chapters. It is worth drawing the main regimes together, because they now shape product decisions in much the way that safety and electromagnetic compatibility rules always have. Table 5 summarises the principal ones as they stood in 2026. Table 5. Principal consumer IoT security regimes. Regime Jurisdiction Status Core obligations PSTI product security regime United Kingdom In force since 29 April 2024 No universal default passwords; disclosure policy; stated support period Radio Equipment Directive cybersecurity delegated act European Union Applies since 1 August 2025 Network protection, data protection and fraud resistance for radio devices Cyber Resilience Act European Union In force; reporting from September 2026; full application December 2027 Secure design, vulnerability handling, updates, SBOM, support period US Cyber Trust Mark United States Voluntary labelling, launched January 2025 Baseline criteria derived from NIST IR 8259 Cybersecurity Labelling Scheme Singapore Voluntary, tiered, since 2020 Tiers from baseline requirements up to independent testing The United Kingdom's regime is narrow and mandatory. It addresses only the three issues of default passwords, vulnerability disclosure and support periods, but it applies to almost every consumer connectable product sold in the country, and manufacturers, importers and distributors all carry obligations. The European Union's approach is broader and arriving in stages. The delegated act under the Radio Equipment Directive, which has applied since August 2025, brought basic cybersecurity requirements into the existing conformity regime for wireless devices, supported by a set of harmonised standards in the EN 18031 series. The Cyber Resilience Act, Regulation (EU) 2024/2847, entered into force in December 2024 and covers products with digital elements in general, not only radio devices. Its obligation to report actively exploited vulnerabilities and severe incidents to the authorities applies from September 2026; its full essential requirements, covering secure-by-design development, a secure default configuration, protection against unauthorised access, timely security updates, vulnerability handling and documentation including an SBOM, apply from December 2027. Products in certain more critical categories face stricter conformity assessment, including third-party evaluation for some. Penalties for breaching the essential requirements reach fifteen million euros or two and a half per cent of worldwide turnover. The United States has relied mainly on voluntary measures for consumer products, alongside procurement rules for federal agencies under the IoT Cybersecurity Improvement Act of 2020 and the NIST guidance it relies on. The NIST IR 8259 series, first published in 2020, sets out baseline capabilities that device makers should provide, including device identification, configuration, data protection, logical access control, software update capability and cybersecurity state awareness, together with non-technical supporting activities such as documentation and customer communication. The Federal Communications Commission's Cyber Trust Mark, adopted in 2024 and launched in January 2025, is a voluntary label for products tested against criteria derived from this baseline. Its administration has been unsettled; the testing organisation initially chosen as lead administrator withdrew in 2025, and the programme's long-term reach remains to be seen. Other jurisdictions are moving in similar directions. Singapore's tiered labelling scheme, introduced in 2020, has mutual recognition arrangements with several other countries' schemes, and a number of governments have adopted or adapted ETSI EN 303 645 as the reference for their requirements. At the industry level, ecosystem certifications such as Matter and the PSA Certified and SESIP schemes discussed in Chapter 4 provide additional assurance. For a device maker, the practical consequence is convergence. The regimes differ in scope and procedure, but their technical substance overlaps heavily, and nearly all of it can be traced to the provisions of EN 303 645 and the NIST baseline. A product designed to the principles of this booklet, with no universal defaults, a hardware-anchored chain of trust, a secure and reliable update system, a protected pipeline, a disclosure process and a stated, honoured support period, will meet the substance of every regime listed. What differs is the documentation, and documentation is easier to produce for a product that was designed securely than to invent for one that was not. Building the organisation that sustains it Lifetime security is ultimately a property of organisations rather than devices. The device can only stay as secure as the people responsible for it are able and willing to keep it. A few organisational practices separate makers that sustain security from those that announce it. Security has an owner with authority. Someone senior is accountable for the security of products in the field, has a budget that does not disappear when a product line is discontinued, and can stop a release that fails security criteria. Without that authority, security requirements tend to be negotiated away in the last weeks before launch. Security is part of the product definition. The support period, the root-of-trust hardware and the update architecture are decided at the start of a project, alongside the processor and the battery, because they constrain each other and because they are nearly impossible to add later. A product whose bill of materials was finalised without a secure element, or whose flash was sized without room for a second firmware image, has made its security decisions whether or not anyone thought about them. The field is monitored. Telemetry on firmware versions, update success rates and confirmation failures shows how much of the fleet is actually protected, which is often very different from how much has been offered an update. Threat intelligence on the maker's own products, from researchers, scanning services and law enforcement advisories, shows which vulnerabilities attackers are using. And the process is rehearsed. Signing key compromise, a critical vulnerability in a widely deployed product, or the sudden end of a chip vendor's support are events that will occur at some point over a product's life. Makers that have practised their response, including key rotation and emergency update rollout, handle them in days. Makers that have not will handle them in months, during which their customers' devices are someone else's. Conclusion It is tempting, ten years after Mirai, to tell the story of IoT security as one of slow progress. Default passwords are now illegal in major markets. Secure elements cost little. Open-source bootloaders implement verified, resumable updates. Standards bodies have specified manifests, attestation and survivable update metadata. Regulators in Europe and the United Kingdom have made vulnerability disclosure and support periods conditions of sale. All of this is true, and none of it has stopped botnets of routers and cameras from setting new records for attack volume, or state-backed groups from hiding inside hundreds of thousands of forgotten devices. The gap between those two facts is the subject this booklet has been circling. The tools to build defensible devices exist. What determines whether a particular device is defensible is whether its makers used them, together, as a system, and kept using them for as long as the device stayed connected. What follows from the argument The controlling claim of this booklet was that device security is decided in two places: the chain of trust that runs from silicon to application at boot, and the update channel that keeps that chain intact over the device's life. Having followed both through, it is possible to say more precisely what that implies for the people who build these products. The first implication is that the most consequential security decisions are made earliest. Whether the processor has a usable root of trust, whether the board has room for a secure element, whether flash is large enough for two firmware images, whether the boot ROM can verify more than one signature algorithm, and whether the software stack is one that can be maintained for a decade: these are settled when the bill of materials is fixed and the platform chosen, long before anyone writes a line of application code or a penetration tester picks up the product. A security review held a month before launch can find a debug port left open. It cannot add a hardware key store to a board that was laid out without one. Device makers who want secure products need security engineers in the room when platforms are selected, not only when releases are tested. The second is that the factory is part of the security architecture. Unique credentials, per-device identity keys, enforced secure boot and locked debug interfaces all depend on what happens on the production line, and the production line is the place where security designs most often silently degrade. Universal default passwords persisted not because engineers could not imagine alternatives but because the alternatives required changes to manufacturing that no one was willing to pay for. The same pattern recurs with secure boot fuses left unblown and JTAG left open. The remedy is to treat provisioning and lockdown as engineered, tested, audited processes, and to verify their output on units drawn from the line, rather than trusting that a specification was followed. The third is that the update system and the pipeline behind it are the real security perimeter. A device that verifies every signature is precisely as secure as the key that makes those signatures and the process that decides what gets signed. This shifts attention from the device, where most security effort has traditionally been spent, to build servers, signing ceremonies and update repositories, which are often less protected than the devices they serve. The frameworks described in Chapter 7, with their separated roles, offline roots, thresholds and expiring metadata, exist because their designers assumed some key would one day be compromised. Device makers should make the same assumption. The fourth is that security is a service, not a feature. It does not ship once; it is delivered continuously for the length of the support period, and that period must be chosen to match how long customers will really use the product. The cost of that service is real and recurring, which is why markets left to themselves underprovided it, and why regulation now requires it to be stated, and in Europe, to meet a minimum. For makers, the honest response is to price the support period into the product and build the organisation to deliver it, rather than to treat it as a compliance statement. What remains unsettled Several problems remain genuinely open, and it would be misleading to pretend otherwise. The installed base is the largest. Regulation applies to products placed on the market after it takes effect. The hundreds of millions of devices already in homes and businesses, many with default credentials, unpatched vulnerabilities and no update path, will remain in service for years, and they are where today's botnets live. Nobody yet has a satisfactory answer for them. Internet service providers can detect and quarantine some infected devices on their networks; law enforcement can disrupt particular botnets; makers can push final updates where an update path exists. But a meaningful part of the problem will only end when the devices themselves wear out. The economics of the cheapest products remain difficult. Regulation is effective against makers and importers that can be identified and held to account. It is much less effective against anonymous sellers shipping inexpensive devices directly to consumers through online marketplaces, the channel through which the pre-infected streaming boxes of the BadBox campaigns reached buyers. Making marketplaces share responsibility for the security of what they sell is an active policy debate, and its resolution will matter more for the bottom of the market than any engineering advance. The long-term cryptographic transition is uncertain in timing but not in direction. Devices built now with root keys that can only verify classical signatures may outlive the security of those signatures. The work of making boot chains and update systems algorithm-agile is only beginning across the industry. And the concentration of trust in vendor clouds, which solved the problem of internet-exposed devices, has created a new class of single points of failure, from cloud breaches to company shutdowns. How to give owners the security benefits of centrally managed updates without making their homes dependent on a company's continued existence and good conduct is a question the industry has not yet answered well. What to do differently For a device maker, the practical summary is short. Expose nothing to the internet that does not need to be exposed, and nothing by default. Give every device a unique identity anchored in hardware, and never ship a shared secret. Enforce secure boot and close debug interfaces in the factory, and check that you did. Design updates to be verified, atomic, confirmed and resistant to rollback, and make them automatic. Protect signing keys in hardware, separate their roles and plan for their compromise. Know every component you ship. Publish how to report vulnerabilities and how long you will fix them, then do so. Plan the end of life before the beginning. For a security engineer, the summary is to look past the device. The most damaging compromises of connected products have come through credentials no one thought of as part of the attack surface, factories no one audited, and update pipelines no one threat-modelled. The device is where the attack lands. It is rarely where the attack is decided. Mirai's authors did not need to be clever, because the devices they attacked had been built by people who did not expect anyone to try. The reason to build differently now is not only that the law requires it. It is that every device leaving a factory today will spend years connected to a network full of people who will try, and the only defences it will have are the ones its makers gave it before it left. Further Reading Manos Antonakakis et al., "Understanding the Mirai Botnet," Proceedings of the 26th USENIX Security Symposium (2017). The definitive measurement study of Mirai's growth, victims and attacks, drawing on telescope, honeypot and DNS data. Essential for anyone who wants the evidence behind the story told in Chapter 1. Justin Samuel, Nick Mathewson, Justin Cappos and Roger Dingledine, "Survivable Key Compromise in Software Update Systems," Proceedings of the 17th ACM Conference on Computer and Communications Security (2010). The paper that introduced the ideas behind The Update Framework. Still the clearest argument for separating signing roles and planning for key compromise. The Update Framework and Uptane specifications (theupdateframework.io and uptane.github.io). The living specifications of the frameworks discussed in Chapter 7, with design rationale and deployment guidance. IETF RFC 9019, "A Firmware Update Architecture for Internet of Things" (2021), and RFC 9124, "A Manifest Information Model for Firmware Updates in Internet of Things (IoT) Devices" (2022). Concise, vendor-neutral statements of what a secure update for constrained devices must carry and do. NIST Special Publication 800-193, "Platform Firmware Resiliency Guidelines" (2018). Frames firmware security as protection, detection and recovery. Written for servers and PCs, but its principles apply directly to embedded devices. NIST IR 8259, "Foundational Cybersecurity Activities for IoT Device Manufacturers," and NIST IR 8259A, "IoT Device Cybersecurity Capability Core Baseline" (2020). The baseline that underpins US federal guidance and the Cyber Trust Mark criteria; a practical checklist for product planning. ETSI EN 303 645, "Cyber Security for Consumer Internet of Things: Baseline Requirements." The thirteen provisions that most national regimes, including the UK's, draw on. Short, readable and the best single statement of the consumer baseline. Regulation (EU) 2024/2847, the Cyber Resilience Act. The legal text, with its annex of essential requirements, is more readable than its length suggests and is required reading for anyone selling connected products in Europe. Ross Anderson, Security Engineering: A Guide to Building Dependable Distributed Systems, third edition (Wiley, 2020). The standard reference on how security systems fail in practice, with chapters on physical tamper resistance, hardware security and the economics of security that bear directly on this booklet's themes. Jasper van Woudenberg and Colin O'Flynn, The Hardware Hacking Handbook (No Starch Press, 2021).** A thorough introduction to fault injection, side-channel analysis and other physical attacks on embedded devices, valuable for defenders who need to understand what their countermeasures face. Adam Shostack, Threat Modeling: Designing for Security (Wiley, 2014).** The practical guide to the discipline recommended in Chapter 2. Bruce Schneier, Click Here to Kill Everybody: Security and Survival in a Hyper-connected World (W. W. Norton, 2018).** A policy-oriented account of why markets underprovide device security and what governments might do about it, written just before the current wave of regulation. Hashtags: #IoTSecurity #InternetOfThingsSecurity #EmbeddedDeviceSecurity #EmbeddedSystems #EmbeddedSecurity #IoTCybersecurity #ConnectedDeviceSecurity #SmartDeviceSecurity #SecureIoT #IoTProtection #FirmwareSecurity #FirmwareProtection #FirmwareIntegrity #SecureFirmware #FirmwareUpdates #OverTheAirUpdates #OTAUpdates #SecureOTAUpdates #FirmwareUpdateSecurity #SecureUpdateMechanisms #DeviceSecurityArchitecture #HardwareRootOfTrust #RootOfTrust #ChainOfTrust #SecureBoot #VerifiedBoot #MeasuredBoot #BootloaderSecurity #AntiRollbackProtection #FirmwareSigning #DigitalSignatures #CodeSigning #CryptographicVerification #HardwareSecurity #SecureElements #TrustedPlatformModule #TPM20 #TrustedExecutionEnvironment #ArmTrustZone #OpenTitan #Caliptra #DeviceAttestation #CryptographicKeyManagement #SecureKeyProvisioning #FactoryProvisioning #ZeroTouchProvisioning #SecureManufacturing #IoTThreatModeling #AttackSurfaceManagement #AttackSurfaceReduction #STRIDEThreatModeling #IoTVulnerabilities #VulnerabilityManagement #VulnerabilityDisclosure #SecurityPatchManagement #DeviceLifecycleSecurity #EndOfLifeSecurity #IoTBotnets #MiraiBotnet #DDoSProtection #DefaultPasswordSecurity #UniqueDeviceCredentials #PasswordlessIoT #SecureDeviceOnboarding #MatterSecurity #MutualTLS #NetworkSecurity #CloudIoTSecurity #APIAuthentication #AccessControl #LeastPrivilege #DebugPortSecurity #JTAGSecurity #FirmwareSupplyChain #SoftwareSupplyChainSecurity #SoftwareBillOfMaterials #SBOM #SecureBuildPipeline #FirmwareSigningKeys #TheUpdateFramework #TUF #Uptane #AtomicUpdates #FailSafeUpdates #ABPartitioning #FirmwareRollbackProtection #RecoveryMechanisms #EmbeddedLinuxSecurity #MicrocontrollerSecurity #IoTSecurityStandards #ETSIEN303645 #CyberResilienceAct #PSTIAct #SecurityByDesign #SecureByDefault #IoTCompliance #ConnectedProductSecurity #IoTSecurityGovernance
- Grief and Loss on Campus (Coping Mechanisms and Seeking Accommodations While Mourning)
Download the Book (PDF): Introduction The call usually comes at a bad time, because there is no good time for it. It comes during a lab, or at two in the morning, or in the middle of a seminar when your phone buzzes three times in a row and you know before you look that something is wrong. Sometimes it does not come as a call at all. It comes as a slow illness that has been unfolding at home for months while you sat in lectures pretending to concentrate, and the final message is simply the end of a long waiting. Either way, you are now a student who is also a mourner, and the university around you continues as if nothing has happened. Deadlines keep arriving. The dining hall still opens at seven. Your roommate still asks whether you want to go to the thing on Friday. This book is for that situation. It is written for university and college students who have lost someone they love while they are away at school: a parent, a grandparent, a sibling, a friend, a partner, a mentor, sometimes a person whose importance nobody else quite understood. It is also written for the friends, resident advisers and staff who want to help and are not sure how. It is a short book, and it makes one argument. Grief is not a task you finish before returning to your studies, and it is not a set of stages you pass through on a timetable. It is a process that moves back and forth, sometimes in the same hour, between facing the loss and attending to ordinary life. The practical question for a bereaved student is therefore not "How do I get over this in time for finals?" but "How do I design a term, or a year, that has room for both?" That design has several parts: understanding what grief actually is, so that you stop judging yourself against a false map; asking for academic flexibility early, while the university's structures can still bend; building coping habits that support rather than numb; finding ways to keep a relationship with the person who died; and knowing, clearly and without shame, the signs that grief has become something that needs professional treatment or emergency help. Why a book specifically for students Bereavement is common on campus, far more common than the silence around it would suggest. A study by David Balk and colleagues, which used a randomly selected sample of undergraduates rather than volunteers, found that about 30 percent of students were within twelve months of the death of a family member or friend, and 39 percent were within twenty-four months. Earlier studies using convenience samples had produced similar estimates. At any moment, then, a meaningful share of the people in your lecture hall are grieving. Most of them are not talking about it. The student situation has its own shape. You are probably young, and for many students this is the first major death they have experienced as an adult. You are probably far from the people who shared the relationship with the person who died, so the family's grief happens in one place and yours happens in another. You live inside an institution built around cumulative, scheduled performance: problem sets, midterms, registration windows, scholarship requirements, visa conditions. You may be financially dependent on the person who died, or on a family now in disarray. And you are surrounded by peers who, with the best will in the world, often have no idea what to say. Research on bereaved students confirms what many of them already feel. Heather Servaty-Seib and Louis Hamilton, studying students at a large American university, found that bereaved students had lower grade point averages in the semester of their loss than comparable students who had not been bereaved. That finding is not a prediction about you, and it is not a verdict on your ability. It is a description of what happens when a demanding academic structure meets a major loss without adjustment. The adjustment is what this book is about. What current grief science says, in brief Much of what people believe about grief comes from a model that was never designed to describe it. The famous "five stages" of denial, anger, bargaining, depression and acceptance were proposed by the psychiatrist Elisabeth Kübler-Ross in 1969 to describe the reactions of people who were themselves dying, and they were drawn from clinical observation rather than systematic study of the bereaved. The model became a cultural fixture, and it is still printed on posters in counselling waiting rooms. But grief researchers have spent decades pointing out that bereaved people do not reliably move through those stages in order, that many never experience some of them at all, and that treating the stages as a checklist can make people feel they are grieving incorrectly. Contemporary grief science offers better tools. The dual process model, developed by the Dutch researchers Margaret Stroebe and Henk Schut, describes healthy grieving as an oscillation between two kinds of activity: confronting the loss, and attending to the changes and demands of life after the loss, including taking breaks from grief altogether. The continuing bonds perspective, associated with Dennis Klass, Phyllis Silverman and Steven Nickman, argues that people do not need to "let go" of the dead in order to recover, and that most people maintain some kind of ongoing inner relationship with the person they lost. Research on resilience, led in large part by George Bonanno, has shown that many bereaved people function reasonably well relatively soon after a loss, which is not a sign of coldness. And the recognition of prolonged grief disorder in the World Health Organization's ICD-11 and in the American Psychiatric Association's DSM-5-TR has given clinicians a clearer way to identify the minority of people whose grief becomes persistently disabling, and for whom specific treatments exist. These ideas are not academic decoration. Each one changes something practical. If grief oscillates, you can plan for oscillation. If bonds continue, you need not treat your ongoing connection to the dead as a problem. If resilience is common, you need not feel guilty for the days you manage well. If prolonged grief is a recognised condition, you can seek help for it without feeling that you have failed at mourning. How the book is organised Chapter 1 sets out what grief is, and why the five stages are a poor guide. Chapter 2 examines what is distinctive about grieving as a student, at a distance from home and inside an institution. Chapter 3 deals with the first weeks: getting the news, travelling, returning, and making the early decisions about your enrolment. Chapter 4 explains how academic accommodations actually work and how to ask for them, including sample wording. Chapter 5 applies the dual process model to the practical problem of studying while grieving. Chapter 6 looks at coping, separating the habits that help from those that cost more than they give. Chapter 7 turns to continuing bonds, meaning, ritual and anniversaries. Chapter 8 addresses, clearly and directly, when grief becomes a crisis: prolonged grief disorder, depression, trauma, suicidal thoughts, and what to do in an emergency. The conclusion draws out what follows for students and for institutions. You do not need to read it in order. If you have just received the news, Chapter 3 may be the place to start. If you are months into a loss and something feels wrong, go to Chapter 8. If you are frightened for your own safety right now, stop reading and contact emergency services: in the United States, call 911, or call or text 988 to reach the 988 Suicide and Crisis Lifeline. Emergency numbers differ in other countries, and your university's counselling service or security office can connect you with local crisis support at any hour. A note on scope. Many of the practical details in this book, such as specific offices, federal financial aid rules and disability law, reflect the American higher education system, and I say so where it matters. Universities in the United Kingdom, Canada, Australia, Europe and elsewhere have equivalent structures under different names, and every institution's policies differ. Treat the book as a guide to what to ask for and whom to ask, and check the details with your own university. Finally, this book is educational. It draws on current research and professional practice, but it is not a substitute for individual advice from a doctor, psychologist, counsellor or other qualified professional who knows your situation. Where something in these pages resonates strongly, especially in the last chapter, treat that as a reason to talk to one. Grief is not a problem with a solution. It is the form that love takes when the person is gone. But the conditions under which you grieve can be made more humane, and much of that is within your reach. Chapter 1: What Grief Actually Is Consider a second-year engineering student whose father dies suddenly in March. In the first week she feels almost nothing, and is ashamed of that. In the second week she sits an exam and does well, and is ashamed of that too. In the third week she cries in a bathroom stall after catching a smell of aftershave on a stranger in the corridor. By the end of the month she is angry, not at anyone in particular, and then one afternoon she laughs properly at a friend's joke and immediately feels she has betrayed him. At some point a well-meaning relative tells her she is "in the anger stage" and that acceptance will come. She wonders whether she skipped denial, or whether the numb first week counted, and whether the laughter means she is doing it wrong. Nothing in that sequence is abnormal. What is unhelpful is the map she has been handed. This chapter replaces it with a better one. Where the five stages came from, and why they mislead In 1969 the Swiss-American psychiatrist Elisabeth Kübler-Ross published On Death and Dying, based on her interviews with terminally ill patients in Chicago. She described five reactions that dying people often showed as they confronted their own death: denial, anger, bargaining, depression and acceptance. It was a humane and influential book. It helped push medicine toward talking honestly with dying patients instead of hiding the truth from them. Two things happened afterwards. First, the stages were transferred, without much examination, from dying people to bereaved people, although the book was not a study of mourners. Second, the stages hardened in popular culture from a loose description of reactions into a sequence, and then from a sequence into a prescription. Grief came to be imagined as a staircase: you enter at denial and exit at acceptance, and if you are not moving up the stairs, something is wrong. When researchers tested that idea, it did not hold up well. The most frequently cited study is the Yale Bereavement Study, reported by Paul Maciejewski and colleagues in JAMA in 2007. They followed 233 people bereaved by natural causes, measuring disbelief, yearning, anger, depression and acceptance between one and twenty-four months after the death. The study is sometimes described as supporting the stages, because when the researchers rescaled each indicator, the peaks occurred roughly in the order the stage model predicts. But the raw pattern told a different story. Acceptance was the most commonly endorsed response from the very first measurement, not something that arrived at the end. Yearning, which is not one of Kübler-Ross's five stages at all, was the dominant negative feeling throughout. Disbelief was not the dominant initial reaction. In other words, the reactions overlapped heavily, coexisted from the start, and the central emotional experience was missing from the model. In 2017 Margaret Stroebe, Henk Schut and Kathrin Boerner published a review in the journal Omega whose title began simply "Cautioning Health-Care Professionals". They concluded that stage theory lacks sound empirical support, conceptual clarity and practical usefulness, and, most importantly, that the expectation that people will or should pass through stages can harm those who do not. Their recommendation was blunt: the stage model should be set aside, including by bereaved people themselves. Why does this matter for a student? Because a false map produces false alarms. If you think grief has an order, you will worry when yours does not follow it. If you think acceptance comes last, you will distrust the moments early on when you calmly understand that the person is gone. If you think anger is mandatory, you may manufacture it, or worry that its absence means you did not love them enough. And if you think grief is linear, you will interpret the return of intense pain in month six, perhaps triggered by a song or an anniversary or a happy piece of news you cannot share with them, as a relapse, when it is simply how grief behaves. Table 1 sets out the main differences between the popular stage picture and the picture that emerges from current research. Table 1. The popular stage picture of grief compared with current evidence. Question Popular "five stages" picture What current research suggests What is the core experience? A sequence of distinct emotions Yearning and longing for the person, mixed with many other feelings Is there a set order? Yes: denial to acceptance No reliable order; reactions overlap and recur When does acceptance appear? At the end Often present from early on, alongside pain Is returning pain a setback? Yes No; waves of grief are expected, often triggered by reminders What does recovery mean? Letting go and moving on Adapting while often keeping an inner bond with the person Where did the model come from? Assumed to describe mourners Observations of dying patients (Kübler-Ross, 1969) What grief actually looks like If not stages, then what? The clearest way to describe grief is as the response of an attached mind to the permanent absence of someone it is attached to. Human beings are built to seek proximity to the people who matter to them, to notice their absence, and to feel distress until they are reunited. Death sets that system off, and the system does not immediately understand that reunion is impossible. Part of grieving is the slow, uneven process by which the mind learns a fact it already knows. This is why yearning sits at the centre. You reach for your phone to tell them something. You see someone with their walk in the street and your heart jumps. You dream about them and wake briefly believing they are alive. The psychologist Mary-Frances O'Connor, who studies the neuroscience of grief, describes this as a conflict between two kinds of knowledge: the explicit knowledge that the person has died, and a deep, habitual expectation that they are still out there, somewhere, reachable. Grief is, in part, the painful updating of the second by the first. Around that core, grief takes many forms. It is emotional: sadness, longing, guilt, anger, relief, anxiety, loneliness, numbness, and sometimes moments of unexpected peace or even joy. It is physical: exhaustion, disrupted sleep, loss of appetite or comfort eating, tightness in the chest, headaches, a lowered resistance to minor illness. It is cognitive: difficulty concentrating, forgetfulness, rereading the same paragraph five times, losing track of time, a sense of unreality. It can be perceptual: many bereaved people report sensing the presence of the person who died, hearing their voice, or briefly seeing them, and these experiences are common and not in themselves a sign of mental illness. It is behavioural: withdrawing, or throwing yourself into activity; avoiding reminders, or seeking them out; carrying their jumper, keeping their voicemail. Grief is also shaped by culture, faith and family. In some traditions mourning is loud and communal, with days of visitors, prescribed prayers and a clear sequence of rites; in others it is private and restrained. Some families expect a year of formal mourning; others expect everyone back at work after the funeral. Some traditions speak freely to and about the dead; others avoid saying their names. International students, and students from families whose customs differ from the campus mainstream, can find themselves grieving in a way that the people around them do not recognise, or unable to take part in rites happening thousands of miles away. There is no culturally neutral "normal" grief against which yours should be measured. The research described in this chapter is about broad patterns; how those patterns are expressed is yours and your family's. For students, the cognitive effects deserve particular attention, because the university is essentially a machine for measuring cognition. Many bereaved students are alarmed by how poorly they can read, remember or reason in the weeks after a death. This is not a sign that you have lost your intelligence. It is a predictable consequence of a mind occupied by an overwhelming event, compounded by poor sleep and stress. It usually eases, but it does not ease on a university's schedule, and that mismatch is the reason academic accommodations exist. Relief also deserves a word. If the person who died had been ill for a long time, or if the relationship was difficult, or if their death ended suffering, you may feel relief alongside sorrow. Many people are frightened by this and keep it secret. Relief after a long illness is common and does not cancel love. It often reflects how much you were carrying before the death. Many paths through, and why doing well is not doing it wrong One of the most important findings in modern bereavement research is that people differ enormously in how grief unfolds over time. George Bonanno and his colleagues, following people from before a spouse's death to well after it, identified several distinct trajectories. Some people show intense, prolonged distress. Some were already depressed before the loss and remain so. Some improve gradually. And a large group, the most common pattern in much of this research, show relatively stable functioning throughout: they feel pain, often acute pain, but they continue to sleep, work, laugh and connect with others within a fairly short time. Bonanno called this resilience, and he argued that it had long been mistaken for denial or "absent grief" by clinicians who assumed that everyone must fall apart. For students this is liberating in two directions. If you are coping reasonably well, you are not repressing or failing to grieve; you may simply be one of the many people whose grief coexists with continued functioning. You can still love and miss the person intensely. Conversely, if you are struggling badly, you are not weak; you may be on a different path that deserves more support. The comparison with your siblings, your parent or your friends is not a measure of who loved most. Several factors tend to shape how hard grief is. The nature of the relationship matters: the death of a parent, child, sibling or partner usually hits harder than a more distant loss, though closeness is personal and cannot be read off a family tree. The manner of death matters: sudden, violent or unexpected deaths, including accidents, suicides, overdoses and homicides, are associated with higher rates of severe and prolonged grief. Your prior history matters, including earlier losses, existing mental health conditions and past trauma. Your circumstances matter: financial stress, isolation, conflict within the family and the amount of practical disruption the death brings. And support matters, particularly whether the people around you allow you to talk about the person and your grief. None of these factors is destiny. They are reasons to take your own grief seriously and, in some cases, to seek support earlier rather than waiting to see whether things improve on their own. Grief before a death, and grief that has no single moment Not every loss arrives as a phone call. Many students spend months or years watching a parent or grandparent decline from cancer, dementia, heart failure or addiction, while trying to keep up with their courses from a distance. Researchers and clinicians call the grief that begins before a death anticipatory grief. It can include sorrow for the losses already happening, such as a grandmother who no longer recognises you or a father who can no longer do the things he loved, as well as dread about what is coming. Anticipatory grief has particular hazards for students. You may be living in two places at once: physically on campus, mentally at the hospital bedside. You may travel home repeatedly, each time wondering whether this is the last visit, and each time returning to a pile of missed work. You may feel guilty for being away, and guilty for wanting to be away. And when the death finally comes, you may find that the grief does not match what you expected. Some people feel they have already done much of their mourning and experience something close to calm. Others find that the death, however expected, is still a shock, because hope has a way of persisting until the very end. Both responses are common. If you are in this situation now, the practical advice in Chapters 3 and 4 applies before the death as well as after it. Universities can often offer flexibility for a student supporting a dying family member, and it is far easier to arrange that flexibility in advance than to reconstruct a collapsed term afterwards. Telling an adviser or dean "my mother is in hospice care and I may need to leave at short notice" is a legitimate and useful conversation to have early. There are also losses without a clear single moment: a parent who is alive but lost to severe dementia, a sibling who has disappeared into addiction, a family that has broken apart. The researcher Pauline Boss named this kind of experience ambiguous loss. It is not the main subject of this book, but many of the coping approaches described here apply to it, and a campus counsellor can help you think it through. Loss also compounds. A student who loses a grandparent in October and a friend in February is not simply grieving twice; each loss can reawaken the other, and the cumulative weight is real. If you have lost more than one person in a short period, or if a new death has brought back an older one, give that fact its due when you judge how much you can reasonably expect of yourself. Whatever shape your loss has taken, it is worth being precise about language, because students often receive mixed messages. Grief is not a mental illness. It is the normal response to losing someone you love, and for most people it softens over time without professional treatment, though it rarely disappears entirely. At the same time, grief is not trivial. It affects the body, the mind, relationships and functioning, sometimes profoundly. And for a minority of people, grief becomes stuck in a way that is persistently disabling. That condition, now called prolonged grief disorder, is discussed in Chapter 8. It is recognised by the major diagnostic systems precisely so that the people affected can receive effective treatment, not so that ordinary mourners can be labelled. The practical upshot is simple. You do not need a diagnosis to deserve support, and you do not need to be in crisis to see a counsellor. Most campus counselling services see bereaved students routinely. A few sessions with someone trained in grief can help you make sense of what you are experiencing, even if nothing is going wrong. Return, then, to the engineering student at the start of this chapter. She was not in the wrong stage. Her numb first week was the mind protecting itself while it absorbed a shock. Her good exam was a sign that she could still function, not that she did not care. The aftershave in the corridor was yearning, the central experience of grief, set off by a reminder. The anger was real and does not need a target. The laughter was a moment of respite, which, as Chapter 5 will show, is not a betrayal of grief but a necessary part of it. What she needed was not a map of stages but permission to grieve unevenly, and a university that could make room for that unevenness. Chapter 2: Grieving at a Distance Grief is always shaped by where it happens. For most of human history, people mourned among the people who shared the loss: the same household, the same village, the same congregation. They saw the body, attended the rites, and then lived for months among others who remembered the person and noticed their absence. Much of that structure did useful work. It confirmed that the death was real, it distributed the burden of sorrow, and it gave the bereaved a place where talking about the dead was normal. A student away at university often has almost none of this. The death happens somewhere else. The family gathers somewhere else. For a few days you may be home, inside the thick of it, and then you return to a place where nobody knew the person, where the rhythms of life have not paused, and where your grief is invisible unless you announce it. This chapter looks at what is distinctive about that situation, because naming it makes it easier to manage. Two places at once The first distinctive feature is geographical. Your grief is split between the place where the death happened and the place where you live now. Many students describe a strange unreality on returning to campus after a funeral. At home, everything was about the person: photographs on the table, relatives telling stories, food brought by neighbours, a shared understanding that normal life had stopped. On campus, nothing has stopped. The same posters are on the noticeboards. People are arguing about the same things. You walk through it like a ghost, carrying something enormous that nobody else can see. Some students find that the unreality makes the death feel less real, and they experience a delayed shock weeks later. Others find it a relief, a place where they can briefly be someone other than the bereaved one. The split is sharpest for international students and for students whose families live very far away. Consider a first-year student from Nigeria studying in Canada whose grandmother, who helped raise him, dies in November. The funeral will take place within days, the flight is long and expensive, a return trip in the middle of term may not be possible, and he is uncertain whether leaving the country will cause problems with his study permit. He may end up following the funeral on a video call at four in the morning, in a dormitory room, while his roommate sleeps. His grief is real and his family's rites are happening, but he is outside them in almost every sense. Students in his position often carry an added burden of guilt at not being present, and they may have no one nearby who knows the customs that would normally hold them. If this is your situation, it is worth speaking early to your university's international student office, both about the practical questions of travel and enrolment status and because staff there often understand this particular kind of distance better than anyone else on campus. The split also affects your connection to your family's grief. Your surviving parent or siblings are grieving too, and you may worry about them, feel guilty for leaving them, or feel excluded from what they are going through together. You may hear through a sibling that your mother is not eating, and be unable to do anything about it from four hundred miles away. You may feel pressure, spoken or unspoken, to come home, or pressure to stay at school and "keep going for Dad". These pressures can pull in opposite directions at once. There is no single right answer to how much to be at home and how much to be at school, and Chapter 3 discusses the decision in practical terms. What helps here is recognising that being physically distant from your family's grief does not mean you are grieving less, or loving less. It means you are grieving in a different place, under different conditions, and those conditions need attention of their own. The split does not end after the funeral. It returns at every break in the academic year. Going home for the winter holidays or the summer can be one of the hardest points of the first year after a death, because home is where the absence is most concrete: the empty chair, the coat still on the hook, the parent who is visibly diminished, the sibling who has been carrying the household while you were away. Students often expect the break to be restful and find it the opposite. Some families lean heavily on the returning student; others seem to have moved on in ways the student has not, or have rearranged the house so that the person's traces are gone. It helps to anticipate this, to talk with your family beforehand about what the holidays might look like, and to keep some contact with friends from university while you are home so that you are not wholly absorbed in the household's grief. Equally, returning to campus afterwards can bring a renewed sense of distance and guilt. The back-and-forth is part of the terrain. Distance also has a practical dimension that nobody warns you about. The person who died may have been the one who handled your finances, paid part of your tuition, signed your financial aid forms or provided health insurance. Their death may trigger urgent practical tasks: an estate, a house, a younger sibling who needs care, a business that someone has to run. Students sometimes become, overnight, the most capable adult available to a grieving family. These secondary losses and responsibilities are part of bereavement, and they often weigh as heavily on your studies as the grief itself. The developmental moment The second distinctive feature is timing within your own life. Most undergraduates are in what the psychologist Jeffrey Arnett called emerging adulthood, a period roughly between the late teens and mid-twenties in which people are forming an adult identity, loosening their dependence on family, and experimenting with who they might become. It is a period of transition by design. A death in the family during this period can collide with those transitions in painful ways. If you lose a parent, you may lose the person you were in the middle of renegotiating your relationship with. Many students describe a particular regret: they were just beginning to know their mother or father as a person rather than a parent, and now that conversation will never happen. Others are left with unresolved conflict, perhaps from arguments about independence, study choices, sexuality or religion that were supposed to be worked out eventually. The loss is not only of the person but of the future relationship you expected to have with them. A death can also stall or accelerate your own development. Some students find themselves pulled back into the family, taking on responsibilities they had just left behind. Others feel suddenly and prematurely adult, the first of their friends to have buried a parent, separated from their peers by an experience they cannot share. Future milestones acquire a shadow: graduation, a first job, a wedding, a first child, each of which the person will not see. These reactions are normal and they tend to recur. Grief in young adulthood is not something that happens once and is finished. It is often revisited at each later threshold, as you meet the loss again from the perspective of a new stage of life. Knowing this in advance helps. When graduation arrives and you find yourself unexpectedly devastated, that is not a relapse; it is grief meeting a new milestone. Peers, silence and the grief nobody recognises The third distinctive feature is social. Your daily companions on campus are mostly your own age, and many of them have not yet experienced a serious death. They do not know what to say, so they often say nothing, or say something clumsy, or ask once and then never again. Some avoid you. Some are too curious. Some assume that because you seem fine at a party three weeks later, the matter is closed. This is usually not unkindness. It is inexperience, combined with a common fear of saying the wrong thing and making it worse. But the effect on a bereaved student can be isolating. You may start to feel that your grief is a burden on your friendships, or that you have to perform being fine in order to be good company. Over time, many bereaved students report feeling fundamentally different from their peers, as if they have aged in a way their friends have not. A few practical responses can help. The first is to tell people what you want, since they are guessing. It is reasonable to say to a friend, "I don't need you to say anything wise. It helps if you ask about him sometimes, and if you don't disappear." Or, "Some days I want to talk about it and some days I really don't. If I change the subject, that's fine." Most friends are relieved to be given instructions. The second is to find at least one place where talking about the dead is normal. That might be a campus grief support group, a counsellor, a faith community, an online community for young adults who have lost parents, or a single friend who has been through something similar. Many universities host peer support groups for bereaved students, sometimes through the counselling centre and sometimes through student-run organisations; in the United States, the national organisation Actively Moving Forward, part of the nonprofit HEAL Grief, supports chapters on campuses for students who are grieving. Being in a room where nobody flinches when you mention the person can be a profound relief. The third is to recognise when your grief falls into a category that others may not acknowledge. The grief researcher Kenneth Doka coined the term disenfranchised grief for losses that are not openly acknowledged, socially supported or publicly mourned. On campus, disenfranchised grief is common. It includes: · the death of a grandparent who raised you, when others assume a grandparent's death is a minor event; · the death of a friend, especially a friend from home whom nobody at university knew; · the death of a former partner, or of a secret partner; · the death of a parent from whom you were estranged, where people either assume you feel nothing or cannot understand why you feel so much; · the death of someone from overdose, suicide, or another cause surrounded by stigma, where people avoid the subject entirely; · the death of a mentor, teacher, coach or colleague; · a pregnancy loss, whether yours or your partner's; · the death of a pet who was a central companion through years of your life. Disenfranchised grief is not smaller grief. Often it is harder, because the usual supports are missing. If your loss falls into one of these categories, it is still legitimate to seek support, and it is still legitimate to ask for academic flexibility. Many university policies are written narrowly around "immediate family", but deans and instructors frequently have discretion, and a clear, honest explanation of the relationship often goes further than you expect. Chapter 4 addresses how to make that case. If you are reading this as the friend rather than the bereaved student, the most useful things you can do are simpler than you might fear. Say something rather than nothing: "I heard about your dad. I'm so sorry" is enough. Use the person's name. Offer specific help rather than a general "let me know if you need anything", which puts the work back on the grieving person: offer to bring food, to walk with them to a lecture, to sit with them while they write an email to a professor, to take notes in a class they miss. Keep checking in after the first fortnight, when most support falls away; a message at one month, three months and on the anniversary often means more than anything said at the funeral. Tolerate silence, tears and repetition without trying to fix them. Avoid comparisons ("I know exactly how you feel, my dog died") and avoid sentences that begin "At least". And if you are worried about your friend's safety, do not keep it to yourself: Chapter 8 explains what to do. Consider, too, how the same situation can feel from both sides. A third-year student returns after her brother's funeral. Her housemates, not knowing what to do, decide to give her space, and for two weeks nobody mentions him. She interprets their silence as indifference, and begins spending more time alone. They interpret her withdrawal as a wish to be left alone, and give her even more space. Nobody has done anything wrong, and yet a gap has opened that neither side intended. A single honest sentence from either side, "I don't know what to say, but I think about you and him a lot", or "I'd actually like it if you asked about him", can close it. Grief in public and in private: phones, feeds and group chats A final feature of the modern student's situation is that grief now has a digital life. News of a death may travel through group chats and social media before your family has told everyone. You may learn of a friend's death from an Instagram story. You may find yourself fielding messages from dozens of people while still in shock. After a death, the person's social media profile may remain, continuing to generate birthday reminders and "memories" at unexpected moments. None of this is inherently good or bad. Digital spaces can support grief: many people find comfort in shared memorial posts, in rereading old messages, or in hearing a recorded voice. For students far from home, a family group chat can become the place where mourning is shared across distance. But digital grief has hazards too. Public posting can create pressure to grieve performatively. Comparing your grief to others' visible grief can make you feel you are doing it wrong. Automated reminders can ambush you in the middle of a lecture. And scrolling late at night through old photographs can disrupt sleep in a way that compounds everything else. A few small decisions can give you more control. You can mute or archive threads for a while without leaving them. You can ask a friend to handle messages for you in the first days, letting people know you have received their kindness but are not able to reply. If you are grieving a friend whose death is being discussed widely online, you can decide deliberately when and how much to read. And if you want to keep the person's messages, voicemails or photographs, back them up before accounts change or phones are replaced; many bereaved people later treasure these more than they expected, and some are distressed to find them lost. Most major platforms have processes for memorialising the account of someone who has died, which a family member can usually initiate when ready. These four features, geographical split, developmental timing, peer silence and digital exposure, are the reasons why generic grief advice often does not fit students well. Advice to "lean on your community" assumes your community is present and understands. Advice to "take all the time you need" ignores the fact that your term has a fixed end date and your scholarship may have conditions. Advice to "get back into your routine" can be hollow if your routine consists of competitive assessments you can no longer concentrate on. The rest of this book tries to be specific to your situation. It assumes you are grieving in one place while your family grieves in another; that you are at an age when a death reshapes your developing sense of self; that the people around you may not know how to help unless you tell them; and that you are inside an institution with rules, deadlines and offices that can either work against you or, if you know how to use them, work for you. The next chapter begins with the most urgent part of that: the first weeks. Chapter 3: The First Weeks In the first days after a death, most people are in some degree of shock. Thinking is slow and fragmented. Simple decisions feel enormous, and enormous decisions get made in a blur. It is a poor state in which to manage a university career, and yet the first weeks are precisely when several consequential decisions tend to arrive: whether to go home and for how long, whom to tell, what to do about the exam on Thursday, whether to stay enrolled at all. This chapter is deliberately practical. It is organised around the sequence of events most students face, and its central advice is simple: do the minimum necessary in the first days, tell one person at the university who can tell others, and postpone large, irreversible decisions until you are thinking more clearly, while protecting your options in the meantime. The first forty-eight hours When the news arrives, your first job is to take care of yourself in the most basic way and to get to where you need to be. Everything else can wait, including coursework, and including the email to your professors. If you are alone when you receive the news, contact someone. That might be a friend down the corridor, your resident adviser, a housemate, or your campus counselling service, many of which have an after-hours line. You do not need to know what you want from them. It is enough to say, "I've just heard that my mother has died, and I don't want to be on my own." Shock can make people behave in ways they do not expect: sitting motionless, cleaning obsessively, going to class as though nothing has happened, or feeling physically unwell. Having someone with you makes it safer and less lonely. If you need to travel home, ask for help with the logistics. Booking flights or trains, packing and arranging lifts are exactly the kind of tasks a friend can take over. Some airlines have historically offered compassionate or bereavement fares, but these policies vary, are often limited, and have been withdrawn by many carriers, so do not spend precious energy chasing them; a friend or family member can check. Some universities have emergency funds that can help with the cost of urgent travel home; the dean of students' office or the student support office will know. International students should contact their international student office before leaving the country if at all possible, because leaving mid-term can have implications for visa status or enrolment requirements, and advisers there can explain what paperwork or permissions are needed. If you cannot reach them before you leave, email them as soon as you can. Before you go, or within a day or two of arriving, send one message to one office at your university. In most American universities this is the dean of students' office, sometimes called student affairs, student support or the student advocacy office. In the United Kingdom it may be your personal tutor, your department's student support officer or the student wellbeing service. In many institutions, one of these offices will notify your instructors on your behalf that you are away because of a family emergency, without you needing to write to each one. This is one of the most useful and least known services universities offer. A short message is enough: "My father died yesterday and I am travelling home for the funeral. I expect to be away until at least next Wednesday. Could your office let my instructors know? My courses this term are [list]. I will be in touch about next steps when I am back." You do not need to provide documentation at this point, and you do not need to have decided anything. The purpose is only to create a record that something serious has happened, and to stop the clock from running silently on missed work. At home, and coming back At home, your role in your family's mourning will depend on many things: your relationship with the person who died, your family's customs, your age and your siblings'. Some students find themselves at the centre, organising the funeral, supporting a surviving parent, speaking at the service. Others feel peripheral, uncertain where they fit. Both are hard in different ways. Try, if you can, to take part in whatever rites matter to you and your family. Research on funerals and rituals is less conclusive than people assume about specific psychological benefits, but many bereaved people describe these occasions as important moments of recognition and connection, and missing them is a common source of later regret. If you cannot be there, whether because of distance, cost, immigration constraints or family conflict, it is worth finding some way to mark the occasion where you are: watching a livestream with a friend beside you, lighting a candle at the time of the service, visiting a place of worship, writing a letter to the person. Chapter 7 returns to the value of ritual. The question of when to return to campus has no universal answer, but a few considerations help. Staying away too briefly can mean arriving back in a state of shock, unable to function, surrounded by people who expect you to be back to normal. Staying away too long can mean facing an insurmountable backlog and losing the structure that university life provides, a structure that some students find genuinely helpful in grief. For many students a middle path works: stay for the funeral and a few days after it, return to campus, but return to a reduced load with explicit arrangements for catching up. Coming back is often harder than students expect. The first days on campus can bring a second wave of grief as the unreality described in Chapter 2 sets in. It helps to plan the return a little. Let a friend know when you are arriving and ask them to meet you. Arrange to see someone from the dean's office or your academic adviser in the first week. Book an appointment with the counselling service, even if you do not feel you need it yet; waiting lists at many campus services can be long, and it is easier to cancel an appointment than to get one in a hurry. And give yourself explicit permission to do less for a while. A particular trap awaits students whose loss falls close to an exam. It is common for a bereaved student to decide, out of shock, pride or a wish not to make a fuss, to sit an exam a day or two after the death. Sometimes that works; for some people, the familiar task is a welcome distraction. But students should know that in some systems, particularly in the United Kingdom, universities operate a "fit to sit" principle: if you choose to sit an assessment, you are usually deemed to have declared yourself fit to do so, and it can then be much harder to claim afterwards that your performance was affected by circumstances you already knew about. American institutions handle this less formally, but the same logic often applies in practice: it is easier to reschedule an exam before you take it than to have the grade set aside afterwards. If an exam falls in the days after a death, it is almost always worth asking whether it can be deferred, even if you then decide to sit it anyway. There is also the question of what to tell the people you live with. Your roommate, housemates or resident adviser do not need details, but they will be living alongside your grief, and a short explanation spares everyone a lot of awkward guessing. Something like "My grandad died last week. I'm okay some of the time, but I might be quiet or up at odd hours for a while" is enough. Resident advisers are trained to connect students with support, and letting yours know means that someone nearby will notice if you begin to struggle. Deciding what to do about your enrolment Once you are back, or once the first shock has passed, you will face a question that can feel overwhelming: can you actually do this term? The options are more varied than most students realise, and they differ considerably in what they protect and what they cost. Table 2 summarises the main options found in many universities; the names and rules differ between institutions and countries, so treat it as a guide to what to ask about. Table 2. Common options for adjusting enrolment after a bereavement. Option What it usually means What it protects Main things to check Stay enrolled with extensions Keep all courses; negotiate new deadlines Progress, scholarships, visa status Whether extensions are realistic given all deadlines Incomplete grade Finish remaining work after the term ends Credit already earned in the course Deadline for completion; who supervises; effect on aid Drop one or more courses Reduce load, sometimes after the normal deadline Grades in the remaining courses Minimum credit load for aid, housing, visa and insurance Withdraw from the term Leave all courses for the current term Transcript from a failing term Tuition refund rules; aid repayment; re-enrolment process Leave of absence Pause studies for a defined period Place at the university; time to recover Length allowed; conditions for return; housing and insurance Some general principles help in choosing among these. First, protect options before deciding. In the first weeks you are unlikely to know how you will be functioning a month from now. Deadlines for dropping courses without penalty, or for withdrawing with a partial refund, may be approaching. If one is imminent, ask the dean's office whether, given your bereavement, you can have it extended or have a late withdrawal considered later if needed. Many universities can grant retroactive or late withdrawals for serious circumstances, but it is much easier when the circumstances were documented at the time. Second, look for the smallest adjustment that makes the term survivable. Often that is not withdrawal but a combination: extensions in two courses, an incomplete in a third, perhaps dropping a fourth. Students sometimes withdraw entirely in the first week of grief, then discover a month later that they would have preferred structure and company, and are now at home with nothing to do but grieve. Others push on at full load and find themselves failing everything by the end of term. The middle options are often the wisest, and advisers can help you piece them together. Third, check the knock-on effects of every change. In the United States, dropping below full-time status can affect federal financial aid, institutional scholarships, health insurance through a parent's plan, campus housing eligibility and, for international students on F-1 visas, immigration status. International students in the US can sometimes receive authorisation for a reduced course load for medical reasons, and your designated school official in the international office is the person to ask before you change anything. Athletes need to speak with their compliance officer about eligibility rules. In other countries the particular risks differ, but the principle is the same: before you drop or withdraw, ask what else it affects. Fourth, treat a leave of absence as a legitimate choice, not a failure. For some students, especially after a traumatic death, the death of a parent with major family consequences, or a loss that has triggered a mental health crisis, stepping away for a term or a year is the right decision. Many students who take a leave return and complete their degrees. The key is to leave in a way that makes return straightforward: understand the process for coming back, keep in touch with an adviser, and use the time for recovery and support rather than simply isolation. Fifth, do not make the decision alone. An academic adviser, a dean of students, a counsellor and a trusted family member can each see different parts of the picture. It is reasonable to say to an adviser, "I don't know whether I can finish this term. Can you walk me through my options and what each one would mean for me?" It may help to see how these principles play out. The following are composite, illustrative scenarios rather than real individuals. Consider a junior studying chemistry at an American university whose mother dies of cancer in the ninth week of a fifteen-week semester. He had known for months that she was very ill and had travelled home several times. After the funeral he returns to campus exhausted, behind in all four courses, with two midterms missed. He emails the dean of students, who notifies his instructors and arranges a meeting. With his adviser he works out a plan: he takes an incomplete in organic chemistry, where the missing work is a lab component that can be done the following term; he negotiates a two-week extension on a history paper; he sits a make-up exam in statistics; and he drops a fourth course that he was taking as an elective, having first confirmed with the financial aid office that his remaining load keeps him at full-time status. He also begins weekly counselling sessions. The semester is hard, but it is survivable, and it does not leave a failing record behind it. Now consider a first-year student at a British university whose older brother dies in a road accident in her first term. The death is sudden and violent; she learns of it by phone late at night. When she returns to university after the funeral she cannot sleep, has intrusive images of the accident, and cannot sit in a lecture hall without feeling panicky. After talking with her personal tutor and the university's wellbeing service, she decides to interrupt her studies for the rest of the academic year and return the following September. Her tutor explains the process, confirms that her place will be held, and helps her understand the implications for her student finance. At home she begins trauma-focused therapy through her general practitioner. When she returns the next year, she starts again in first year alongside a new cohort, with a support plan in place. Neither student made the "right" choice in any universal sense. Each made a choice that fitted the loss, their state and their circumstances, and each made it with the help of people who understood the system. That, more than any specific option, is what the first weeks require. What to leave for later Finally, it is worth naming what does not need to be decided in the first weeks. You do not need to decide whether to change your major, transfer to a university closer to home, drop out, or reorganise your life around the loss. Grief often generates powerful impulses toward change: to go home permanently, to abandon a path that the person who died had encouraged, or conversely to commit instantly to a career they wanted for you. Some of these impulses will turn out to be wise. Many will look different in six months. A useful rule of thumb, widely offered by grief counsellors, is to avoid major irreversible decisions for some months after a significant loss where you can. Where a decision truly cannot wait, make it with advice, and prefer the option that keeps the most doors open. You are allowed, in these first weeks, simply to get through. The task is not to grieve well, study well or make excellent decisions. It is to keep yourself safe, keep your options open, and let people help you. The more considered work of arranging a term that can hold your grief comes next. Chapter 4: Asking for Accommodations Many bereaved students never ask for help from their university. Some do not know it is possible. Some assume that grief is private and that asking for special treatment would be inappropriate or even dishonest. Some worry about being judged, about being seen as weak or as using a death to get out of work. Some simply cannot face writing the emails. And some try to ask too late, after a term has already collapsed, when the options available have narrowed sharply. This chapter makes a simple case: asking for academic flexibility during bereavement is legitimate, common and usually successful, and it works best when done early, clearly and through the right channels. Universities are not always well designed for grieving students, and policies vary widely, but most institutions have more flexibility than students expect. The skill is knowing where it lives and how to reach it. Why asking is legitimate Start with the principle. Academic assessment is supposed to measure what you have learned and what you can do. It is not supposed to measure how recently someone you love has died. When a major bereavement impairs concentration, memory, sleep and motivation, as Chapter 1 described, an unadjusted assessment ceases to measure what it is meant to measure. An extension or a deferred exam is not a favour; it is a way of restoring the assessment's validity. Framed that way, asking is not an evasion of standards but a way of meeting them. There is also a practical argument. Instructors and administrators deal with bereaved students every term; given the prevalence figures cited in the introduction, a professor teaching a large course can expect to have several grieving students in it at any time. Most would far rather hear from you early, when an extension is easy to grant, than at the end of term, when you are failing and the options are poor. And in most institutions, faculty and staff are explicitly encouraged to be flexible in these circumstances. It is worth naming the internal objections that stop students from asking, because they are usually more powerful than any external obstacle. One is comparison: "Other people have it worse; my grandfather was old; my friend's mum died last year and she didn't ask for anything." Grief is not a competition, and the question is not whether your loss is the worst on campus but whether it is affecting your work. Another is the fear of seeming to exploit a death, which is often strongest in the most conscientious students, precisely the ones least likely to exploit anything. A third is a sense that the person who died would have wanted you to push on. Perhaps they would; but most people who love a student would also want that student to finish their degree in good health, rather than fail a term out of loyalty. And a fourth is simple exhaustion: the emails feel impossible. That one has a practical answer. Ask a friend, a resident adviser or a counsellor to sit with you while you write them, or to draft them for you to edit. Many students find that the hardest part is the first message, and that once one office is involved, the rest follows. Finally, asking does not commit you to using what is offered. If you secure an extension and then find you can meet the original deadline, nothing is lost. Many students discover that simply knowing flexibility exists relieves enough pressure to let them keep going. The map of offices The biggest obstacle for many students is not reluctance but confusion: universities are complex, and it is not obvious whom to ask. The general structure is similar across many institutions, even where the names differ, and Table 3, later in this chapter, summarises it. Here is what each part of the system typically does. The dean of students or student support office. In American universities, this is usually the first place to go. Staff there can notify your instructors of an absence, verify circumstances, advise on your options, connect you with other offices, and in some cases advocate on your behalf. Some institutions have a specific bereavement policy administered through this office, which may specify a number of days of excused absence for the death of a family member and the documentation required. In the United Kingdom, equivalent roles are often held by personal tutors, senior tutors, student support officers and wellbeing services; in Australia and Canada, by student services and academic advisers. The key point is that there is usually someone whose job includes coordinating help, so that you do not have to negotiate separately with every instructor. Your instructors. Individual professors and lecturers usually control day-to-day flexibility in their own courses: short extensions, permission to miss classes, alternative arrangements for participation. Even when a central office has notified them, it helps to write to each instructor briefly yourself once you are able, because it opens a direct line of communication. Your academic adviser or department. Advisers can help you think through your overall load, the consequences of dropping courses, and your path to graduation. Departments often control decisions about incompletes, course substitutions and program requirements. The registrar. The registrar's office handles the formal mechanics: late drops, withdrawals, incompletes, leaves of absence. Its deadlines are strict, but most registrars have processes for exceptions supported by a dean. The exams or assessment office, and extenuating circumstances processes. In British, Australian, Irish and many European universities, there is often a formal process for claiming that circumstances beyond your control affected your ability to complete assessments. In the UK it is typically called extenuating or mitigating circumstances; in Australia, special consideration. These processes usually have their own forms, deadlines and evidence requirements, and a bereavement of a close family member is almost always among the recognised grounds. Missing the deadline for such a claim can be costly, so it is worth finding out early how it works at your institution. Disability or accessibility services. Grief itself is not a disability. But if your bereavement leads to, or worsens, a mental health condition that substantially affects your studies, such as major depression, post-traumatic stress disorder, an anxiety disorder or prolonged grief disorder, you may be eligible for formal accommodations. In the United States, the Americans with Disabilities Act and Section 504 of the Rehabilitation Act require colleges to provide reasonable accommodations for students with qualifying disabilities, including psychiatric ones. Registering with the disability services office typically requires documentation from a clinician. Formal accommodations can include extended time on exams, a reduced course load treated as full-time for some purposes, flexible attendance or deadlines, or a quieter exam room. The advantage of this route is that it is ongoing and does not rely on the goodwill of each instructor. The disadvantage is that it takes time to set up, which is another reason to start early if you think you may need it. Other countries have comparable legal frameworks, such as the Equality Act 2010 in Great Britain, and comparable university services. Counselling and health services. Campus counsellors and doctors provide support, and they can also provide documentation where needed, subject to your consent. They cannot usually grant academic accommodations themselves, but their letters often carry weight with the offices that can. Financial aid. If a bereavement affects your academic performance or your enrolment, it may affect financial aid. In the United States, students receiving federal aid must maintain satisfactory academic progress, and institutions must allow students who fall short to appeal on grounds that include the death of a relative. Financial aid officers can also sometimes adjust aid to reflect a changed family financial situation, for example after the death of a parent who was a wage earner, through what is often called a professional judgement review. It is worth asking. The international student office. For international students, this office is essential before any change to enrolment, for the reasons discussed in Chapter 3. How to ask: wording, documentation and timing When you write to an instructor, keep it short, factual and specific. You do not owe anyone a detailed account of your grief, and long emails are harder to write and to read. A structure that works well has four parts: what happened, how it affects your work, what you are asking for, and an offer to discuss. For example: "Dear Professor Ahmed, I am writing to let you know that my grandmother, who raised me, died last week, and I was away for the funeral. The dean of students' office has sent you a notification. I am finding it hard to concentrate at the moment and I am behind on the reading. I would be grateful for a one-week extension on the essay due on the 14th. I'd be happy to meet during your office hours if that would help. Thank you for your understanding. Best wishes, [Name, student number, course]" Several details in that example matter. It names the relationship and, where it might not be obvious, explains its significance ("who raised me"). It connects the loss to its effect on the work. It makes a specific, reasonable request rather than an open-ended one. And it offers conversation without demanding it. If you are unsure what to ask for, you can ask for advice instead: "I'm not sure what would be most helpful. Could we talk about what options are available?" Many instructors will suggest arrangements you had not thought of. Documentation. Many universities ask for some evidence of a bereavement, especially for formal processes such as exam deferrals, extenuating circumstances claims or late withdrawals. Acceptable evidence varies but commonly includes an obituary or death notice, a funeral programme, a death certificate, or a letter from a funeral director, faith leader, doctor or counsellor. Being asked for evidence can feel intrusive, even insulting, at a time of grief. It helps to understand that the requirement usually exists to ensure fairness across many students rather than because anyone doubts you. If documentation is hard to obtain, for example because the death occurred abroad, the family is estranged, or customs do not include printed materials, say so; offices can often accept alternatives such as a letter from a counsellor who has seen you. For disenfranchised losses, a brief explanation of the relationship's significance, perhaps supported by a counsellor's letter, is often enough to persuade a dean to exercise discretion. Timing. The single most important principle is to ask early. It is easier to extend a deadline that has not yet passed than to excuse one that has. It is easier to defer an exam than to annul a result. Extenuating circumstances processes usually have deadlines that fall soon after the assessment period. And a record made at the time of the loss is far more persuasive later, if you need a retroactive withdrawal or a financial aid appeal, than a reconstruction made months afterwards. Even if you do not yet know what you need, a note to the dean's office in the first week or two creates that record. Planning the catch-up. Extensions pile up. A student who receives five one-week extensions in the same fortnight may simply have moved the crisis to a later date, when the new deadlines collide with the ordinary ones. When you negotiate extensions, look at the whole calendar, and ask your adviser to help you stagger them realistically. Sometimes the right answer is not an extension at all but a different form of flexibility: an incomplete, a reweighted assessment, or dropping a course. When the answer is no, and other complications Most requests are granted, but not all. Some instructors are inflexible; some courses have genuine constraints, such as laboratory sequences or clinical placements, where work cannot simply be postponed; and some policies are narrowly written. If an instructor declines a reasonable request, it is appropriate to go to the dean of students' office or your department's director of undergraduate studies and ask for advice. You are not complaining about the instructor; you are asking the institution to help you find a workable arrangement. Deans often have authority, or influence, that individual instructors lack. In British universities, a senior tutor or the extenuating circumstances panel usually has the final word, and appeals procedures exist if you believe a decision was wrong. Sometimes, despite everything, students do not ask in time. Consider a sophomore whose best friend from high school dies by suicide in October. She tells almost no one at university, keeps attending classes in a daze, stops submitting work in November, and ends the semester with two failing grades. In January, now seeing a counsellor, she realises what has happened. Her situation is not hopeless. Many American universities allow retroactive withdrawal from a term, or from individual courses, when a student can show that serious circumstances prevented them from functioning or from seeking help at the time. A letter from her counsellor describing the timing and impact of the loss, together with an obituary and a personal statement, may be enough for a dean to approve replacing the failing grades with withdrawals. The process is slower and less certain than an early request would have been, and financial aid consequences may still need to be addressed separately. But it exists, and students in her position should ask about it rather than assuming the record is fixed. British and other systems have their own routes for late claims, usually requiring an explanation of why the circumstances could not be reported earlier; the fact that grief itself made reporting impossible is often a recognised reason. Bereavement policies defined in terms of "immediate family" can be a particular obstacle for students grieving grandparents, friends, partners or others outside the narrow definition. In practice, such policies usually set a minimum rather than a maximum, and deans can exercise discretion. The approach that tends to work best is a calm explanation of what the person meant to you and how the loss is affecting your work, supported where possible by a counsellor's letter. Graduate students face some additional complications. Their work may be less structured by deadlines and more by the expectations of a supervisor, a funding body or a lab. Graduate students who are also teaching assistants or research assistants may be employees, subject to employment leave policies as well as academic ones. It is usually worth speaking both to your supervisor and to the graduate school or graduate programme director, and, where relevant, checking whether your funding body has provisions for bereavement or compassionate leave. Professional programmes, such as medicine, nursing, law or teacher training, often have strict attendance or placement requirements tied to professional accreditation. Flexibility may be more constrained, but programmes of this kind usually also have established procedures for deferring placements or repeating a rotation. Ask the programme's student affairs office early. Table 3 draws together the main offices described in this chapter and what each is usually able to do. The names vary, so use the second column to recognise the equivalent at your institution. Table 3. Where to go for different kinds of help after a bereavement. What you need Office to contact (common names) What they can often do Notify all instructors at once Dean of students; student support; personal tutor Send an absence notice; coordinate help Extensions or missed classes Course instructor Adjust deadlines and attendance in one course Exam deferral or formal claim Exams office; extenuating circumstances or special consideration panel Defer assessments; set aside affected results Drop, incomplete, withdrawal or leave Registrar; academic adviser; dean Process late changes; advise on consequences Ongoing accommodations for a mental health condition Disability or accessibility services Formal, term-long adjustments with documentation Money, aid or scholarship problems Financial aid; scholarships office Appeals; review of changed family finances Visa or enrolment status International student office Advise on reduced load and travel One last point. Many students hesitate because they fear that using these processes will leave a mark on their record. In general, the opposite is true. A documented bereavement with an extension or an incomplete leaves little trace. A term of failed courses, without explanation, leaves a far larger one. The system is designed, imperfectly, to allow you to grieve without that grief becoming a permanent academic penalty. Using it is not special pleading. It is what it is there for. Chapter 5: Oscillation: Studying While Grieving Bereaved students often describe a painful double bind. When they study, they feel guilty for not grieving, as though an hour spent on problem sets is an hour of forgetting the person who died. When they grieve, they feel anxious about falling behind, as though every tear costs them a grade. The result is that neither activity is done wholeheartedly, and both feel like failures. The most useful framework in contemporary grief research for escaping that bind is the dual process model of coping with bereavement, developed by Margaret Stroebe and Henk Schut and first set out in the journal Death Studies in 1999. It is, among other things, a theory of why grieving and getting on with life are not opposites. This chapter explains the model and then applies it to the ordinary, concrete problem of studying while you mourn. The dual process model Older theories of grief, influenced by psychoanalysis, tended to emphasise "grief work": the idea that the bereaved person must confront the loss, work through painful feelings and memories, and gradually detach from the person who died, and that avoiding this work was unhealthy. There is truth in part of this. Many people do need, at some point, to face the reality of a death and feel its pain. But the grief work idea had two problems. It did not match how many healthy people actually grieve, since plenty of people adapt well without extended confrontation, and it gave no account of the other enormous task of bereavement: living in a changed world. Stroebe and Schut proposed that bereaved people deal with two broad kinds of stressor. The first is the loss itself, and they called activities focused on it loss-oriented: crying, yearning, remembering, looking at photographs, visiting the grave, thinking about the relationship, feeling the absence. The second is everything that changes as a consequence of the death, and they called activities focused on these changes restoration-oriented: taking on new roles and responsibilities, dealing with practical tasks, building a life that does not include the person, forming new relationships and identities, and also, importantly, taking time off from grief through distraction, rest and ordinary activity. The central idea of the model is oscillation. Healthy grieving, in this view, involves moving back and forth between the two orientations. Sometimes you confront the loss; sometimes you set it aside and attend to life. Sometimes you are overwhelmed; sometimes you are absorbed in something else entirely. This oscillation is not a failure to grieve properly. It is how people regulate a process that would be unbearable if faced continuously, and how they adapt to a world that has changed. Staying permanently in loss orientation would mean endless, exhausting confrontation with pain. Staying permanently in restoration orientation would mean avoiding the loss altogether. Moving between them allows people to grieve in doses. The model makes a few further points that are useful for students. First, both orientations involve both approach and avoidance. Distraction from grief is a legitimate part of the process, not a sign of denial, unless it becomes total and rigid. Second, people differ in the balance that suits them. Some people, and some cultures, lean more toward expression and confrontation; others toward activity and restraint. There is no single ideal ratio. Third, the model has been extended by its authors to acknowledge that bereaved people can become overloaded: when the demands of both orientations pile up at once, as they do when a student is grieving intensely while also facing exams, family responsibilities and financial problems, the result can be exhaustion rather than oscillation. Reducing the load, not simply coping harder, is sometimes the only real answer. Why this matters for a student Seen through the dual process model, studying is not the opposite of grieving. It is one of the main forms that restoration orientation takes in a student's life. When you sit in the library for two hours and become absorbed in an essay, you are not betraying the person who died; you are doing one half of the work of bereavement. When you take an evening to cry, look at photographs and talk to your sibling on the phone, you are not wasting study time; you are doing the other half. The model also explains why neither extreme works well. Students who throw themselves entirely into their work, refusing to let grief in, often find it breaking through anyway, in the middle of an exam or at three in the morning, or find themselves months later in a delayed collapse. Students who withdraw entirely from their studies and spend every day immersed in the loss often find that the grief grows heavier rather than lighter, and that the absence of structure, purpose and company makes everything worse. Most people do better with some of each. And the concept of overload explains why grieving students so often feel they are failing when they are not. If your grief is intense, your family needs you, you are dealing with practical tasks at home, and you are trying to maintain a full course load, you may simply be carrying more than a person can carry. In that situation the problem is not your coping. It is the load. That is the strongest argument for the accommodations discussed in Chapter 4: they reduce the load so that oscillation becomes possible again. Designing a grieving term How can you use these ideas in practice? The core principle is to make room for both orientations deliberately, rather than letting grief ambush your study and study crowd out your grief. The following suggestions are not rules, and not all of them will suit everyone. They are tools, drawn from the model and from what many bereaved students have found helpful. Give grief a place in the day. Some people find it helps to set aside a regular time, perhaps twenty or thirty minutes in the evening, for loss-oriented activity: writing to the person, looking at photographs, listening to their music, crying, calling family. This does not mean grief will only arrive at that time; it will not. But knowing that there is a time for it can make it easier to set it aside during a lecture, because you know you are postponing it rather than suppressing it. Some students find this idea strange or artificial at first and then discover that it works. Study in shorter blocks. Grief affects concentration, and it is often more realistic to aim for several focused periods of thirty to fifty minutes than for long sessions. Start with tasks that require less deep thinking, such as organising notes, reviewing flashcards, reading a familiar text, or attending a problem class, and build up to demanding work as your capacity returns. Expect your capacity to vary from day to day. Externalise your memory. If you are forgetting deadlines, losing track of readings or walking into lectures without knowing what the topic is, do not rely on your memory for a while. Write everything down, use a calendar with reminders, and keep a single list of what is due and when. This is not a character flaw; it is an adjustment to a temporary impairment, like using a lift while your ankle heals. Study with other people. Working in a library, a study group or simply alongside a friend can make study easier, because company provides structure and a gentle form of accountability, and because it is a relief from being alone with your thoughts. It also keeps you connected at a time when withdrawal is tempting. Plan for grief in public. Grief will sometimes arrive in the middle of a lecture, a seminar or an exam. It helps to decide in advance what you will do. You might sit near the door so that you can step out without drawing attention. You might tell one instructor or classmate in advance that you might need to leave. You might carry a small reminder of a slow-breathing routine or a few grounding steps, such as noticing five things you can see, four things you can hear and so on, that you can use until the wave passes. For exams, ask whether you can sit in a smaller room, or whether there is a procedure if you become distressed during the paper. Anticipate triggers. Some reminders are predictable: the person's birthday, the anniversary of the death, holidays, Mother's or Father's Day, a course that touches on the illness that killed them, a novel on the syllabus about a death. Look ahead through your term calendar and your syllabi for these. If a date or topic is likely to be difficult, it may be worth warning an instructor in advance or arranging a lighter workload around that time. Chapter 7 returns to anniversaries. Protect sleep. Sleep is where much of the brain's work of memory consolidation happens, and it is one of the first things grief disrupts. Chapter 6 discusses practical steps; here it is enough to say that no study strategy compensates for sustained sleep loss, and that protecting sleep is itself a form of academic work. Use the academic support that already exists. Most universities have writing centres, tutoring services, study skills advisers, peer mentors and office hours, all of which are underused by students who are struggling. A bereaved student who has missed three weeks of a mathematics course may find that two sessions with a tutor recover more ground than twenty hours of solitary reading. You do not need to explain your bereavement to use these services, though it can help to mention it so that the tutor understands why your concentration is uneven. Lower the bar, temporarily and on purpose. Many high-achieving students find that grief collides with their standards. They are used to working at full capacity and producing their best, and they experience anything less as failure. For a term, it may be wiser to aim for "good enough": to submit an essay that is sound rather than brilliant, to pass a course you would normally have aimed to excel in. This is not giving up. It is a deliberate allocation of limited resources, and it is usually temporary. Allow real respite. The dual process model insists that time off from grief is necessary. That means not only study but also fun, rest, laughter, sport, time with friends, films and music. Bereaved students often feel that enjoyment is disloyal, especially in the early months. It is not. It is one of the ways the mind makes grief bearable. Consider, as an illustration, a second-year history student whose father died in the first month of the autumn term, and who has chosen, after talking with her adviser, to stay enrolled with a slightly reduced load and extensions on two early essays. In the first weeks after she returns, her days are chaotic: she wakes at four, cannot read for more than ten minutes, cries unexpectedly in the dining hall, and spends evenings scrolling through photographs until the early hours. With a counsellor's help she tries a different pattern. Mornings, when her concentration is best, are for study, in forty-minute blocks in the library with a friend from her seminar group. Afternoons are for classes and lighter tasks. Early evening is for a walk, often while phoning her mother or brother. After dinner she spends half an hour on what she calls "Dad time": writing in a notebook addressed to him, looking at photographs, sometimes crying. Then she puts the notebook away and watches something with her housemates, or reads a novel. She sets her phone to charge outside her bedroom. The pattern does not stop her grief, and there are many days when it collapses entirely. But over several weeks she notices that she can study more, that the evening "Dad time" is something she looks forward to rather than dreads, and that she has stopped feeling guilty about the hours she spends not thinking about him. She has, in the language of the model, learned to oscillate. By the end of term she submits all her work, most of it on the extended deadlines, and she carries the pattern, in looser form, into the following term. A contrasting case shows the opposite imbalance. Consider a final-year engineering student whose younger sister dies in a car crash over the winter break. He returns to campus in January determined not to let it affect his degree. He declines the dean's offer of extensions, takes on an extra project, works late every night and spends weekends in the lab. He tells friends he is fine and changes the subject when they ask. For two months his grades are excellent. Then, in March, after a friend mentions a holiday his family took together years ago, he finds he cannot stop crying, cannot get out of bed for three days, and misses a major deadline. He has been living almost entirely in restoration orientation, using work to keep the loss at bay, and the loss has broken through. With a counsellor's help he begins, slowly, to let grief in on purpose, in small, planned doses, rather than waiting for it to overwhelm him. He also accepts, belatedly, the extension he had turned down. His story is common, and it is not a story of weakness. It is a demonstration of why oscillation, rather than avoidance, is the more sustainable pattern. Neither of these stories is a formula. The first shows one student's version of oscillation; the second shows what happens when it is missing. Your own version might look completely different. Some people prefer to grieve in the morning and work at night; some grieve best in motion, running or walking, and some in stillness; some prefer to grieve with others and some alone. What matters is not the specific timetable but the underlying idea: a term that has deliberate room for both grief and life, and a willingness to move between them without treating either as a failure. When oscillation stops The model also offers a way to notice when something is going wrong. If, months after a loss, you find yourself stuck at one pole, that may be a sign that you need more support. Being stuck in loss orientation might look like being unable to think about anything but the death, spending most of each day ruminating on the circumstances or on what you could have done differently, being unable to engage with study or friends at all, and feeling no relief from the pain. Being stuck in avoidance might look like being completely unable to tolerate any reminder of the person, organising your whole life to avoid their name, photographs, places associated with them or conversations about them, and feeling that if you let grief in at all it would destroy you. Neither of these is a moral failure. Both are recognised patterns in people whose grief becomes prolonged, and research on prolonged grief has identified both persistent rumination and extensive avoidance as factors that tend to keep grief stuck. Treatments for prolonged grief, discussed in Chapter 8, are in part designed to restore oscillation: helping people approach what they have been avoiding and re-engage with the life they have withdrawn from. If you recognise yourself in either description, especially if it has persisted for many months, that is a good reason to speak to a counsellor or doctor. For most bereaved students, though, the lesson of the dual process model is gentler. You do not have to choose between grieving and studying. You can do both, not at the same moment, but in alternation. And the moments when you forget, briefly, that the person is gone are not lapses. They are part of how you will learn to carry them. Chapter 6: Coping That Helps and Coping That Costs Every bereaved person copes somehow. The question is not whether you will develop ways of getting through the days, but which ways, and at what price. Some coping strategies ease pain in the short term and also support recovery in the longer term. Others ease pain now and add to it later. Many sit somewhere in between, helpful in moderation and harmful in excess. This chapter does not offer a programme. Grief is not a condition to be managed with a checklist, and people differ widely in what helps them. It offers instead a way of thinking about coping that is honest about trade-offs, and a set of specific practices that research and clinical experience suggest are useful for many people. It pays particular attention to the student context, where some of the most readily available ways of numbing pain, especially alcohol, are woven into social life. Table 4, at the end of the chapter, summarises many of these trade-offs in compact form. The underlying principle is simple: good coping allows you to oscillate, in the sense described in Chapter 5, between facing the loss and living your life. Poor coping tends to lock you into one or the other, or to create new problems that add to the load. The body grieves too: sleep, food and movement Grief is a physical experience, and the body's needs do not pause for mourning. Students often neglect this because it seems trivial next to the enormity of the loss. It is not trivial. Poor sleep, poor nutrition and physical inactivity each worsen mood, concentration and resilience, and together they can turn manageable grief into something much harder to bear. Sleep. Grief commonly disrupts sleep: difficulty falling asleep because the mind will not quiet, waking in the early hours, vivid or distressing dreams, or sleeping far more than usual. Student life adds its own disruptions: irregular schedules, late nights, shared rooms, screens. A few steps help many people. Keep a roughly regular time for getting up, even if the night was bad, because a regular wake time is one of the strongest anchors of the body clock. Build a short wind-down routine before bed that does not involve reviewing photographs, reading old messages or scrolling social media, since these tend to reactivate grief just when you want it to settle. If you wake in the night and cannot sleep, it often helps to get up for a while, do something quiet in dim light, and return to bed when sleepy, rather than lying awake ruminating. Be cautious about daytime napping that becomes a substitute for night sleep. If poor sleep persists for more than a few weeks and is affecting your functioning, speak to your campus health service. Cognitive behavioural therapy for insomnia, a structured, short-term treatment, is widely recommended as a first-line approach for persistent insomnia, and some services offer it or can refer you. Sleep medication has a role in some circumstances, but it should be discussed with a doctor rather than borrowed from a friend. Food. Many bereaved people lose their appetite for a while; others eat for comfort or at irregular times. Neither is a disaster in the short term. But weeks of skipped meals, or of living on snacks and caffeine, take a toll on energy, mood and concentration. Aim for small, regular meals rather than perfect nutrition. If cooking feels impossible, this is an ideal task to hand to friends: a housemate who says "I'm making dinner, I'll leave you a plate" is providing real support. If you have a history of disordered eating, be aware that grief and loss of routine can reactivate it, and seek help early if you notice old patterns returning. Movement. Physical activity is one of the more reliable supports for mood, sleep and stress in general, and many bereaved people find it helpful. It does not need to be strenuous. A daily walk, especially outdoors and in daylight, is enough to make a difference for many people, and walking with a friend combines movement with company. Some people find that exercise becomes a place for grief, a time to think about the person or to cry without anyone watching; others use it as respite. Both are useful. The aim is movement that feels restorative. Grief is not something to be punished out of the body, and there is no evidence that pushing yourself into pain or discomfort helps you mourn. If you notice that you are exercising compulsively, to exhaustion, or in a way that feels like punishment, that is worth discussing with a counsellor. Physical health generally. Grief places the body under stress, and bereaved people sometimes neglect their own medical care: missing appointments, stopping medications, ignoring symptoms. Keep taking any regular medication, and if you have a chronic condition such as diabetes, asthma or a mental health condition, let your clinician know about the bereavement. If you experience chest pain, breathlessness or other alarming physical symptoms, seek medical attention promptly rather than assuming they are only grief. Alcohol, cannabis and other ways of numbing On many campuses alcohol is the default social lubricant and the default response to bad news. "Let's get you a drink" is a common, well-meant reaction to hearing that a friend's father has died. For some students, occasional moderate drinking in the company of friends is part of how they reconnect with ordinary life after a loss. But alcohol and grief are an uneasy combination, and it is worth being clear about why. Alcohol is a depressant. It may lift mood briefly, but as it wears off it tends to lower mood, increase anxiety and disrupt sleep, particularly the later part of the night. Heavy drinking can release emotions in an uncontrolled way, turning a night out into a night of distress, conflict or risk. It impairs judgement at a time when judgement is already stretched. It can become a way of avoiding grief altogether, which, as Chapter 5 described, tends to postpone rather than remove the pain. And in people who are struggling with low mood or suicidal thoughts, alcohol increases risk, because it lowers inhibitions and increases impulsivity. Research on bereavement has linked loss with increased alcohol use in some people, and bereaved people with pre-existing drinking problems are particularly vulnerable. Cannabis carries similar trade-offs. Some students use it to sleep or to calm distress, and it may help in the short term. But regular use can worsen mood and anxiety for some people, interfere with motivation and memory, and become a way of not feeling. Other substances, including stimulants used to compensate for lost study time and sedatives or opioids used to blunt pain, carry more serious risks, including dependence and overdose. None of this is a moral argument. It is a practical one. If you find yourself drinking or using substances more than usual since the death, it is worth asking honestly whether it is helping you oscillate or helping you avoid. Some questions that can help: Am I drinking or using alone? Am I using it to get to sleep most nights? Am I doing things I regret? Have friends expressed concern? Do I feel I need it to face the day? If the answers trouble you, campus counselling and health services see this pattern often and can help without judgement. Consider a sophomore whose closest friend from home dies of an overdose in the spring. His friends on campus, wanting to help, take him out the weekend he returns, and for several weeks afterwards the weekends become a blur of heavy drinking, with long, tearful nights and mornings of hangover and shame. During the week he drinks alone in his room to get to sleep. He tells himself it is what everyone does. It is only when his resident adviser finds him distressed and very drunk on a Tuesday night that he begins to see a pattern. In counselling, he realises that drinking has been the only place where he lets himself feel the loss, and that it has also stopped him grieving in any other way. With support he cuts back sharply, starts meeting a friend for a walk on the evenings he used to drink, and joins a campus group for students bereaved by overdose and suicide. His grief does not diminish quickly. But it becomes something he can carry awake and sober, rather than something he visits only when drunk. The story is illustrative, but the pattern is familiar to anyone who works in campus counselling. It is also a reminder that friends play a large part in how a grieving student copes. If you are the friend, suggesting a meal, a film or a walk instead of a night out, at least some of the time, is a real kindness. There are also quieter forms of numbing that deserve a mention: endless scrolling, gaming through the night, bingeing television, working compulsively. In moderation these are ordinary forms of respite. They become costly when they crowd out sleep, contact with people and any engagement with the loss at all. People, words and kindness toward yourself If there is one finding that runs consistently through research on stress and bereavement, it is that social connection matters. Not every form of support helps, and not every relationship can bear grief, but people who have somewhere to turn tend to fare better than people who are isolated. For students, the challenge described in Chapter 2 is that the people who understand the loss are often far away and the people nearby often do not understand it. A useful approach is to think about support as several different things, which different people can provide. Some people are good at practical help: meals, lifts, notes from missed lectures. Some are good at company: sitting with you, watching a film, walking to class. Some are good at listening: letting you talk about the person without trying to fix anything. Some share the memory of the person and can talk about them with you. Rarely does one person provide all of these. It is reasonable to ask different friends for different things, and to accept that some well-meaning people cannot provide any of them. Staying in regular contact with family can help too, though family relationships after a death can be complicated. Grief can bring families closer, and it can also expose tensions: over the funeral, over money, over who was closest, over how to mourn. You are not obliged to carry your family's grief for them, and it is reasonable to set limits if contact is consistently distressing. A counsellor can help you think this through. Talking to a professional. You do not need to be in crisis to see a counsellor. Many bereaved students benefit from a few sessions early on, simply to have a space where the loss is the whole subject. Research on grief counselling suggests that it is most clearly beneficial for people whose grief is severe or complicated, and less clearly beneficial as a routine intervention for everyone who has lost someone. That is a reason not to feel obliged to seek counselling, but it is not a reason to avoid it if you want it. If you find you are struggling to function, feeling increasingly isolated, or experiencing any of the signs described in Chapter 8, counselling becomes more clearly important. Writing. Many people find it helps to write about the loss: in a journal, in letters addressed to the person who died, in poetry, or in more structured exercises. The psychologist James Pennebaker's research on expressive writing showed benefits of writing about emotional experiences in a range of groups, though studies of structured writing with bereaved people in particular have produced mixed and generally modest results. The practical conclusion is that writing is worth trying if it appeals to you, not a treatment you should force yourself through. If writing about the death leaves you more distressed day after day rather than eventually calmer, it may be better to put it aside for a while or to do it with a counsellor. Self-compassion. Grieving students are often remarkably hard on themselves. They berate themselves for not studying enough, for not grieving enough, for grieving too much, for things they said or did not say to the person before they died. The psychologist Kristin Neff's work on self-compassion describes an alternative stance: treating yourself with the same kindness you would offer a friend in the same situation, recognising that suffering and imperfection are part of shared human experience, and noticing painful thoughts without being consumed by them. A simple question can open this stance: "If my best friend were in my position, what would I say to them?" Most students would never tell a grieving friend that they are lazy or that their grief is excessive. They say it to themselves constantly. Guilt deserves special attention, because it is one of the most common and most painful features of grief. You may feel guilty for not being there when they died, for not visiting more often, for an argument, for not noticing they were ill, for moving away to university in the first place. Some of this guilt reflects genuine regret and is part of loving someone imperfectly. Much of it reflects hindsight: judging past decisions by knowledge you did not have at the time. It can help to ask what you knew then, what you could reasonably have done with that knowledge, and what the person who died would say if they could hear your self-accusation. If guilt becomes intense and persistent, especially after a suicide or an accident, it is a strong reason to seek professional help. Anything that hurts you. Occasionally, bereaved people find themselves wanting to hurt themselves physically: to feel something, to punish themselves, or to make internal pain visible. If you notice this urge, please treat it as a sign that you need more support than you currently have, and talk to a counsellor, doctor or crisis service. There are always better ways to carry pain than by adding to it. Table 4 brings together the main coping responses discussed in this chapter, with a note on when each tends to help and when it tends to cost more than it gives. Table 4. Common ways of coping with grief: when they help and when they cost. Coping response Tends to help when Tends to cost when Keeping busy with study or work It offers structure and respite It becomes a way never to feel the loss Talking about the person It is with people who can listen It is forced, or only with people who dismiss it Looking at photographs or messages It is chosen and time-limited It fills the night and replaces sleep Alcohol or cannabis Rarely; at most light, social and occasional It is used to sleep, alone, or to avoid grief Exercise and walking It restores energy and mood It becomes compulsive or punishing Writing It helps order thoughts and feelings It leaves you more distressed day after day Withdrawing to be alone It is brief and restful It becomes isolation lasting weeks No one copes perfectly. You will have nights of too much wine, days lost to scrolling and weeks of poor sleep. The point of thinking about coping is not to judge yourself for these, but to notice patterns, and to tilt gently, over time, toward the forms of coping that let you carry your grief and live your life at the same time. Chapter 7: Continuing Bonds and the Search for Meaning A student whose mother has died keeps her voicemail greeting and listens to it before exams. Another wears his grandfather's watch to every interview. A third talks aloud to her brother on the walk to the library, telling him about her day. A fourth has taken up running because her father ran, and thinks about him on every long run. None of these students has "let go". All of them, in the language of contemporary grief research, are maintaining a continuing bond. For much of the twentieth century, such behaviour was viewed with suspicion by professionals. This chapter explains why that view changed, what continuing bonds look like, when they help and when they may signal difficulty, and how rituals, anniversaries and the search for meaning fit into a student's life after loss. From letting go to holding on differently The older view has a long history. In his 1917 essay "Mourning and Melancholia", Sigmund Freud described mourning as a process in which the bereaved person gradually withdraws emotional energy from the lost person so that it can be invested elsewhere. Healthy grief, in this account, ended in detachment. Later clinicians built on this idea, and for decades the goal of grief counselling was often framed as helping people to "let go", "move on" or achieve "closure". Continuing to talk to the dead, to feel their presence or to keep their possessions close was sometimes interpreted as a failure to accept reality. In 1996, Dennis Klass, Phyllis Silverman and Steven Nickman published an edited volume, Continuing Bonds: New Understandings of Grief, that challenged this view directly. Drawing on studies of bereaved parents, widows, children who had lost a parent, and on anthropological evidence from many cultures, they argued that maintaining an ongoing relationship with the dead is common, often healthy, and not incompatible with accepting the death. Silverman and Nickman's research on children who had lost a parent, for example, found that many children maintained a sense of connection with the dead parent, locating them in heaven, talking to them, keeping their belongings, or trying to live up to their values, and that this connection seemed to help rather than hinder their adjustment. The anthropological evidence was equally striking: in many cultures, from ancestor rituals in Japan to the Day of the Dead in Mexico, the relationship with the dead is explicitly maintained and publicly honoured. What the older Western model had treated as pathology was, in much of the world and in many private lives, simply normal. The continuing bonds perspective reframes the task of grief. The aim is not to sever the relationship but to transform it: from a relationship with a living person, present in the flesh, to a relationship with someone who is absent but still part of your inner world. The person is no longer here to talk to, but you can still carry their voice, their humour, their advice, their values and their love. Many people find that over time the relationship settles into a form they can live with, less raw, less constantly present, but still real. Continuing bonds take many forms. Some are internal: remembering, imagining what the person would say, feeling guided by their values, sensing their presence. Some are expressed in objects: keeping clothes, jewellery, letters, recordings, a favourite mug. Some are expressed in action: taking up their hobbies, cooking their recipes, supporting causes they cared about, living in a way they would have been proud of. Some are spiritual or religious: prayer, belief in an afterlife, rituals on the anniversary. And some are social: telling their stories, keeping in touch with their friends, naming a child after them years later. The research does not suggest that every form of bond is always helpful. Studies by Nigel Field and colleagues, among others, have suggested that the effects of continuing bonds depend on their form and on the person's circumstances. Bonds that involve internalising the person's values and memory, and that coexist with a clear acknowledgement that they have died, tend to be associated with better adjustment. Bonds that reflect an ongoing inability to accept the death, such as persistent searching, an insistence that they will return, or keeping their room untouched for years as a shrine that must not be disturbed, may be a sign that grief has become stuck. Early in grief, many people do things that would look strange years later, and that is normal. The question is not whether you keep a connection, but whether the connection helps you live. Continuing bonds are not only a matter for loving relationships. Some students grieve a parent who was absent, harsh, addicted or abusive, or someone with whom they had been in conflict for years. Here the idea of a continuing bond can seem either irrelevant or unbearable. Yet the relationship does continue, in the sense that it remains part of your inner world and your history, and grief after a difficult relationship is often especially confusing. You may grieve not only the person but the relationship you never had and now never will, the apology that will not come, the reconciliation that is no longer possible. You may feel relief and sorrow, anger and longing, all at once. Others may not understand why you are so affected, or why you are not affected more. There is no obligation to idealise the dead, and no obligation to maintain a bond that harms you. For some people the healthiest form of continuing relationship is a clearer understanding of what happened and how it shaped them, perhaps worked through with a counsellor, rather than a warm internal presence. That, too, is a way of carrying the relationship forward on terms you can live with. Consider, as an illustration, a graduate student whose father, from whom she had been estranged for five years, dies suddenly. She had not planned to reconcile, and yet the news leaves her unable to work for weeks. She is angry at him, angry at herself for being upset, and embarrassed to ask for time off for a man she had not spoken to in years. A counsellor helps her see that she is grieving several things at once: the father she had as a small child, the father she wished she had, and the possibility, now closed, that things might one day have changed. Her supervisor, once told the outline of the situation, grants her a month's flexibility without asking for details. Over the following year she writes a long letter to him that she never sends, and finds that it allows her to put down some of what she had been carrying. Her bond with him is not a warm one. But it is no longer a silent weight. For students, there is often a specific question about possessions. You may have inherited objects, or be involved in clearing the person's home, and feel pressure to decide quickly what to keep. If you can, keep more rather than less in the early months, especially small, portable things that hold memory, and postpone decisions about the rest. Many people regret discarding things too soon; few regret keeping a box in a relative's attic for a year. Rituals, anniversaries and the academic calendar Rituals are among the oldest human responses to death, and one of their functions is to give shape to what is otherwise shapeless. A funeral marks the death. A memorial gathering brings together those who remember. Religious calendars prescribe periods of mourning and moments of remembrance: in Judaism, the week of shiva and the annual yahrzeit; in many Christian traditions, memorial masses and All Souls' Day; in Islam, prescribed prayers and customs around the period of mourning; in many Hindu traditions, rites continuing for days after cremation and annual observances; in Chinese traditions, Qingming, when families visit and tend graves. Secular families create their own rituals: an annual walk, a meal of the person's favourite food, a gathering on their birthday. For students away at school, ritual is often disrupted. You may miss the rites at home, or find that your family's customs have no equivalent on campus. It is worth knowing that you can create rituals of your own, and that small ones count. You might light a candle on the anniversary, cook their recipe for your housemates, visit a place of worship on campus, go for a walk they would have enjoyed, write them a letter each year on their birthday, or play their favourite song and tell a friend why it mattered. Campus chaplaincies and multifaith centres can often help students observe mourning rites far from home, and some universities hold annual memorial services for members of the university community who have died, and sometimes for students' family members too. Anniversaries deserve particular planning, because they tend to arrive with more force than people expect. Many bereaved people experience a surge of grief in the days around the anniversary of the death, and often in the weeks leading up to it, sometimes before they have consciously noticed the date approaching. Birthdays, holidays and family occasions have a similar effect, especially in the first year, which grief counsellors sometimes call the year of firsts: the first birthday without them, the first holiday season, the first Mother's Day. For students, the academic calendar adds a particular twist. Because a death is often tied in memory to the term in which it happened, the anniversary may fall in the same part of the academic year: the same exam period, the same stretch of midterms, the same week of the spring term. A student whose father died in the week before finals may find that the following year's finals bring back not only the grief but the whole atmosphere of those days. It is worth looking ahead and noting difficult dates in your calendar at the start of each term. If an anniversary coincides with an assessment, it is reasonable to raise it with the instructor or the dean's office in advance, just as you would with any other foreseeable circumstance. Many students do not realise that the anniversary of a death can be a legitimate reason for a modest accommodation, and many instructors are sympathetic if asked early. Graduation raises the same issues more sharply. It is, for many families, a moment when the absence of the person is felt intensely: an empty seat, a photograph that cannot be taken, a celebration shadowed by loss. Some students choose to mark the person's presence explicitly: wearing something of theirs, decorating a cap, mentioning them in a speech, sitting with a family member who knew them. Others prefer to keep it private. There is no right way. But it helps to anticipate that the day may be mixed, and to give yourself permission for both joy and sorrow. Meaning, identity and what the loss asks of you Beyond the question of the relationship with the dead lies a broader one: what does this death mean for the life you are living? The psychologist Robert Neimeyer and his colleagues have argued that grief is, in large part, a process of meaning reconstruction. A significant death can disrupt a person's sense of how the world works, what is fair, what is safe, who they are and where their life is going. Part of adapting is rebuilding a story that can hold the loss. Research in this tradition has identified several aspects of meaning-making. One is sense-making: finding some way to understand why the death happened or how it fits into your understanding of life. For some people this comes through religious faith, for others through a scientific understanding of illness, for others through philosophical acceptance of mortality. Another is finding some benefit or growth in the experience, which does not mean being glad the person died, but noticing, for example, that the loss has deepened your relationships or clarified your priorities. A third is identity change: coming to understand how you have been altered by the loss, and who you are now. Difficulty making sense of a death appears to be one of the factors associated with harder grief. In a study of more than a thousand recently bereaved college students, Joseph Currier, Jason Holland and Robert Neimeyer found that students bereaved by violent deaths, such as accidents, suicides and homicides, reported more symptoms of complicated grief, and that this association was explained by greater difficulty making sense of the loss. For students grieving a sudden or violent death, this suggests that the struggle to understand is not incidental. It is often at the heart of the grief, and it may be worth addressing directly, with a counsellor, a faith leader or others who have faced similar losses. It is important not to turn meaning-making into another demand. Popular culture sometimes insists that suffering must produce growth, that every loss has a lesson, that tragedy happens for a reason. Many bereaved people find such statements offensive, and rightly so. The concept of post-traumatic growth, developed by Richard Tedeschi and Lawrence Calhoun, describes positive changes that some people report after major adversity, but it was never meant to suggest that growth is required, or that failing to find it is a failure. Some deaths make no sense and never will. Meaning, if it comes, tends to come slowly and cannot be forced. For students, meaning-making often becomes entangled with choices about the future. The death of a parent may prompt questions about whether to continue on a path they encouraged, or whether you chose it for them rather than yourself. The death of a sibling from an illness may prompt an interest in medicine or research. The death of a friend by suicide may lead to involvement in mental health advocacy on campus. These can be profound and authentic developments, and many people build lasting commitments out of loss. But as Chapter 3 suggested, it is wise to let such decisions ripen rather than making them in the first raw months. Changing your major, choosing a dissertation topic about the illness that killed your mother, or dedicating yourself to a cause can be ways of honouring the dead and also, sometimes, ways of staying so close to the loss that there is no room for anything else. A useful test is whether the choice feels like an expansion of your life or a narrowing of it. Identity questions also arise more quietly. Who are you now that you are someone whose father has died? How do you answer the question "How many siblings do you have?" when one of them is dead? What do you say when someone asks what your parents do? These questions come up constantly in student life, often from strangers, and there is no single right answer. Some people always mention the person; some decide case by case; some develop a short answer for casual contexts and a fuller one for friends. All are legitimate. You are allowed to protect yourself in a seminar icebreaker, and you are allowed to speak about the person proudly when you choose. In the end, the continuing bonds perspective and the meaning-making perspective point in the same direction. The goal of grief is not to return to the person you were before the death, as if it had not happened, nor to remain frozen in the moment of loss. It is to become someone who carries the loss and the relationship forward into a life that continues. That process takes much longer than a semester. It is not finished when you graduate. But it begins in small acts, in the watch worn to an interview, the voicemail before an exam, the recipe cooked for friends, and those acts are available to you now. Chapter 8: When Grief Becomes a Crisis Most of this book has emphasised that grief, however painful, is normal. It varies enormously, it moves in waves, it does not follow stages, and for most people it gradually becomes more bearable without professional treatment. All of that remains true. But it is also true that for some people, grief becomes something more: a persistent condition that disables them for months or years, a depression that deepens rather than lifts, a traumatic reaction to the circumstances of the death, or a crisis in which they are no longer safe. This chapter describes those possibilities clearly, so that you can recognise them in yourself or in a friend, and know what to do. If you are in danger right now, please do not read on. In the United States, call 911 if your life is at immediate risk, or call or text 988 to reach the 988 Suicide and Crisis Lifeline, which is available at any hour. In Canada, you can also call or text 9-8-8. In the United Kingdom and Ireland, Samaritans can be reached free on 116 123 at any hour; for an emergency in the UK call 999, and in Ireland and across the European Union, 112. In Australia, call 000 in an emergency or Lifeline on 13 11 14. Elsewhere, contact your local emergency number, go to the nearest hospital emergency department, or call your campus security or counselling service, which can connect you with help at any hour. Services and numbers change over time, and your university's website will list current local options. Prolonged grief disorder For most bereaved people, the intensity of grief diminishes over months, even if it never disappears. For a minority, it does not. Months or years later, they remain consumed by longing for the person, unable to accept the death, cut off from other people and from any sense of purpose, and unable to function in work, study or relationships. Clinicians have described this pattern under various names over the years, including complicated grief and traumatic grief. It is now formally recognised as prolonged grief disorder, first in the World Health Organization's International Classification of Diseases, 11th revision (ICD-11), and then, in 2022, in the American Psychiatric Association's DSM-5-TR. The two systems define the disorder in slightly different ways, but the core is the same. In DSM-5-TR, prolonged grief disorder can be diagnosed in adults when at least twelve months have passed since the death, and the person has experienced, nearly every day for at least the past month, intense yearning for the deceased or preoccupation with thoughts or memories of them, together with at least three further symptoms from a list of eight. These are: a disruption of identity, such as feeling that part of oneself has died; marked disbelief about the death; avoidance of reminders that the person is dead; intense emotional pain, such as anger, bitterness or sorrow, related to the death; difficulty reintegrating into relationships and activities; emotional numbness; feeling that life is meaningless; and intense loneliness. The disturbance must cause significant distress or impairment, and it must exceed what would be expected in the person's cultural and religious context. ICD-11 uses a shorter minimum period of six months, with a similar emphasis on persistent, pervasive longing or preoccupation, intense emotional pain and functional impairment beyond cultural norms. It is important to be precise about what these criteria mean. They do not mean that grief lasting a year is abnormal; many people still grieve deeply after a year, and continue to miss the person for the rest of their lives. They do not mean that intense grief in the first months is a disorder. And they are not a checklist for self-diagnosis. They describe a pattern in which grief has become persistently disabling, and they exist so that people in that pattern can be identified and helped. How common is it? A 2017 meta-analysis by Marie Lundorff and colleagues, pooling studies of adults bereaved by non-violent deaths, estimated that roughly one in ten such adults experienced prolonged grief disorder, with the caution that the underlying studies varied considerably in method. A later meta-analysis by Manik Djelantik and colleagues, focusing on deaths by accident, suicide, homicide and disaster, produced a much higher pooled estimate, close to half of bereaved adults, though again with considerable variation between studies and a caution that the figures should be interpreted carefully. Whatever the precise numbers, the pattern is consistent: most bereaved people do not develop prolonged grief disorder, but a meaningful minority do, and the risk is substantially higher after sudden and violent deaths. Other factors associated with higher risk include the death of a child or partner, a very close or dependent relationship with the person who died, a history of mental health problems, and a lack of social support. The good news is that prolonged grief disorder is treatable. Several forms of psychotherapy designed specifically for it have been tested in randomised controlled trials. The best known is complicated grief treatment, developed by the psychiatrist M. Katherine Shear, which combines elements of cognitive behavioural therapy and interpersonal therapy with techniques drawn from trauma treatment, and is explicitly built around the dual process model: it helps people both to confront the reality of the death and to re-engage with their lives. Cognitive behavioural approaches developed by Paul Boelen and others have also shown benefit. These grief-focused treatments appear to be more effective for prolonged grief than general supportive counselling. Medication is not a primary treatment for prolonged grief itself, though antidepressants can be important for treating depression that occurs alongside it. For a student, the practical message is this. If a year or more after a death you find that grief still dominates almost every day, that you cannot accept that the person has died, that you have withdrawn from friends and studies and that life feels meaningless without them, you are not failing at grief and you are not simply "a sensitive person". You may have a recognised condition that responds to specific treatment. Ask your campus counselling service or your doctor whether they can assess you for prolonged grief, or refer you to someone who can. Depression, trauma and substance problems Prolonged grief disorder is not the only way grief can become complicated. Bereavement can also trigger or worsen other mental health conditions, and these can occur much earlier than the twelve-month window of the prolonged grief criteria. Depression. Grief and depression share many features: sadness, crying, poor sleep, low energy, difficulty concentrating, withdrawal. Until 2013, the American diagnostic manual contained a "bereavement exclusion" that discouraged diagnosing depression in the first months after a death. The DSM-5 removed that exclusion, on the grounds that major depression can and does occur after bereavement and should not go untreated simply because it followed a loss. Clinicians distinguish grief from depression partly by its focus and texture. In grief, painful feelings tend to come in waves, often triggered by reminders, and are focused on the person who died; self-esteem is usually preserved, and moments of warmth, humour and even pleasure are possible. In depression, low mood tends to be more pervasive and constant, pleasure is lost across almost all areas of life, and there is often a sense of worthlessness, self-loathing or hopelessness that extends beyond the loss. If you recognise the second pattern, especially if it persists for more than two weeks, speak to a counsellor or doctor. Depression is highly treatable, with psychotherapy, medication or both. Trauma. If the death was sudden and violent, if you witnessed it, found the person, or were involved in the circumstances, or if you learned of it in a way that was itself shocking, you may develop symptoms of post-traumatic stress: intrusive memories or images of the death, nightmares, intense distress at reminders, avoidance of anything connected with the event, feeling constantly on edge, being easily startled. Some of these reactions are common in the early weeks after a traumatic death and ease with time. If they persist for more than a month, or are severe from the outset, they may indicate post-traumatic stress disorder, which also has effective, trauma-focused treatments. Traumatic loss often combines trauma with grief in a way that makes each harder; clinicians who work with traumatic bereavement can address both. Substance problems. As Chapter 6 described, some bereaved people turn to alcohol or other drugs to manage pain, and for some this becomes a problem in its own right. If you are drinking or using most days, cannot cut down when you try, or are experiencing consequences that concern you or your friends, treat this as a health issue, not a moral failing, and seek help. These conditions often overlap. A student may have grief, depression and heavy drinking all at once, each worsening the others. Table 5 offers a simplified guide to distinguishing ordinary grief from the patterns that suggest more help is needed. It is a guide for recognising when to seek assessment, not a diagnostic tool. Table 5. Recognising when grief needs more help. Pattern What it often looks like Timing What to do Ordinary grief Waves of longing and sadness, triggered by reminders; moments of relief and connection Intense early; eases gradually, with returns Support, self-care, counselling if wanted Possible depression Pervasive low mood, loss of pleasure in everything, worthlessness, hopelessness Two weeks or more See a counsellor or doctor soon Possible traumatic stress Intrusive images of the death, nightmares, strong avoidance, constant alertness Persisting beyond about a month Seek trauma-informed assessment Possible prolonged grief Daily yearning or preoccupation, disbelief, meaninglessness, withdrawal, impaired functioning Twelve months or more (DSM-5-TR); six months (ICD-11) Ask for grief-specific assessment and treatment Crisis Thoughts of suicide or of joining the person, plans, feeling unable to stay safe Any time Contact emergency or crisis services now When you are not safe Grief can bring thoughts of death. Many bereaved people have fleeting thoughts such as "I wish I could be with her" or "I don't care what happens to me now". These thoughts are common, and on their own they do not mean that someone is going to act on them. But they deserve attention, and they sometimes become more serious. Thoughts of suicide are more likely when grief is combined with depression, hopelessness, heavy substance use, social isolation, a history of self-harm or previous suicide attempts, or when the person who died took their own life. That last point needs saying directly. People bereaved by suicide carry particular burdens: the shock, the unanswerable questions, the guilt, the stigma that silences others, and sometimes a sense that suicide has become thinkable in a way it was not before. A national survey of young adults in Britain, analysed by Alexandra Pitman and colleagues, found that those bereaved by the suicide of a relative or friend were more likely to attempt suicide themselves than young adults bereaved by sudden natural deaths. A further analysis of the same survey found that young adults who felt highly stigmatised by a sudden bereavement, of any kind, were more likely to report suicidal thoughts and attempts, which is one more reason why silence around a death can be dangerous and why being able to talk about it matters. These findings are not predictions about any individual, and most people bereaved by suicide do not attempt suicide. But they are a reason for anyone in that situation, and for their friends, to take support seriously and to seek it early. Organisations focused on suicide bereavement, such as the American Foundation for Suicide Prevention in the United States, offer support groups and resources specifically for survivors of suicide loss. Signs that you may be in danger include: thinking frequently about ending your life or about joining the person who died; making plans, or looking up methods; having access to means and thinking about using them; feeling like a burden to others; feeling trapped or that the pain is unbearable and will never end; giving away belongings or saying goodbyes; increasing your use of alcohol or drugs; or, sometimes, a sudden calm after a period of intense distress, which can reflect a decision having been made. If you recognise any of these in yourself, please tell someone now. You do not need to be certain that you are in danger, and you do not need to have a plan. It is enough that you are worried. Call or text 988 if you are in the United States, or use the numbers given at the start of this chapter or your local equivalent. Tell a friend, a resident adviser, a family member or a counsellor. If you have access to anything you might use to harm yourself, such as stockpiled medication, a firearm or other means, ask someone to hold it for you or remove it, because reducing access to means during a crisis saves lives. If you feel you might act on your thoughts, go to an emergency department or call emergency services. Crisis services exist precisely for moments like these. The people who answer are trained to listen without judgement and to help you get through the next hours, and they talk every day with people who are grieving. Calling them is not an overreaction. Getting help, and helping someone else Knowing that you need more help is one step; getting it is another, and students often stumble at the practical stage. Most campus counselling services offer an initial consultation, sometimes on the same day for urgent concerns, followed by a limited number of short-term sessions or a referral. Some services have a clinician with particular experience of grief or trauma, and it is reasonable to ask. If the service offers only a few sessions and you need longer-term or specialised treatment, such as complicated grief treatment or trauma-focused therapy, staff can usually help you find a clinician in the community or through telehealth. In the United States, the cost and availability of that care depend heavily on your health insurance, whether a student plan or a parent's plan, and it is worth asking the counselling centre or student health service to help you understand your coverage; the death of a parent can itself change your insurance situation, which is another reason to ask early. In countries with national health systems, a general practitioner is often the route to specialist care, and waiting times vary. In the meantime, university services, crisis lines and bereavement charities can provide support. If you are seeing a counsellor and do not feel the work is helping after a reasonable period, say so. It is appropriate to ask whether a different approach, a different clinician, or a grief-specific treatment might suit you better. Not every therapist is experienced with grief, and the fit between client and therapist matters. If you are receiving treatment for prolonged grief, depression or trauma, your university may also be able to offer formal accommodations through disability services, as Chapter 4 described. Many students find that the combination of treatment and a lighter or more flexible academic load, arranged together, allows them to stay enrolled through a period that would otherwise have forced them to leave. You may also be reading this chapter because you are worried about a friend. The most important thing to know is that asking directly about suicide does not put the idea into someone's head. Research on this question has consistently found that asking does not increase suicidal thinking, and many people are relieved to be asked. You might say, "You've been going through so much since your brother died. Sometimes when people are in that much pain, they think about ending their life. Are you having thoughts like that?" If the answer is yes, stay calm, listen, and take it seriously. Do not promise to keep it secret. Do not try to argue them out of their feelings or tell them how much they have to live for; instead, acknowledge how much pain they are in and tell them you want to help them get support. If they are in immediate danger, do not leave them alone: call emergency services, or, in the United States, call 988 together. If the danger is less immediate, help them contact the counselling service, a doctor or a crisis line, and offer to go with them or sit with them while they call. Tell a resident adviser, a dean or another responsible adult, even if your friend asks you not to; their safety matters more than their momentary annoyance. And look after yourself too. Supporting a friend in crisis is heavy, and it is reasonable for you to talk to a counsellor about it as well. If you are worried about a friend's grief but it is not an emergency, the same principles apply more gently. Tell them what you have noticed, without accusation. Ask how they are really doing. Offer to help them find support. Keep in touch. Most people who are struggling do not need a friend to be their therapist. They need a friend who notices, asks and stays. Grief is not a disorder, and most bereaved students will never need the information in this chapter. But for those who do, it can be the difference between a year of suffering and a year of recovery, or, in rare cases, between life and death. There is no shame in needing more help than time and friends can give. Recognising that you do is itself a form of care: for yourself, and for the person whose memory you are carrying. Conclusion: Grief Keeps Its Own Calendar Set the chapters of this book side by side and a single mismatch comes into view. The university runs on one calendar and grief runs on another. The university's calendar is front-loaded. Its bereavement policies, where they exist, typically grant a few days of excused absence around the funeral. Its sympathy is concentrated in the first weeks, when instructors are notified, extensions are offered and friends bring food. Its structures assume that a disruption is an event: something that happens, is accommodated, and is then over. Grief's calendar looks very different. As Chapter 1 showed, it does not proceed through stages toward a finishing line; the central experience of yearning persists, and acceptance and pain coexist from the start. As Chapter 5 showed, it oscillates, so that a student who seems to be coping in week three may be overwhelmed in week nine. As Chapter 7 showed, it returns at anniversaries, which on a campus often coincide with the same exam season in which the death occurred, and at milestones such as graduation. And as Chapter 8 showed, the most serious complication of grief, prolonged grief disorder, is by definition something that becomes visible only after six or twelve months, long after the institutional response has ended. The cognitive fog, the sleep disruption, the loss of motivation described in Chapter 1 do not respect the add-drop deadline. This mismatch is the real subject of the book, and it has consequences that could not have been stated at the start. What follows for students The first consequence is that the right unit for planning is not the week of the funeral but the year that follows. The early steps in Chapters 3 and 4, telling one office, protecting your options, asking for extensions, remain essential. But a student who treats them as the whole job will be unprepared when grief returns in the second term or at the first anniversary. It is worth, early on, marking in your calendar the dates you can foresee: the birthday, the anniversary, the holidays, the exam period that will echo the one in which the person died. It is worth telling your adviser or dean that you may need to come back for further help later, so that a second request does not feel like an imposition. And it is worth knowing that the documentation you create in the first weeks is also the evidence you may need months from now, for a retroactive withdrawal, a financial aid appeal, or an extenuating circumstances claim. The second consequence is that the most important skill is not endurance but calibration. The dual process model does not say that grieving students should push on or step back; it says they should move between the two, and that when the load becomes too great, the answer is to reduce it rather than to cope harder. Read with the chapters on accommodations and coping, that becomes a practical principle: judge your load honestly and often, and adjust it before collapse rather than after. The students in this book who fared worst were not those who grieved most intensely. They were those who, from pride, fear or exhaustion, carried a full load in silence until it broke. The third consequence concerns the line between ordinary and complicated grief. Because prolonged grief disorder is defined by persistence, the student most at risk is not necessarily the one who is most distraught in the first month. It may be the one who is still unable to function at month ten, whose friends have long since stopped asking. That is why this book has insisted both that grief is normal and that a recognised disorder exists. The two claims are not in tension. The first protects the great majority of bereaved students from being pathologised. The second protects the minority from being ignored. Holding both is what allows a student, or a friend, to notice when something has changed. What remains unsettled Several questions remain open, and honesty requires naming them. The formal recognition of prolonged grief disorder is still debated. Critics worry about turning a universal human experience into a medical category, about whether diagnostic thresholds fit people whose cultures mourn for longer or differently, and about the fact that the two major diagnostic systems disagree on whether six or twelve months is the right threshold. Supporters point to the real suffering of people whose grief does not ease and to the existence of effective treatments. Both sides have a point, and the evidence will continue to develop. The research on continuing bonds has also moved from a simple rejection of "letting go" to a more careful question: which kinds of bond help which people, and when? The answer is not yet settled. And specific research on bereaved students remains thinner than it should be, given how many of them there are. We know that bereavement is common on campus and that it can depress academic performance. We know much less about which forms of institutional support actually work best. What should change If the central problem is a mismatch between two calendars, then institutions carry part of the responsibility for closing it. A university that took grief seriously would design its bereavement policies around the long curve rather than the funeral week: allowing students to request flexibility later in the year without having to prove that their grief is still justified, and making retroactive withdrawal straightforward where a loss clearly impaired a term. It would write its policies around the significance of a relationship rather than a narrow list of family members, so that the student grieving a grandmother who raised her, or a friend from home, is not left to plead. It would train instructors to respond well, and it would make the dean's notification service and the extenuating circumstances processes visible to students who do not yet know they exist. It might even, with consent, reach out to bereaved students at the points on grief's calendar where support is most likely to have fallen away: at three months, at the anniversary, before graduation. Friends, too, can shift their timing. The most valuable support after a death is often not the flowers in the first week but the message in the fourth month, the invitation on the anniversary, the question asked long after everyone else has stopped asking: "How are you doing with it, really?" And bereaved students themselves can hold a different expectation. You are not trying to get back to normal by a particular date. You are learning to live alongside a loss that will change shape over years, carrying a relationship that continues in altered form, inside an institution that was not built for this but can often be made to bend. That is slower and less tidy than the five stages promised. It is also closer to the truth, and truer maps make for kinder journeys. Notes Introduction Balk, D. E., Walker, A. C., & Baker, A. (2010). Prevalence and severity of college student bereavement examined in a randomly selected sample. Death Studies, 34(5), 459–468. Servaty-Seib, H. L., & Hamilton, L. A. (2006). Educational performance and persistence of bereaved college students. Journal of College Student Development, 47(2), 225–234. Kübler-Ross, E. (1969). On Death and Dying. New York: Macmillan. Stroebe, M., & Schut, H. (1999). The dual process model of coping with bereavement: Rationale and description. Death Studies, 23(3), 197–224. Klass, D., Silverman, P. R., & Nickman, S. L. (Eds.). (1996). Continuing Bonds: New Understandings of Grief. Washington, DC: Taylor & Francis. Chapter 1 Maciejewski, P. K., Zhang, B., Block, S. D., & Prigerson, H. G. (2007). An empirical examination of the stage theory of grief. JAMA, 297(7), 716–723. Stroebe, M., Schut, H., & Boerner, K. (2017). Cautioning health-care professionals: Bereaved persons are misguided through the stages of grief. Omega: Journal of Death and Dying, 74(4), 455–473. O'Connor, M.-F. (2022). The Grieving Brain: The Surprising Science of How We Learn from Love and Loss. New York: HarperOne. Bonanno, G. A., Wortman, C. B., Lehman, D. R., et al. (2002). Resilience to loss and chronic grief: A prospective study from preloss to 18-months postloss. Journal of Personality and Social Psychology, 83(5), 1150–1164. Boss, P. (1999). Ambiguous Loss: Learning to Live with Unresolved Grief. Cambridge, MA: Harvard University Press. Chapter 2 Arnett, J. J. (2000). Emerging adulthood: A theory of development from the late teens through the twenties. American Psychologist, 55(5), 469–480. Doka, K. J. (Ed.). (1989). Disenfranchised Grief: Recognizing Hidden Sorrow. Lexington, MA: Lexington Books. Chapter 4 Americans with Disabilities Act of 1990, 42 U.S.C. § 12101 et seq.; Section 504 of the Rehabilitation Act of 1973, 29 U.S.C. § 794 (United States). Equality Act 2010 (Great Britain). Satisfactory academic progress requirements for US federal student aid, including appeals based on the death of a relative: 34 C.F.R. § 668.34. Chapter 5 Stroebe, M., & Schut, H. (2010). The dual process model of coping with bereavement: A decade on. Omega: Journal of Death and Dying, 61(4), 273–289. Stroebe, M., & Schut, H. (2016). Overload: A missing link in the dual process model? Omega: Journal of Death and Dying, 74(1), 96–109. Chapter 6 Pennebaker, J. W. (1997). Writing about emotional experiences as a therapeutic process. Psychological Science, 8(3), 162–166. Stroebe, W., Schut, H., & Stroebe, M. S. (2005). Grief work, disclosure and counseling: Do they help the bereaved? Clinical Psychology Review, 25(4), 395–414. Currier, J. M., Neimeyer, R. A., & Berman, J. S. (2008). The effectiveness of psychotherapeutic interventions for bereaved persons: A comprehensive quantitative review. Psychological Bulletin, 134(5), 648–661. Neff, K. D. (2003). Self-compassion: An alternative conceptualization of a healthy attitude toward oneself. Self and Identity, 2(2), 85–101. Chapter 7 Freud, S. (1917/1957). Mourning and melancholia. In J. Strachey (Ed. and Trans.), The Standard Edition of the Complete Psychological Works of Sigmund Freud (Vol. 14). London: Hogarth Press. Field, N. P., Gao, B., & Paderna, L. (2005). Continuing bonds in bereavement: An attachment theory based perspective. Death Studies, 29(4), 277–299. Neimeyer, R. A. (Ed.). (2001). Meaning Reconstruction and the Experience of Loss. Washington, DC: American Psychological Association. Currier, J. M., Holland, J. M., & Neimeyer, R. A. (2006). Sense-making, grief, and the experience of violent loss: Toward a mediational model. Death Studies, 30(5), 403–428. Tedeschi, R. G., & Calhoun, L. G. (2004). Posttraumatic growth: Conceptual foundations and empirical evidence. Psychological Inquiry, 15(1), 1–18. Chapter 8 American Psychiatric Association. (2022). Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition, Text Revision (DSM-5-TR). Washington, DC: APA. World Health Organization. International Classification of Diseases, 11th Revision (ICD-11), code 6B42, Prolonged grief disorder. Lundorff, M., Holmgren, H., Zachariae, R., Farver-Vestergaard, I., & O'Connor, M. (2017). Prevalence of prolonged grief disorder in adult bereavement: A systematic review and meta-analysis. Journal of Affective Disorders, 212, 138–149. Djelantik, A. A. A. M. J., Smid, G. E., Mroz, A., Kleber, R. J., & Boelen, P. A. (2020). The prevalence of prolonged grief disorder in bereaved individuals following unnatural losses: Systematic review and meta regression analysis. Journal of Affective Disorders, 265, 146–156. Shear, K., Frank, E., Houck, P. R., & Reynolds, C. F. (2005). Treatment of complicated grief: A randomized controlled trial. JAMA, 293(21), 2601–2608. Shear, M. K., Reynolds, C. F., Simon, N. M., et al. (2016). Optimizing treatment of complicated grief: A randomized clinical trial. JAMA Psychiatry, 73(7), 685–694. Boelen, P. A., de Keijser, J., van den Hout, M. A., & van den Bout, J. (2007). Treatment of complicated grief: A comparison between cognitive-behavioral therapy and supportive counseling. Journal of Consulting and Clinical Psychology, 75(2), 277–284. Pitman, A. L., Osborn, D. P. J., Rantell, K., & King, M. B. (2016). Bereavement by suicide as a risk factor for suicide attempt: A cross-sectional national UK-wide study of 3432 young bereaved adults. BMJ Open, 6(1), e009948. Pitman, A., Rantell, K., Marston, L., King, M., & Osborn, D. (2017). Perceived stigma of sudden bereavement as a risk factor for suicidal thoughts and suicide attempt: Analysis of British cross-sectional survey data on 3387 young bereaved adults. International Journal of Environmental Research and Public Health, 14(3), 286. Dazzi, T., Gribble, R., Wessely, S., & Fear, N. T. (2014). Does asking about suicide and related behaviours induce suicidal ideation? What is the evidence? Psychological Medicine, 44(16), 3361–3363. Further Reading Books George A. Bonanno, The Other Side of Sadness (Basic Books, 2009).** An accessible account of research on resilience after loss by one of its leading figures, and a direct challenge to the assumption that everyone must fall apart. Mary-Frances O'Connor, The Grieving Brain (HarperOne, 2022).** A clear, humane explanation of what neuroscience and psychology reveal about why grief feels the way it does, and how it changes over time. Julia Samuel, Grief Works (Penguin Life, 2017).** Stories from a British grief psychotherapist's practice, including the deaths of parents, siblings and friends, with practical reflections on what helps. Megan Devine, It's OK That You're Not OK (Sounds True, 2017).** Written by a therapist after the sudden death of her partner; a firm rejection of platitudes and pressure to "move on". David E. Balk, Helping the Bereaved College Student (Springer, 2011).** Written mainly for staff and faculty, but useful to students who want to understand the research on bereavement in higher education. J. William Worden, Grief Counseling and Grief Therapy (5th ed., Springer, 2018).** A standard professional text, readable by non-specialists, on the tasks of mourning and when grief becomes complicated. Dennis Klass, Phyllis R. Silverman and Steven L. Nickman (eds.), Continuing Bonds: New Understandings of Grief (Taylor & Francis, 1996).** The academic volume that reshaped thinking about ongoing relationships with the dead. Joan Didion, The Year of Magical Thinking (Knopf, 2005) and C. S. Lewis, A Grief Observed (Faber, 1961).** Two classic first-person accounts of grief, very different in tone, which many readers find they return to. Organisations and online resources 988 Suicide and Crisis Lifeline (United States): call or text 988, or visit 988lifeline.org. Free, confidential crisis support at any hour. American Foundation for Suicide Prevention (afsp.org): resources and support groups for people bereaved by suicide, and guidance on helping someone at risk. Center for Prolonged Grief, Columbia University (prolongedgrief.columbia.edu): information about prolonged grief disorder and its treatment, from the team that developed complicated grief treatment. Actively Moving Forward, part of HEAL Grief (healgrief.org): a US network supporting college students who are grieving, including campus chapters. Cruse Bereavement Support (cruse.org.uk): a UK charity offering information and support for bereaved people of all ages. Samaritans (samaritans.org): free support at any hour in the United Kingdom and Ireland, on 116 123. Hashtags: #GriefAndLossOnCampus #CampusGrief #StudentGrief #StudentBereavement #GriefAndLoss #CopingWithGrief #GriefSupport #BereavementSupport #CollegeStudentWellness #UniversityStudentWellbeing #StudentMentalHealth #CampusMentalHealth #MentalHealthAwareness #GriefAwareness #UnderstandingGrief #GriefEducation #GriefPsychology #BereavementResearch #GriefScience #GriefRecovery #GriefAndHealing #HealthyGrieving #GriefCopingStrategies #CopingMechanisms #EmotionalWellbeing #EmotionalResilience #PsychologicalResilience #DualProcessModel #LossOrientedCoping #RestorationOrientedCoping #ContinuingBonds #GriefAndMemory #MemorialRituals #RememberingLovedOnes #AnticipatoryGrief #AmbiguousLoss #DisenfranchisedGrief #ProlongedGriefDisorder #ComplicatedGrief #GriefAndDepression #TraumaticLoss #GriefAndTrauma #GriefCounseling #CampusCounseling #ProfessionalMentalHealthSupport #PeerSupport #GriefSupportGroups #SocialSupport #FamilyBereavement #LossOfAParent #LossOfASibling #LossOfAFriend #InternationalStudentGrief #GrievingAwayFromHome #StudentIsolation #CampusCommunity #CompassionateCampus #BereavementAccommodations #AcademicAccommodations #AcademicFlexibility #BereavementLeave #CompassionateLeave #AssignmentExtensions #ExamDeferrals #ExtenuatingCircumstances #MitigatingCircumstances #AcademicAdvising #StudentSupportServices #DeanOfStudents #DisabilitySupportServices #AcademicContinuity #StudyingWhileGrieving #GriefAndConcentration #GriefAndAcademicPerformance #StudentSuccess #TimeManagement #SelfCareForStudents #HealthyCoping #MindfulnessAndGrief #GriefJournaling #SleepAndWellbeing #StressManagement #StudentResilience #UniversitySupportSystems #FinancialAidSupport #LeaveOfAbsence #ReturningToCampus #MentalHealthResources #SuicidePrevention #CrisisSupport #CompassionateEducation #HigherEducationWellbeing
- The Opioid Litigation (Mass Torts, Public Nuisance, and Corporate Accountability)
Download the Book (PDF): Introduction In the spring of 2019, a courtroom in Norman, Oklahoma, became the unlikely center of American tort law. There was no jury. There was no injured plaintiff describing a particular harm done to a particular body. There was a state, suing one of the largest health-care companies in the world, on the theory that the company had helped create a condition in Oklahoma that was injurious to the public as a whole, and that it should pay to fix it. The legal vehicle was public nuisance, a tort whose ancestors were the blocked highway and the fouled river, and whose modern textbooks give it a few pages at most. The trial judge agreed with the state and ordered Johnson & Johnson to fund an abatement plan valued at more than half a billion dollars. Two years later, the Oklahoma Supreme Court threw the judgment out, holding that the state had stretched the tort past anything it had ever been. That reversal did not end the story. By then, thousands of cities, counties, tribes, school districts, hospitals, and states had filed their own opioid suits, most of them built around the same nuisance theory, and by then the defendants had decided that the risk of losing some of those cases was too great to bear. Across 2021 and 2022, the three dominant drug distributors, Johnson & Johnson, the generic manufacturer Teva, Allergan, and the three largest pharmacy chains agreed to national settlements whose headline values, together with later deals and the bankruptcy plans of Purdue Pharma, Mallinckrodt, and Endo, run well past fifty billion dollars. Those payments are now flowing, year by year, into state treasuries, county accounts, and new foundations, where they are supposed to pay for treatment, recovery, and prevention for a generation. This is a book about how that happened and what it means. It treats the opioid litigation as three things at once. It is a story about an epidemic, and about the evidence that the companies who made, marketed, shipped, and dispensed prescription opioids knew or should have known what their products were doing. It is a story about legal doctrine, and specifically about the attempt to press an old and elastic tort into service against a modern national market, an attempt that won some of the most important battles while losing most of the reported appellate decisions. And it is a story about institutions: the federal multidistrict litigation, the coalition of state attorneys general, the bankruptcy court, and the novel settlement machinery that turned tens of thousands of fragmented claims into a handful of national deals and now distributes the money they produced. The argument The book's controlling claim is this: the opioid litigation succeeded as a settlement system far more than it succeeded as law, and that fact explains both its achievements and its dangers. As law, the public nuisance theory had a mixed and in some places poor record. The courts that looked hardest at it, including the highest courts of Oklahoma and Ohio and a federal judge in West Virginia after a full bench trial, concluded that it did not fit, or that a legislature had displaced it. Other courts, juries, and judges disagreed, and a few plaintiffs won large verdicts. What public nuisance supplied was not a settled rule of liability but a credible threat: the possibility that a defendant would face, in thousands of jurisdictions, a theory that required no proof of individual injury, looked past the doctors who wrote the prescriptions, and asked for the cost of repairing an entire community. That threat, multiplied by the number of plaintiffs and amplified by the documents that discovery and journalism pried loose, was enough to bring every major defendant to the table. As a settlement system, the litigation built something new. It created national agreements that bound states and local governments together through participation thresholds and incentive payments; it assigned money among jurisdictions through formulas based on pills shipped, overdose deaths, and addiction rates; it restricted most of the money to a list of approved public-health uses; and it attached conduct rules that changed how distributors monitor orders and how manufacturers market. It also used bankruptcy in ways that the Supreme Court eventually refused to endorse, in Harrington v. Purdue Pharma L.P. in 2024, forcing the most notorious of the settlements to be rebuilt on a consensual footing. The dangers follow from the same fact. Because liability was never firmly established, the terms of accountability were set by negotiation rather than adjudication. Because the money went to governments rather than to individuals, its value depends on the choices of thousands of local officials, some of whom have spent it well and some of whom have not. And because the model worked, it is now being copied, in litigation over vaping products, social media platforms, industrial chemicals, and climate change, by plaintiffs who hope that the same threat will produce the same result. What this book covers The chapters move in order from the harm to the remedy. The first chapter sets out the factual predicate of the litigation: how prescription opioids spread through American medicine after the mid-1990s, how the supply chain worked, what regulators did and did not do, and what the evidence later showed. The second explains why ordinary product liability and individual injury suits, which had been tried against opioid makers for years, largely failed, and why that failure pushed plaintiffs toward governmental claims. The third examines public nuisance itself: its history, its elements, its earlier deployment against lead paint and firearms, and the features that made it attractive in the opioid cases. The fourth chapter follows the theory into the courtroom, through the trials in Oklahoma, California, West Virginia, Ohio, San Francisco, and Baltimore, and explains why they came out so differently. The fifth turns to the procedural machinery: the federal multidistrict litigation in Cleveland, the parallel state-court and attorney-general litigation, and the experiments, some of them rejected on appeal, that the federal judge used to push the cases toward resolution. The sixth describes the national settlements themselves, how they were structured, what they required, and why nearly every eligible government signed on. The seventh deals with bankruptcy, the Sackler family, and the Supreme Court's decision on non-consensual third-party releases. The eighth follows the money, from settlement formulas to state-local allocation agreements to the individual spending choices now drawing scrutiny. The ninth asks what accountability, beyond money, the litigation actually produced: criminal pleas, conduct restrictions, market exits, public documents, and the limits of all of them. A conclusion considers what the opioid model means for the future of public nuisance and of mass litigation generally. The book concentrates on American law, because the litigation was American, though it notes comparisons where they help, particularly the tobacco litigation of the 1990s, which served as both inspiration and warning. It is written for readers who are not lawyers but who want to understand what actually happened and why it matters. It is educational and is not legal advice. A note on numbers Opioid settlements are reported in headline figures that can mislead. A settlement "worth" several billion dollars is usually a maximum, payable over a decade or more, contingent on participation levels, and reduced by attorneys' fees and costs. Some of it may consist of donated medicine valued at a wholesale price. Where this book gives figures, it gives the publicly announced values and notes, where it matters, the difference between headline and cash. The same caution applies to mortality data: overdose deaths are counted by the Centers for Disease Control and Prevention with a lag, and the categories overlap, because many deaths involve more than one drug. What the numbers do not capture is the scale of the underlying loss. Hundreds of thousands of Americans died of opioid overdoses in the quarter century covered by this book; many more lived with addiction, lost custody of children, went to prison, or buried family members. The litigation cannot restore any of that. Its claim, and the measure by which it should be judged, is narrower: that it forced some of those responsible to pay for part of the repair, changed some of the practices that caused the harm, and put the facts on the public record. Whether it did those things well is the question the rest of the book tries to answer. Chapter 1: How the Crisis Was Made Every piece of litigation rests on a story about what happened, and the opioid cases rest on a particularly contested one. The defendants' version is that they made, sold, and shipped lawful medicines approved by the Food and Drug Administration, in quantities authorized by the Drug Enforcement Administration, on the orders of licensed physicians, to licensed pharmacies, for patients in genuine pain; that the epidemic was driven increasingly by illicit heroin and fentanyl that none of them touched; and that responsibility, if it lies anywhere, lies with regulators, prescribers, traffickers, and individual choices. The plaintiffs' version is that the companies manufactured demand through deceptive marketing, shipped volumes of pills that no legitimate medical need could explain, ignored red flags they were legally obliged to report, and thereby opened the door through which illicit opioids later poured. Both versions contain true statements. The litigation was, in large part, a fight over which of them described the causes that mattered. A change in medicine Opioids are among the oldest drugs in medicine, and for most of the twentieth century American physicians used them cautiously outside of surgery, acute injury, and end-of-life care. The reasons were not mysterious: the drugs produce tolerance, so that patients need more over time for the same effect, and they produce dependence and, in a significant share of long-term users, addiction. Through the 1980s and early 1990s, however, a movement within pain medicine argued that doctors were leaving patients in unnecessary suffering because of exaggerated fears. Some of the evidence offered for that view was thin. A five-sentence letter published in the New England Journal of Medicine in 1980 by Jane Porter and Hershel Jick, reporting that addiction was rare among hospitalized patients in one monitoring program who received narcotics, came to be cited hundreds of times as if it showed that addiction was rare among patients treated with opioids for chronic pain in general. It showed nothing of the kind, and its authors later said so. Into that shifting climate, Purdue Pharma launched OxyContin in 1996. The drug was a controlled-release formulation of oxycodone, designed to deliver the opioid steadily over roughly twelve hours. The FDA-approved label at the time stated that the delayed absorption was believed to reduce the abuse liability of the drug, a line that became central to Purdue's marketing and, later, to the legal cases against it. Purdue promoted OxyContin aggressively, through a large sales force, speakers' programs, and funding for pain education and advocacy groups, and emphasized its use not only for cancer pain but for common chronic conditions such as back pain and arthritis. It paid sales representatives substantial bonuses. The drug became a blockbuster. Other manufacturers followed with their own branded and generic opioids. Johnson & Johnson's Janssen subsidiary sold the fentanyl patch Duragesic and later the tapentadol products Nucynta; J&J also owned, until 2016, the companies Tasmanian Alkaloids and Noramco, which grew and processed the raw material from which much of the American oxycodone supply was made. Endo sold Opana and Opana ER. Teva, through its acquisition of Cephalon, sold the fentanyl products Actiq and Fentora. Mallinckrodt became one of the largest producers of generic oxycodone. A small company, Insys Therapeutics, would later become notorious for bribing doctors to prescribe its fentanyl spray Subsys to patients who did not have the cancer pain it was approved for. By the late 2000s, prescribing had risen to levels without precedent. Regulators and public-health researchers began to track the consequence: rising rates of emergency visits, treatment admissions, and deaths involving prescription opioids. Public-health researchers commonly describe the epidemic as arriving in three waves. The first, beginning in the late 1990s, was driven by prescription opioids. The second, beginning around 2010, was marked by a sharp rise in heroin deaths, as reformulation of OxyContin in 2010 to make it harder to crush and inject, together with tighter prescribing, pushed some dependent users toward a cheaper street drug. The third, beginning around 2013, was driven by illicitly manufactured fentanyl and its analogues, which are far more potent and far more lethal. According to the Centers for Disease Control and Prevention, nearly 645,000 people died from overdoses involving opioids, prescription and illicit, between 1999 and 2021. Total drug overdose deaths of all kinds peaked at more than 100,000 a year in 2022 and 2023, before falling sharply in 2024, to roughly 80,000, the steepest decline on record. The relationship between the waves is at the heart of the causation fight. Defendants argue that the later deaths, which dominate the totals, were caused by criminal drug trafficking. Plaintiffs answer that the prescription wave created a large population of dependent people who then became the market for heroin and fentanyl, and that surveys of heroin users consistently found that many had first misused prescription opioids. Neither claim settles the question by itself, and courts treated it very differently, as later chapters show. The supply chain and its gatekeepers Prescription drugs reach patients through a chain with several links, and the opioid litigation reached every one of them. Manufacturers make the drug and market it to prescribers. Wholesale distributors buy from manufacturers and deliver to pharmacies, hospitals, and clinics; three companies, McKesson, Cardinal Health, and AmerisourceBergen (renamed Cencora in 2023), handle the great majority of the American market. Pharmacies dispense on prescriptions, and national chains such as CVS, Walgreens, and Walmart account for a large share of retail dispensing. Behind them sit pharmacy benefit managers, which manage drug coverage for insurers and employers, and consultants who advised manufacturers on sales strategy. What made the distributors and pharmacies especially important to the litigation is that federal law assigned them a gatekeeping role. The Controlled Substances Act of 1970 created a closed system of distribution for drugs with abuse potential. Every entity that handles a controlled substance must register with the DEA, and every registrant has obligations to guard against diversion, meaning the movement of drugs from legitimate to illegitimate channels. A long-standing DEA regulation, 21 C.F.R. § 1301.74(b), requires registrants to design and operate a system to disclose suspicious orders of controlled substances and to inform the DEA of them, and it defines suspicious orders to include orders of unusual size, orders deviating substantially from a normal pattern, and orders of unusual frequency. The DEA took the position, repeated in letters to distributors in 2006 and 2007 and upheld by the federal court of appeals in Washington in Masters Pharmaceutical, Inc. v. DEA in 2017, that a distributor must not only report a suspicious order but must decline to ship it unless it can dispel the suspicion. Congress wrote the reporting duty into the statute itself in the SUPPORT for Patients and Communities Act of 2018. Pharmacists carry a parallel duty. Under 21 C.F.R. § 1306.04, a prescription for a controlled substance is valid only if issued for a legitimate medical purpose by a practitioner acting in the usual course of professional practice, and the regulation places a "corresponding responsibility" on the pharmacist who fills it. A pharmacist who knowingly fills an illegitimate prescription violates the law just as the prescriber does. The chains' role in the litigation turned on whether they had given their pharmacists the tools, staffing, and policies to exercise that responsibility, and whether their corporate dispensing data should have alerted them to patterns their individual stores could not see. The DEA itself set annual production quotas for opioid ingredients and, for years, raised them as demand grew. The agency had enforcement tools, including the power to suspend a registration immediately if a registrant posed an imminent danger. It used them against distributors on several occasions: it suspended a Cardinal Health distribution center in Lakeland, Florida, in 2012, and McKesson paid $150 million in 2017 to resolve allegations that it failed to report suspicious orders, after an earlier settlement in 2008. But enforcement was uneven, and a 2016 statute, the Ensuring Patient Access and Effective Drug Enforcement Act, raised the standard for immediate suspensions. A 2017 investigation by the Washington Post and 60 Minutes, built around a former DEA official, Joseph Rannazzisi, argued that the law had weakened the agency at a critical moment. The law's defenders disputed that account. What matters for the litigation is that the defendants could and did argue that they had operated under the eye of a federal regulator, and the plaintiffs could and did argue that the regulator had been outmatched. The evidence that emerged The opioid cases were won and lost on evidence, and much of the most important evidence became public only because of litigation and journalism working together. The first category was internal marketing material. Purdue's own documents, including those gathered in the federal investigation that produced its 2007 guilty plea, showed the company training its sales force to tell doctors that OxyContin carried a lower risk of addiction and abuse than other opioids, claims the government said were false. In 2007 the Purdue Frederick Company pleaded guilty in federal court in the Western District of Virginia to misbranding OxyContin with intent to defraud or mislead, and three executives pleaded guilty to misdemeanor misbranding; the company and executives paid roughly $634 million. The plea did not end the marketing. Documents disclosed in later suits, notably in the complaint filed by the Massachusetts attorney general in 2019, described the company's continuing efforts, in the years after 2007, to target high-prescribing doctors and to encourage higher doses and longer use, and described the involvement of members of the Sackler family on the company's board. In 2020 Purdue pleaded guilty again, this time to three federal felonies: a conspiracy to defraud the United States and to violate the federal food and drug law, and two conspiracies to violate the federal Anti-Kickback Statute. The second category was distribution data. The DEA's Automation of Reports and Consolidated Orders System, known as ARCOS, records every shipment of certain controlled substances from manufacturer to distributor to pharmacy. For years the data were treated as confidential. In 2019, after litigation by the Washington Post and the Charleston, West Virginia, newspaper publisher HD Media, the federal judge overseeing the national opioid litigation allowed release of the database covering 2006 to 2012. The newspaper's analysis found that about 76 billion oxycodone and hydrocodone pills had been shipped nationwide in those seven years. The data could be sorted by county, by pharmacy, and by distributor, and they showed extraordinary concentrations in small places. Reporting by Eric Eyre of the Charleston Gazette-Mail, which won the Pulitzer Prize in 2017, and a 2018 report of the House Energy and Commerce Committee, documented cases such as the town of Kermit, West Virginia, with a few hundred residents, where one pharmacy received millions of hydrocodone pills in a two-year period, and Williamson, West Virginia, a town of roughly three thousand, where two pharmacies received more than twenty million pills over about a decade. Numbers like those did two things in court. They made it hard for a distributor to argue that nothing in its data could have signaled a problem, and they made it plausible to link a particular company's shipments to a particular community. They did not, by themselves, prove that any given shipment was diverted or that any given death followed from it. That gap between aggregate evidence and individual causation is what much of the legal doctrine discussed in later chapters is about. The third category was compliance records. Discovery showed, in varying degrees across the defendants, suspicious-order monitoring systems that set thresholds high enough to trigger rarely, that were adjusted upward at a customer's request, or that produced reports sent to the DEA in bulk after shipments had already gone out. For pharmacies, discovery showed corporate policies on "red flags" such as patients traveling long distances, paying cash, or presenting prescriptions for combinations of drugs associated with abuse, and it showed disputes about whether pharmacists had the time and support to act on them. The defendants had responses, often persuasive in particular cases: that the DEA had approved or known about their systems, that thresholds were a starting point for investigation rather than a rule, that pharmacists refused many prescriptions. But the documents gave plaintiffs something that marketing claims alone did not: evidence of breach of a specific legal duty. The shape of a legal problem By around 2014, then, the elements of a legal case were visible. There was a public-health catastrophe of enormous scale. There was evidence of deceptive marketing by some manufacturers, including at least one criminal conviction. There was evidence that distributors and pharmacies had not performed the monitoring that federal law required. And there were governments paying the costs: for emergency response, naloxone, foster care, jails, medical examiners, and treatment. But there was also a set of facts that made the case hard to fit into the ordinary categories of liability. The products were lawful, approved by the FDA, and prescribed by physicians exercising independent judgment. Many harms were suffered by people who had obtained the drugs illegally or used them in ways not prescribed. The injuries were diffuse, spread across thousands of communities and millions of people, and were produced by a long chain of events in which the defendants' conduct was only one link. The later waves of the epidemic involved drugs the defendants never made. And the government plaintiffs were seeking to recover costs they had incurred as governments, a category of loss that tort law had long been reluctant to recognize. The next chapter explains why the conventional legal tools struggled with those facts, and why that struggle sent the litigation looking for a different theory. Chapter 2: The Limits of Ordinary Tort Law Long before cities and counties began suing, individual patients sued the makers of prescription opioids. They sued under familiar theories: that the product was defectively designed, that the manufacturer failed to warn of its dangers, that its marketing was fraudulent or negligent. Those suits rarely succeeded. Understanding why is the key to understanding everything that followed, because the governmental litigation of the 2010s was built, doctrine by doctrine, to avoid the obstacles that had defeated the individual cases of the 2000s. The individual patient's case Consider the position of a person who became addicted to OxyContin after a doctor prescribed it for back pain in 2001, and who sued Purdue a few years later. To win a product liability case, such a plaintiff generally had to prove that the product was defective in design or warning, and that the defect caused the injury. Each step presented a problem. Design defect was difficult because the dangerous property of an opioid is inseparable from its therapeutic one. An opioid relieves pain by acting on the same receptors that produce euphoria, tolerance, and dependence. Under the prevailing test in many states, a plaintiff must show a reasonable alternative design that would have reduced the risk without sacrificing the product's utility, and for most of the relevant period there was no opioid that relieved pain without the risk of addiction. Some courts also applied the principle, reflected in comment k to Section 402A of the Restatement (Second) of Torts, that certain unavoidably unsafe products such as prescription drugs are not defective merely because they carry inherent risks, provided they are properly prepared and accompanied by adequate warnings. Failure to warn ran into the learned intermediary doctrine. In most American jurisdictions, a prescription drug manufacturer discharges its duty to warn by warning the prescribing physician, not the patient, on the theory that the physician is best placed to weigh risks and benefits for the individual. The FDA-approved label for OxyContin did warn of abuse potential and dependence. A plaintiff could argue that the label understated the risk, and that the marketing contradicted it, but then had to prove that the plaintiff's own doctor had been misled and would have prescribed differently with a better warning. That is a demanding, doctor-specific inquiry, and in many cases the doctor testified that he or she knew opioids were addictive and would have prescribed anyway. Causation was often the deepest problem. Addiction has many contributing causes: genetics, mental health, social environment, the availability of other drugs. A plaintiff who had misused the drug, by crushing it, taking more than prescribed, or combining it with alcohol or other drugs, faced defenses that the misuse, rather than any defect, caused the harm. Plaintiffs who had used opioids from multiple manufacturers, or obtained some from friends or dealers, faced difficulty in identifying which product caused what. The wrongful conduct rule and other barriers Several doctrines worked specifically against plaintiffs whose claims arose from drug misuse. The most striking was the wrongful conduct rule, applied in some states, which bars recovery for injuries that arise from the plaintiff's own illegal conduct. In Foister v. Purdue Pharma L.P., decided by a federal court in eastern Kentucky in 2003, the court dismissed claims by plaintiffs who had, among other things, obtained OxyContin through fraud or illegal means. The Mississippi Supreme Court applied a similar principle in Price v. Purdue Pharma Co. in 2006, holding that a plaintiff who had obtained prescriptions by deceiving multiple doctors could not recover for the resulting addiction. Comparative fault, which reduces recovery in proportion to the plaintiff's own share of responsibility, did similar work in states without a strict bar. Statutes of limitations added another hurdle. Addiction develops gradually, and a plaintiff might not know, or be found to have reason to know, that a product was the source of injury until years after the first prescription. Defendants argued, often successfully, that plaintiffs who had recognized their dependence long before suing were out of time. Class actions, which might have spread the cost of litigation across many plaintiffs, largely failed for the reasons that doom most personal-injury class actions: the questions of who was prescribed what, by whom, with what warning, and with what history, vary from person to person and overwhelm any common questions. Courts in the early 2000s declined to certify classes of OxyContin users. The result was that Purdue, by the company's own account in later years, won dismissal or withdrawal of the great majority of individual suits filed against it in the early 2000s, and settled others confidentially. Plaintiffs' lawyers learned that, however troubling the evidence of marketing might be, an individual addiction case was expensive to try and hard to win. Early government suits State governments tried their hand as well. Several state attorneys general investigated Purdue in the early 2000s, and in 2007, alongside the federal plea, Purdue paid about $19.5 million to settle claims by 26 states and the District of Columbia over its marketing, agreeing to certain restrictions. West Virginia's attorney general had sued Purdue earlier and settled in 2004 for $10 million. Kentucky, whose Pike County had been one of the early epicenters of OxyContin abuse, pursued a long-running suit that Purdue settled in 2015 for $24 million. These state cases relied largely on consumer protection statutes and state Medicaid fraud laws, which gave attorneys general a better position than individual plaintiffs. Consumer protection statutes often dispense with proof of individual reliance and injury when the state sues to stop deceptive practices, and allow civil penalties per violation. Medicaid fraud theories allowed the state to recover payments it had made for prescriptions procured by false marketing. But the recoveries were modest compared with the scale of the harm, and they left untouched the far larger costs that states and localities were incurring for treatment, emergency services, child welfare, and criminal justice. In 2014, a new kind of case appeared. The City of Chicago, and the California counties of Santa Clara and Orange, sued several manufacturers, alleging that their deceptive marketing of opioids for chronic pain had caused the governments to pay for unnecessary prescriptions and to bear the costs of the resulting addiction. Those cases drew on consumer protection law, false claims statutes, and public nuisance. Over the next few years, hundreds and then thousands of counties, cities, and tribes followed, and they added the distributors and pharmacies as defendants. The problem of governmental loss Governmental plaintiffs avoided the doctrines tied to individual users. A county does not use drugs, commit wrongful conduct, or fail to read a warning. But governments faced their own set of obstacles, rooted in tort law's long reluctance to let a public body recover the costs of performing public functions. The first is the municipal cost recovery rule, sometimes called the free public services doctrine. Its classic statement is City of Flagstaff v. Atchison, Topeka & Santa Fe Railway Co., decided by the Ninth Circuit in 1983, which held that a city could not recover from a railroad the costs of responding to a train derailment and chemical spill. The reasoning was that the costs of public services such as fire and police protection are borne by the public through taxation, and that shifting them to individual tortfeasors should be left to legislatures. Applied to opioids, the rule threatened governments' claims for the costs of emergency response, policing, and treatment. Courts have recognized exceptions, for example when the defendant's conduct creates a recurring or continuing condition or when a public nuisance must be abated, and plaintiffs framed their claims to fit them. The second is the remoteness or derivative-injury doctrine. When a defendant harms one party, and that party's injury in turn causes losses to others, the others generally cannot sue; the chain of causation is too long and the risk of double recovery too great. The doctrine had defeated many of the municipal firearms suits of the late 1990s and early 2000s. In Ganim v. Smith & Wesson Corp., decided in 2001, the Connecticut Supreme Court held that the city of Bridgeport's claims against gun makers for the costs of gun violence were too remote, because the city's harms derived from injuries to the direct victims of crimes committed by third parties. Opioid defendants argued that governmental costs were derivative in exactly the same way: of harms to individual users, many of which were themselves caused by criminal diversion and trafficking. The third is the economic loss rule, which in many jurisdictions bars recovery in negligence for purely financial losses not accompanied by physical injury or property damage. A county's increased spending on foster care is a purely economic loss. The fourth, and most fundamental, is proximate cause. Even a plaintiff who can show that the defendant's conduct was a factual cause of harm must show that it was a legally sufficient cause, which usually means that the harm was a foreseeable result and that no independent intervening act broke the chain. Between a distributor's shipment and a county's costs stood the pharmacy, the prescriber, often a diverter or dealer, the user, and, in the later waves, an illicit fentanyl supply. Defendants argued that these intervening acts, many of them criminal, cut off liability. Statutory theories and their limits Governments did not rely only on the common law. The complaints filed in the late 2010s typically pleaded a battery of statutory claims alongside the tort theories, and some of them proved important. State consumer protection statutes, often called unfair and deceptive acts and practices laws, were the attorneys general's natural weapon against manufacturers. They reach false or misleading statements in trade, usually without the need to show that any particular consumer relied on the statement, and they authorize civil penalties for each violation, which can add up quickly when every sales call or promotional mailing is a separate violation. Their limit is that they fit marketing better than distribution: a distributor that ships an order it should have flagged has not made a deceptive statement to anyone. False claims statutes offered a route to recover public money. Where a government's Medicaid program or employee health plan paid for opioid prescriptions that, the government alleged, were induced by false marketing, it could argue that the resulting claims for payment were tainted. These theories required proof tying the marketing to the prescriptions paid for, a demanding showing, and they recovered only the cost of the drugs, not the far larger downstream costs of addiction. Some local governments also pleaded claims under the federal Racketeer Influenced and Corrupt Organizations Act, alleging that manufacturers had formed an enterprise with front groups and key opinion leaders to promote opioids, and that manufacturers and distributors had jointly avoided their suspicious-order obligations in order to keep production quotas high. RICO offered treble damages and attorneys' fees, but it requires an injury to "business or property" caused directly by the racketeering, and governmental expenditures on public services fit that requirement poorly. The federal court handling the national litigation allowed some of these claims to proceed past the pleadings, but they were never the center of gravity. Finally, several states enacted statutes designed to ease their path, much as some had done during the tobacco litigation, for example by authorizing the state to recover costs or by clarifying the attorney general's authority. Such statutes helped the particular states that passed them but did nothing for the thousands of local governments suing under general law. The lesson from tobacco The governmental opioid suits were not the first attempt to overcome these obstacles. They had a model, and a warning, in the tobacco litigation of the 1990s. For four decades, individual smokers had sued cigarette makers and almost always lost, defeated by assumption-of-risk defenses and by the industry's scorched-earth litigation tactics. Beginning with Mississippi in 1994, state attorneys general changed the frame. They sued not as representatives of injured smokers but as payers, seeking reimbursement for the Medicaid costs of treating smoking-related illness, and they pleaded a range of theories, including public nuisance, consumer protection, antitrust, and unjust enrichment. The legal merits of those claims were never tested to final judgment on appeal. Mississippi, Florida, Texas, and Minnesota settled individually, and in November 1998 the remaining states entered the Master Settlement Agreement with the major manufacturers, under which the industry agreed to pay an estimated $206 billion over the first twenty-five years and to accept marketing restrictions. The tobacco experience taught three lessons that shaped the opioid cases. The first was that governmental plaintiffs could extract enormous settlements even on legally untested theories, because defendants facing dozens of suits could not afford to lose one. The second was that the injunctive terms of a settlement, such as advertising restrictions and the dissolution of industry front groups, could matter as much as the money. The third, which became clear only later, was that money paid to governments without strict limits on its use would largely go to general budgets. Studies by public-health groups over the following decades found that states spent only a small fraction of their tobacco settlement revenue on tobacco prevention and cessation, and some securitized their future payments for immediate cash. The architects of the opioid settlements explicitly set out to avoid that outcome, with results examined in Chapter 8. What the tobacco cases did not offer was a doctrinal solution. The state Medicaid-reimbursement suits had been fought largely on statutory and equitable grounds, and some states had passed laws to ease their path. The opioid plaintiffs, and especially the thousands of local governments without such statutes, needed a common-law theory that could reach every link of the supply chain, dispense with proof of individual injury, overcome the municipal cost recovery and remoteness doctrines, and support a remedy large enough to matter. They found it in public nuisance. Chapter 3: An Old Tort for a New Crisis Public nuisance is one of the oldest ideas in the common law and one of the least precisely defined. William Prosser, the most influential American torts scholar of the twentieth century, wrote that there is perhaps no more impenetrable jungle in the entire law than that which surrounds the word "nuisance." That imprecision was, for the opioid plaintiffs, its principal attraction. For the defendants and the courts that ruled against them, it was the principal objection. From obstructed highways to the Restatement The tort began in medieval England as a criminal wrong against the Crown. A person who obstructed the king's highway, encroached on royal land, polluted a public waterway, or kept a disorderly house committed an offense against the community as a whole, and the remedy was prosecution and an order to remove the obstruction. Because the wrong was to the public, an individual could not ordinarily sue for it; only a person who suffered a "special injury," different in kind from that suffered by the public generally, could bring a private action. The public authority's remedy was abatement: an order to stop the offending activity and remove the condition. American law inherited the tort and expanded it through statutes that declared particular conditions to be nuisances, from unsanitary slaughterhouses to houses of prostitution. Public nuisance became a flexible tool of local governance, used to address conditions that threatened public health, safety, or morals when no specific regulation reached them. For most of its history, however, it concerned conditions tied to land or to particular activities in particular places: a polluting factory, a dilapidated building, a blocked road. The modern American statement of the tort is Section 821B of the Restatement (Second) of Torts, published in 1979. It defines a public nuisance as "an unreasonable interference with a right common to the general public." It then lists circumstances that may make an interference unreasonable: whether the conduct significantly interferes with the public health, safety, peace, comfort, or convenience; whether it is prohibited by statute, ordinance, or regulation; and whether it is of a continuing nature or has produced a permanent or long-lasting effect that the actor knows or has reason to know has a significant effect on the public right. Section 821C limits private actions to those who suffer harm of a kind different from the general public, and confirms that public officials may sue to abate. Several features of this formulation mattered enormously in the opioid cases. The concept of a "right common to the general public" is elastic; the Restatement's own examples include public health. Liability can rest on intentional conduct that is unreasonable, or on negligent or reckless conduct, or on abnormally dangerous activity. The unreasonableness of an interference can be established by showing that the conduct violated a statute or regulation, which pointed directly at the Controlled Substances Act and its regulations. And the remedy of abatement looks to the future: it asks what it will cost to eliminate the harmful condition, not what damages the plaintiff has already suffered. Why nuisance fit the opioid plaintiffs' needs Set against the obstacles described in the previous chapter, public nuisance offered something close to a tailored solution. It did not require proof of injury to any individual. The harm alleged was to the community: an epidemic of addiction and overdose, overwhelmed hospitals and child welfare systems, neighborhoods affected by drug markets. The learned intermediary doctrine, the wrongful conduct rule, and individual comparative fault had no obvious role. It was traditionally brought by public officials. A state attorney general, a county prosecutor, or a city attorney had standing to sue to abate a public nuisance without proving special injury, which answered the objection that governments were remote, derivative plaintiffs. Abatement was an equitable remedy, which in many jurisdictions meant it would be decided by a judge rather than a jury, and it offered a way around the municipal cost recovery rule. Courts that recognized that rule had long made an exception for the costs of abating a nuisance. And an abatement remedy, forward-looking and measured by the cost of fixing the problem, could be very large: plaintiffs presented abatement plans encompassing years of treatment, prevention, recovery services, and naloxone distribution, priced in the billions. It reached every link in the chain. A manufacturer that deceptively marketed opioids, a distributor that shipped suspicious orders, and a pharmacy that filled illegitimate prescriptions could each be said to have contributed to creating or maintaining the nuisance. Many courts hold that a defendant need not be the sole cause of a nuisance to be liable for it, and some treat liability as joint and several when the harm is indivisible, meaning that each contributor could, in principle, be ordered to pay for the whole. And it could be pleaded in every state. Although nuisance law varies, the core concept exists everywhere, which allowed a coordinated national litigation effort to bring substantially similar claims in thousands of jurisdictions. The earlier product cases: lead paint and firearms The opioid plaintiffs were not the first to try to use public nuisance against the makers of lawful products, and the earlier attempts had mostly failed. In the late 1990s and early 2000s, cities sued firearms manufacturers and distributors, alleging that their marketing and distribution practices created an illegal secondary market that fed gun violence. The Illinois Supreme Court's decision in City of Chicago v. Beretta U.S.A. Corp., in 2004, is the leading rejection. The court held that there was no public right to be free from the threat that others would misuse lawfully sold firearms, that the defendants' lawful commercial conduct was not an unreasonable interference, and that the intervening criminal acts of third parties broke the chain of proximate cause. The court worried that recognizing the claim would open the door to nuisance suits against the makers of any product that could be misused. Other courts reached similar results, and in 2005 Congress passed the Protection of Lawful Commerce in Arms Act, which largely barred such suits against gun makers and sellers. Lead paint produced a split. In State v. Lead Industries Ass'n, decided in 2008, the Rhode Island Supreme Court reversed a jury verdict that had found former lead pigment manufacturers liable for creating a public nuisance, holding that the state had not identified an interference with a public right as opposed to an aggregation of private injuries, and that the defendants lacked control over the lead paint in private homes at the time of the harm. Courts in New Jersey, Missouri, and Wisconsin rejected similar claims. California went the other way. In People v. ConAgra Grocery Products Co., decided by a California court of appeal in 2017, the court upheld a judgment that three companies had created a public nuisance by promoting lead paint for interior residential use decades earlier with knowledge of its hazards, though it narrowed the remedy to homes built before 1951. The trial court had ordered an abatement fund of more than $1.1 billion; after the appellate narrowing, the companies settled in 2019 for $305 million. The California lead paint decision was the most important precedent available to the opioid plaintiffs. It showed that a court could hold a product maker liable in public nuisance for its promotion, not merely its sale, and that knowledge of the hazard could make the promotion wrongful. The Rhode Island and Illinois decisions showed the arguments the defendants would make. The case against expansion The opioid defendants, supported by business groups and a significant body of scholarly opinion, advanced a set of objections that recurred in every court. The first is doctrinal: public nuisance protects rights common to the public, such as the right to use a highway or breathe clean air, and the harms of opioid addiction are an aggregation of injuries to individuals. On this view, calling them a public nuisance merely relabels a product liability claim to escape product liability's requirements. The second concerns control. Traditional nuisance defendants controlled the offending condition: the factory owner could stop the pollution, the landowner could remove the obstruction. Once a manufacturer sells a product, it no longer controls it, and cannot abate a condition created by others' use of it. Abatement, on this view, becomes simply a damages award in another name. The third concerns displacement. Where a legislature has enacted a product liability statute governing claims for harm caused by products, courts should not allow plaintiffs to evade its limits through common-law nuisance. Where federal and state regulators have approved a drug, its label, and its distribution, a jury or a single judge should not second-guess those judgments through a vague reasonableness standard. The fourth is structural. If public nuisance can reach the lawful sale of a product that causes widespread harm when misused, the argument goes, there is no principled limit: sugary drinks, fossil fuels, social media, automobiles. The American Law Institute gave this view significant support in the Restatement (Third) of Torts: Liability for Economic Harm, finalized in 2020, whose Section 8 and commentary state that public nuisance claims against the makers of products that have caused harm have generally been rejected, and caution that the tort is not a suitable vehicle for such claims. Scholars such as Donald Gifford, whose 2003 article "Public Nuisance as a Mass Products Liability Tort" was an early and influential critique, argued that expanding the tort would let courts make regulatory policy without the institutional capacity or democratic accountability to do so. The plaintiffs had answers. The public right at stake, they argued, was the right to public health and safety, which the Restatement expressly recognizes. The defendants' conduct did not end at the point of sale: distributors and pharmacies controlled each shipment and each dispensing decision in real time, and manufacturers controlled their ongoing marketing. The conduct was not lawful but violated federal and state controlled substances law and consumer protection statutes, and the complaint was not that the products were defective but that the defendants' conduct in marketing and distributing them was unreasonable. As for limits, the requirement of unreasonable conduct, and in the distributor and pharmacy cases, the violation of specific regulatory duties, supplied them. Abatement: a remedy under strain Much of the fight over public nuisance was really a fight about its remedy. In the classic case, abatement is simple: the court orders the defendant to stop the polluting discharge or tear down the unsafe structure, or, if the government abates the nuisance itself, orders the defendant to reimburse the cost of doing so. The remedy is tied to a physical condition that can be identified and removed. The opioid plaintiffs asked for something different. Their abatement plans, prepared by public-health experts, proposed to fund programs over periods of ten or fifteen years: treatment slots, medication for opioid use disorder, naloxone, recovery housing, programs for infants born dependent on opioids, surveillance and data systems, and prevention campaigns. The plans estimated the number of people with opioid use disorder in the jurisdiction, the fraction who would need each kind of service, and the cost per person, and multiplied. The result was a figure, often in the billions for a single county, that looked forward rather than back. Defendants attacked this remedy on three grounds. First, they argued that it was damages in disguise. Paying for a county's treatment programs is compensation for harm, not the removal of a condition the defendant controls, and damages carry legal requirements, including jury trial, proof of past loss, and proximate cause, that equitable abatement sidesteps. Second, they argued that the plans funded responses to the whole epidemic, including harms caused by heroin and fentanyl the defendants never touched, and so assigned the defendants responsibility for far more than their share. Third, they argued that the abatement orders would require courts to supervise public-health programs for years, a task for which judges have neither expertise nor authority. The plaintiffs responded that the epidemic was a continuing condition, not a closed past event, and that treating people with opioid use disorder and preventing new cases was exactly what removing the condition meant. The equitable character of the remedy, they said, reflected its historic nature: abatement has always been decided by courts of equity. And on apportionment, they invoked the principle that where several actors contribute to an indivisible harm, each may be held liable for the whole, leaving the defendants to seek contribution from one another. These arguments mattered in practice. Where courts accepted the plaintiffs' view, as the federal court did in the Ohio pharmacy case, the remedy could be enormous even when the defendant's share of the conduct was modest. Where courts rejected it, as the Oklahoma Supreme Court did, the rejection often turned less on whether the defendant had done wrong than on whether abatement, properly understood, could reach what the plaintiffs asked for. The remedy question and the liability question were, in the end, bound together: an elastic tort paired with an elastic remedy was what made public nuisance powerful, and what made its critics most uneasy. What was really at stake Behind the doctrinal arguments lay a question of institutional choice. The opioid epidemic was a public-health emergency in which the legislative and regulatory responses had been slow and, by the plaintiffs' account, captured. Public nuisance offered courts a way to assign responsibility and marshal resources where other branches had not. The defendants' objection was not only that the doctrine was being misapplied but that courts were the wrong place to do this: that questions about how much opioid supply is appropriate, and who should pay for an epidemic with many causes, belong to legislatures and agencies. Neither side could claim that history settled the matter. Public nuisance had always been a tool for addressing new threats to public welfare that no other law reached; that was its function. But it had also always been tied to particular places, conditions, and defendants who could stop what they were doing. The opioid litigation asked whether it could be stretched to cover a national market in a lawful product whose harms were produced by many hands. The answer, as the next chapter shows, depended on which court was asked. Chapter 4: Nuisance on Trial Very few opioid cases were ever tried. Most were settled, stayed, or swept into national deals before a jury was seated. But the handful that did reach verdict or judgment carried an influence out of proportion to their number, because every defendant and every plaintiff read them as signals of what the thousands of other cases were worth. The striking thing about those trials is how inconsistent they were. Juries and judges looking at broadly similar evidence and similar legal theories reached opposite conclusions, and appellate courts reversed or undermined most of the plaintiffs' largest victories. Table 1 sets out the principal outcomes; the rest of the chapter explains them. Oklahoma: the first test and the first reversal Oklahoma's attorney general, Mike Hunter, sued Purdue, Johnson & Johnson, Teva, and others in state court in Cleveland County in 2017. Before trial, Purdue settled for $270 million, most of it directed to a new addiction research and treatment center at Oklahoma State University, and Teva settled for $85 million. Johnson & Johnson went to trial alone in the spring of 2019. Because the state dropped its damages claims and sought only abatement, the case was tried to a judge, Thad Balkman, without a jury. The state's theory was that J&J, through its Janssen subsidiary, had engaged in false and misleading marketing that overstated the benefits and understated the risks of opioids generally, not just its own products, and that this campaign had contributed to an oversupply of opioids in Oklahoma. The state also emphasized J&J's role, through its former subsidiaries, as a supplier of the raw narcotic ingredients used by other manufacturers. In August 2019, Judge Balkman found J&J liable under Oklahoma's public nuisance statute and ordered it to pay for the first year of an abatement plan, initially computed at about $572 million and later reduced to roughly $465 million after a calculation error was corrected. In November 2021, the Oklahoma Supreme Court reversed, in State ex rel. Hunter v. Johnson & Johnson. The court held that Oklahoma public nuisance law had historically been limited to conduct involving the misuse of property or conduct that was criminal in nature, and that it had never been extended to the manufacturing, marketing, and selling of products. It reasoned that product-related harms were the domain of product liability law; that J&J had no control over its products once sold, and so could not abate the nuisance; and that the abatement remedy ordered was in substance a damages award for past harm, since the plan would pay for government programs rather than stop J&J from doing anything. The court warned that extending nuisance to lawful products would create unlimited liability for any company whose products cause harm. The reversal came months after J&J had agreed in principle to a national settlement. It did not reverse the national deal, but it changed the calculation for every later case. Here was a state high court, considering the fullest trial record then available, holding that the plaintiffs' central theory was not the law of the state. California and West Virginia: judgments for the defendants Two more trials in 2021 ended in defeat for the plaintiffs. In California, the counties of Santa Clara, Orange, and Los Angeles and the city of Oakland, suing in the name of the People, tried their public nuisance and false advertising claims against J&J, Teva, Endo, and Allergan in Orange County Superior Court. After a bench trial, Judge Peter Wilson issued a tentative decision in November 2021, later made final, finding for the defendants. His reasoning was chiefly factual. The plaintiffs had not shown that the defendants' marketing caused medically inappropriate prescriptions in the relevant counties; the evidence showed an increase in prescriptions, but that increase could reflect legitimate medical judgment, and plaintiffs had offered no way to separate appropriate from inappropriate prescribing attributable to the defendants' statements. He also found that the defendants' promotional statements were not shown to be false or misleading in the way the plaintiffs claimed, given the regulatory context and FDA-approved labeling. The decision is a reminder that even in the state that had embraced public nuisance for lead paint, a plaintiff still had to prove causation with evidence tied to the defendant. In West Virginia, the city of Huntington and Cabell County, among the hardest-hit communities in the country, tried their public nuisance claims against the three large distributors in federal court before Judge David Faber, in a bench trial in 2021. The plaintiffs sought roughly $2.5 billion to fund a fifteen-year abatement plan. In July 2022, Judge Faber ruled for the defendants. He held, first, that West Virginia public nuisance law had not been extended to the sale and distribution of a product, and predicted that the state's highest court would not extend it. He also found that the distributors' conduct was not unreasonable given that they shipped only in response to prescriptions written by licensed doctors and dispensed by licensed pharmacies, and that the plaintiffs had not proved that the volume of shipments was due to unreasonable conduct rather than prescribing practices the distributors did not control. And he found that the plaintiffs had not shown that the distributors' conduct proximately caused the harm, given the intervening acts of prescribers, pharmacists, and others. The plaintiffs appealed to the Fourth Circuit, which asked the Supreme Court of Appeals of West Virginia to answer the underlying state-law question directly. The West Virginia court declined to answer, leaving the federal appellate court to predict state law for itself, and the case continued in the federal courts. Whatever its final disposition, the Huntington judgment stood for years as the most thorough judicial rejection of the distributor nuisance theory on a full record, and it was rendered in the very community that the plaintiffs' pill-volume evidence had made emblematic. Ohio and San Francisco: the plaintiffs' victories On the other side of the ledger, plaintiffs won two significant cases against pharmacies. The first was the third bellwether trial in the federal multidistrict litigation, brought by Lake and Trumbull counties in northeastern Ohio against the national and regional pharmacy chains, of which CVS, Walgreens, and Walmart remained at trial after others settled. The trial went to a jury in the fall of 2021, before Judge Dan Aaron Polster in Cleveland. The plaintiffs argued that the pharmacies had failed to maintain effective controls against diversion in their dispensing, including by failing to give pharmacists adequate staffing and data to identify red-flag prescriptions, and by failing to use their corporate dispensing data. In November 2021, the jury found that CVS, Walgreens, and Walmart had substantially contributed to an oversupply of prescription opioids and to their diversion, creating a public nuisance. In August 2022, Judge Polster, deciding the abatement remedy himself, ordered the three chains to pay about $650.6 million over fifteen years to fund the counties' abatement plan, and imposed injunctive requirements concerning their dispensing practices. The victory did not survive. On appeal, the Sixth Circuit asked the Supreme Court of Ohio whether the Ohio Product Liability Act abrogates common-law public nuisance claims arising from the sale of a product. The Ohio legislature had amended the statute in 2007 to specify that the definition of "product liability claim" includes any public nuisance claim or cause of action at common law alleging that the design, manufacture, supply, marketing, distribution, promotion, advertising, labeling, or sale of a product unreasonably interferes with a right common to the general public. In December 2024, the Ohio Supreme Court answered yes: the statute abrogates such common-law public nuisance claims, and the counties' claims fell within it. That answer removed the legal foundation of the verdict under Ohio law. The Ohio decision is a vivid illustration of the displacement objection described in Chapter 3. The legislature had anticipated precisely this kind of suit and closed the door to it. The second plaintiffs' victory came in federal court in San Francisco, where the city sued manufacturers, distributors, and Walgreens. After many defendants settled, Judge Charles Breyer held a bench trial against Walgreens and, in August 2022, found that Walgreens had substantially contributed to the opioid epidemic in San Francisco by dispensing without adequate due diligence on red-flag prescriptions, creating a public nuisance under California law. Before the remedy phase was completed, Walgreens settled with the city for a reported $230 million, payable over fourteen years. Baltimore and the later cases Litigation continued after the national settlements for jurisdictions that had not joined them. The most notable is the city of Baltimore, which declined to join the national distributor settlement and pursued its own case in state court. Baltimore reached separate settlements with a series of defendants, and in November 2024 a jury found McKesson and AmerisourceBergen liable for creating a public nuisance and awarded the city about $266 million. Post-trial proceedings over the size and form of the remedy followed, and the fight over the judgment continued into 2025. The Baltimore case showed both the continuing potency of the nuisance theory before a sympathetic jury and the risk that non-participating jurisdictions took on: they kept their chance at a larger recovery, but also bore the expense, delay, and uncertainty that the national deals were designed to eliminate. A handful of other trials added to the record. New York's attorney general and two Long Island counties tried a case against several manufacturers and distributors in state court in Suffolk County; most defendants settled during trial, and in December 2021 a jury found Teva, its subsidiaries, and Allergan liable for contributing to a public nuisance. Teva and Allergan subsequently resolved their liability through settlements. Table 1. Principal opioid public nuisance trials and their outcomes. Case Court and year Defendants at judgment Trial result Later history State of Oklahoma Cleveland County District Court, 2019 Johnson & Johnson Liability; about $465 million abatement Reversed, Oklahoma Supreme Court, 2021 People of California (Santa Clara and others) Orange County Superior Court, 2021 J&J, Teva, Endo, Allergan Judgment for defendants Plaintiffs later joined national settlements Huntington and Cabell County S.D. West Virginia, 2022 McKesson, Cardinal, AmerisourceBergen Judgment for defendants Appeal to Fourth Circuit; state court declined certified question Lake and Trumbull Counties N.D. Ohio (MDL), 2021 to 2022 CVS, Walgreens, Walmart Jury liability; about $650.6 million abatement Ohio Supreme Court, 2024: claims abrogated by statute City of San Francisco N.D. California, 2022 Walgreens Liability found Settled for a reported $230 million City of Baltimore Baltimore City Circuit Court, 2024 McKesson, AmerisourceBergen Jury award of about $266 million Post-trial remedy proceedings Why the results diverged The inconsistency of these outcomes is not simply a matter of some judges being more favorable to plaintiffs than others. Several structural factors explain much of it. The first is state law. Public nuisance is a creature of state law, and states differ in how far they have extended it beyond property. Oklahoma's highest court read its law narrowly as confined to property and criminal conduct. Ohio's legislature had written a statute that reached public nuisance claims involving products. California's courts, by contrast, had already accepted a product-promotion nuisance theory in the lead paint case. A national litigation strategy built on a single theory was, in this sense, a bet on fifty different bodies of law. The second is the defendant's position in the supply chain. The plaintiffs' strongest cases were against pharmacies, where the relevant decision, whether to fill a particular prescription, was made by the defendant's own employee at the point closest to the patient, and where the regulatory duty of corresponding responsibility was clear. Manufacturer cases turned on marketing, where proving that particular statements caused particular prescriptions was difficult, as the California court found. Distributor cases fell in between, and turned heavily on whether the fact-finder saw distributors as passive conduits filling orders written by doctors or as registrants with an independent duty to stop suspicious shipments. The third is the identity of the fact-finder. The plaintiffs' major verdicts in Ohio and Baltimore came from juries; their major losses in California and West Virginia came from judges. That pattern may be partly coincidental, and the Oklahoma and San Francisco cases went the other way, but it is consistent with the view that the evidence of pill volumes and corporate indifference was more persuasive to lay jurors than the legal arguments about causation and the scope of the tort were to judges. The fourth is causation evidence. The courts that ruled for defendants repeatedly emphasized the plaintiffs' failure to connect the defendant's conduct to the harm through the intervening decisions of prescribers, pharmacists, and users, and in some cases the transition to illicit fentanyl. The courts that ruled for plaintiffs accepted that a defendant need only be a substantial factor in creating the nuisance, and that its contribution need not be precisely measured. For the defendants, the pattern produced a strategic dilemma. They could, and did, win important cases. But they also lost some, and the losses could be enormous, because an abatement remedy measured by the cost of fixing a community's addiction crisis has no natural ceiling. They faced not six trials but thousands of pending cases in dozens of jurisdictions, some of them in states where the law was untested and juries were likely to be receptive. Every appellate win they secured came years after the verdict and applied in one state. The rational response to that exposure, for companies that could afford it, was a global settlement. The next chapter describes the procedural machinery that made one possible. Chapter 5: The Machinery of Aggregation By the end of 2017, several hundred opioid suits by cities and counties were pending in federal courts around the country, and many more were on their way. Many had been filed in state court and removed to federal court by defendants; others were filed in federal court directly. At the same time, dozens of state attorneys general were pursuing their own cases in their own state courts, which federal procedure could not consolidate. The story of how this sprawling mass became a set of national settlements is a story about procedure: about the federal multidistrict litigation system, a federal judge's determination to force a resolution, the limits appellate courts placed on his methods, and the eventual collaboration between the local-government lawyers and the state attorneys general. MDL 2804 Federal law provides a mechanism for handling large numbers of related cases. Under 28 U.S.C. § 1407, the Judicial Panel on Multidistrict Litigation may transfer civil actions pending in different federal districts that involve common questions of fact to a single district court for coordinated or consolidated pretrial proceedings. The transferee judge manages discovery and pretrial motions for all the cases together. In theory, each case returns to its original court for trial once pretrial proceedings are complete. In practice, the great majority of cases in large multidistrict litigations are resolved by settlement in the transferee court, and the MDL has become the principal way the federal system handles mass torts. In December 2017, the Panel created In re National Prescription Opiate Litigation, MDL No. 2804, and assigned it to Judge Dan Aaron Polster of the Northern District of Ohio in Cleveland. The MDL eventually encompassed well over three thousand cases, brought by counties, cities, tribes, hospitals, third-party payors such as union health funds, and others, against manufacturers, distributors, and pharmacies. Judge Polster made his priorities clear from the outset. At the first hearing in January 2018, he said publicly that he did not intend to spend years resolving legal issues and that his aim was to do something meaningful to abate the crisis within the year. He brought the parties together, including, informally, representatives of state attorneys general, and pressed them to negotiate. He also invited the Department of Justice, and later required the DEA to produce the ARCOS data, initially under a protective order and then, after the newspapers' challenge was upheld by the Sixth Circuit, subject to public release. Settlement did not come quickly. The defendants were unwilling to pay the sums the plaintiffs sought without a clearer view of their exposure, and the plaintiffs were a fragmented group, each with its own lawyers and political pressures. So Judge Polster set bellwether trials, the standard MDL technique of trying a few representative cases to give both sides information about what the rest are worth. The first, "Track One," was set for trial in October 2019, brought by Cuyahoga and Summit counties in Ohio against manufacturers, distributors, and pharmacies. On the eve of trial, the three large distributors and Teva settled with the two counties for a package valued at about $260 million, including about $215 million in cash from the distributors and donated addiction treatment medication from Teva. The trial was never held. Its role in the litigation was nonetheless significant: a trial date had forced a settlement, and the dollar figure for two counties gave both sides a benchmark for a national deal. The negotiation class and its rejection The largest obstacle to a global settlement was the fragmentation of the plaintiffs. A defendant that pays billions wants in return a release of all the claims against it. But in a litigation involving thousands of separate governments, each with its own claim and its own lawyer, any government that declined to join the deal could continue to sue. A settlement that left many holdouts was not worth much. In 2019, the plaintiffs' leadership proposed, and Judge Polster certified, a novel device: a "negotiation class." Rule 23 of the Federal Rules of Civil Procedure allows class actions for litigation or for settlement. The proposal was for a class consisting of all American counties and cities, certified for the purpose of negotiating a settlement. Under the proposal, members would decide at the outset whether to opt out; those who remained would be bound by any settlement that received a supermajority vote of class members under a pre-set allocation formula. The idea, developed with the help of the legal scholars Francis McGovern and William Rubenstein, was to give defendants the assurance of closure while protecting members through voting and a fixed allocation. The Sixth Circuit reversed in 2020. In In re National Prescription Opiate Litigation, a divided panel held that Rule 23 does not authorize certification of a class for the purpose of negotiation. The rule recognizes classes certified for litigation and, since 2018 amendments, classes proposed for settlement, but not a new category created by a district court. The court also expressed concern that the device could be used to pressure class members and that it was being used in part to exert leverage over the state attorneys general, who were not part of the class. Around the same time, the Sixth Circuit granted mandamus relief against Judge Polster on another issue, holding that he had improperly allowed the Track One plaintiffs to amend their complaints late in the case to add new claims against pharmacies. The court reminded him that an MDL judge must apply the Federal Rules just as any other judge must, however pressing the goal of settlement. The negotiation class failed, but the idea behind it survived. The national settlements that followed achieved closure through a different means: by structuring payments to depend on how many governments signed on, and by pairing sign-on incentives with state legislation that could bar future local suits. In that sense, the negotiation class was a rehearsal for the architecture described in the next chapter. The states and the problem of dual sovereignty The MDL could manage federal cases, but not the dozens of state attorney general suits in state courts, and in the opioid litigation the states were at least as important as the local governments. Attorneys general had statutory tools, such as consumer protection laws and civil penalties, that localities lacked. They could sue in state courts that the defendants could not remove to federal court. And they claimed a special position as the sovereign representative of the state's people, entitled in some states to control or displace claims brought by the state's own political subdivisions. This created a real conflict. Local governments argued that they bore many of the costs of the epidemic, from emergency response to jails to child welfare, and that they had begun suing earlier and were entitled to their own recoveries. States argued that they had primary responsibility for public health and Medicaid, and that uncontrolled local litigation would allow each county to negotiate separately, leading to inequitable results and letting defendants play governments off against one another. Some states argued that under their law, they alone had authority to sue for statewide public nuisance, and that settlements by the state could release the claims of their subdivisions. The defendants had every reason to encourage this tension and to insist that any settlement bind both levels. The solution, reached in the 2019 to 2021 negotiations, was to bring the states and subdivisions into a single deal. A group of attorneys general from both parties, with North Carolina, Pennsylvania, Tennessee, and Texas among the most prominent, negotiated alongside the plaintiffs' executive committee from the MDL. The deal would pay money to each state, and each state would then allocate it within the state, but only if the state and its subdivisions agreed to join. Local governments received a direct share and a voice in how state funds were spent. And the settlement's value to each state rose with the proportion of its subdivisions that signed on. Parallel tracks and pressure points Throughout the negotiation period, pressure came from several directions at once. Bellwether trials continued in the MDL, including the Lake and Trumbull County pharmacy trial described in the last chapter and cases against pharmacy benefit managers that followed later. Judge Polster remanded some cases, including the Huntington and Cabell County case and the San Francisco case, to their original federal courts for trial, spreading the pressure of trial dates across more venues. State court trials were proceeding in Oklahoma, California, New York, Washington, and elsewhere. The Purdue bankruptcy, filed in 2019, froze the claims against Purdue and, for a time, against the Sacklers. The result was an unusual procedural environment. No single court controlled the litigation. The MDL judge had the most cases but was limited by the Federal Rules and by appellate review. State court judges ran their own trials on their own schedules. The bankruptcy court controlled Purdue. The attorneys general answered to their own voters and legislatures. The national settlements were, in effect, a private contract among all these actors, designed to substitute for the unified adjudication that the courts could not provide. The other plaintiffs Governments were not the only claimants in the MDL, and the fate of the others reveals how much the litigation's structure favored public over private plaintiffs. Hospitals sued, arguing that they had borne the uncompensated cost of treating overdoses and opioid-related illness, particularly among uninsured patients. Third-party payors, including union health and welfare funds and self-insured employers, sued to recover what they had paid for opioid prescriptions and for the treatment of addiction among their members. Guardians of children born with neonatal abstinence syndrome, the withdrawal condition that affects infants exposed to opioids in the womb, sought medical monitoring and compensation on the children's behalf. And individuals who had become addicted, or families of those who died, brought personal injury and wrongful death claims. These plaintiffs faced the doctrinal obstacles described in Chapter 2 in their undiluted form. Hospitals and payors were classic derivative claimants, whose losses flowed from injuries to patients. The infants' claims required individualized proof of exposure and causation, and efforts to proceed as classes met the familiar resistance. Individual injury claims confronted the learned intermediary doctrine, misuse defenses, and limitations periods. They also faced a structural disadvantage. The national settlements were negotiated by and for governments; their releases covered governmental claims, and their money went to governmental programs. Private plaintiffs had to negotiate separately, from a weaker position, and with defendants who had already committed most of what they were willing to pay. In the Purdue, Mallinckrodt, and Endo bankruptcies, private claimants did receive dedicated trusts, but those trusts were small relative to the public ones. Some private claimants, particularly third-party payors, reached separate settlements with particular defendants. For the most part, however, the machinery of the opioid litigation was built for public plaintiffs, and private plaintiffs occupied its margins. Federally recognized tribes occupied a distinct position. As sovereigns, they brought public nuisance and related claims like other governments, but they were not subdivisions of the states and could not be bound by state-local agreements. Tribal leaders and their lawyers negotiated separate settlements with the distributors and J&J, and dedicated tribal shares were built into later agreements, with allocation among tribes by a formula of their own. Assessing the MDL The opioid MDL has attracted significant criticism from scholars of civil procedure. Critics, among them Elizabeth Chamblee Burch in her work on mass tort settlements, have argued that MDL judges who press single-mindedly for settlement can distort the incentives of the parties, marginalize the individual plaintiffs whose cases are being decided, and concentrate enormous power in a small group of lead lawyers whose fees depend on a global deal. In the opioid case, some observers questioned whether the judge's early public statements prejudged the strength of the claims, and whether the pressure to settle, rather than the merits of public nuisance law, drove the outcomes. The Sixth Circuit's repeated interventions reflected these concerns. Defenders respond that the alternative was worse. Without a strong coordinating judge, thousands of cases would have proceeded independently for many years, with inconsistent results, enormous transaction costs, and no money flowing to communities in the meantime. The ARCOS release, produced by the MDL, was an important public good in its own right. And while Judge Polster's inventions did not all survive appellate review, the eventual settlements were reached through negotiation among represented parties, including elected attorneys general, rather than imposed by the court. What is clear is that the MDL's role in the opioid litigation was less as an adjudicator than as an engine of settlement. It produced very few trials and only one jury verdict of its own, which was later undermined by the Ohio Supreme Court. It produced a great deal of discovery, several important procedural rulings, and a continuous threat of trial that helped bring the defendants to the table. That combination, a strong settlement engine and a weak body of substantive precedent, is characteristic of the opioid litigation as a whole. Chapter 6: Architecture of the National Settlements On July 21, 2021, a bipartisan group of state attorneys general announced the first of the great national opioid settlements. McKesson, Cardinal Health, and AmerisourceBergen would pay up to $21 billion over eighteen years, and Johnson & Johnson would pay up to $5 billion over no more than nine years, in exchange for releases of the opioid claims of states and local governments across the country. The agreement ran to hundreds of pages. Within eighteen months, a second wave followed: Teva, Allergan, CVS, Walgreens, and Walmart announced agreements that together were worth roughly $20 billion more. Kroger, the advertising firm Publicis Health, several generic manufacturers, and others followed. Together with the bankruptcy plans discussed in the next chapter, the opioid settlements have committed more than $50 billion to the governments that sued. These agreements were not simply large checks. They were a new kind of legal instrument: a contract designed to do what no court could, namely bind thousands of independent governments to a single resolution, allocate money among them by formula, dictate how most of it could be spent, and regulate the defendants' future conduct. This chapter explains how they work. The central problem: closure A defendant paying tens of billions of dollars needs assurance that it is buying peace. The failure of the negotiation class in the Sixth Circuit meant that no court could simply declare that all local governments were bound. And unlike a private class action, a settlement with a state attorney general does not automatically bind cities and counties that have their own claims and their own lawyers. Local governments are separate legal entities, and in many states they have independent authority to sue. The drafters of the distributor and J&J agreements solved this problem through a system of participation thresholds and incentive payments. The settlement would take effect only if enough states joined, and each participating state would receive its full allocation only if enough of its subdivisions joined too. The amount paid to a state was tied, through a sequence of incentive tiers, to measures such as the percentage of its litigating subdivisions that signed on, the percentage of its population living in participating subdivisions, and whether the state had taken legal steps, such as enacting a statute or obtaining a court ruling, to bar subdivisions that had not joined from suing in the future. Defendants retained the right to walk away if participation fell short. This design turned every state attorney general into a recruiter. Because the state's own recovery rose with local participation, attorneys general had strong incentives to persuade their counties and cities to join, typically by negotiating a state-local agreement giving localities a meaningful share of the money and a voice in how it would be spent. In several states, legislatures passed bar statutes or statutes ratifying such agreements. In others, attorneys general obtained releases from nearly every subdivision through intensive outreach. The approach worked: by the time the distributor and J&J deals became final in early 2022, all but a handful of states had joined, and subdivisions representing the great majority of the population had signed on. The second-wave deals used the same model and achieved similarly broad participation. The few non-participating jurisdictions, such as the city of Baltimore, retained their claims and continued to litigate, with the mixed consequences described in Chapter 4. Their existence was tolerated because the thresholds were set below unanimity; a defendant willing to pay for ninety-odd percent closure could absorb a small number of holdouts. Money: amounts, schedules, and conditions The major settlements, their headline values, and their payment periods are summarized in Table 2. The headline numbers deserve the cautions noted in the introduction. They represent maximums, available only if participation thresholds are met, and they are paid over periods ranging from a few years to eighteen. Their present value is significantly lower than the nominal total. Some payments include product rather than cash: Teva's settlement, for example, allowed states to take part of their share in the form of its generic naloxone, the overdose-reversal drug, valued at a wholesale price. And the agreements provide for attorneys' fees and costs, paid through separate funds that are reviewed by a fee panel, so that the amounts designated for abatement are net of those fees but the total cost to defendants is higher than the abatement figure alone. Table 2. Major national opioid settlements outside bankruptcy (headline values as announced). Defendant Announced Headline value Payment period Notable non-monetary terms McKesson, Cardinal Health, AmerisourceBergen 2021 Up to $21 billion 18 years Independent clearinghouse for order data; suspicious-order controls Johnson & Johnson (Janssen) 2021 Up to $5 billion Up to 9 years Exit from opioid sales for 10 years; no opioid promotion or lobbying Teva 2022 About $4.25 billion, including naloxone 13 years Marketing restrictions; option to take generic naloxone Allergan 2022 About $2.37 billion 7 years Marketing restrictions CVS 2022 About $5 billion 10 years Dispensing controls and red-flag monitoring Walgreens 2022 About $5.7 billion 15 years Dispensing controls and red-flag monitoring Walmart 2022 About $3.1 billion Under 6 years Dispensing controls and red-flag monitoring Kroger 2023 About $1.2 billion 11 years Dispensing controls Sources: announcements of the settling state attorneys general and the national settlement administrators; figures include amounts designated for tribes where announced together. Several other agreements belong to the same family. McKinsey & Company, which had advised Purdue on ways to "turbocharge" sales of OxyContin, according to documents made public in the litigation, agreed in 2021 to pay $573 million to resolve investigations by 49 states, the District of Columbia, and the territories, and later settled with local governments, school districts, and others. Publicis Health, a marketing agency that had worked for Purdue, agreed in 2024 to pay $350 million to the states. Federally recognized tribes negotiated their own settlements with the distributors and J&J, totaling about $590 million, together with separate tribal shares of later agreements, reflecting the disproportionate toll the epidemic took in many Native communities and the tribes' distinct sovereign status. Allocation: from the nation to the states A national settlement must decide how much goes to each state. The distributor and J&J agreements did this through a formula, set out in an exhibit to the agreements, that assigns each state a percentage share. The formula draws on public data measuring the burden of the epidemic in each state: the volume of opioids shipped into the state, the number of opioid overdose deaths, and the number of people with opioid use disorder, adjusted to account for population. The effect is to direct more money to states whose populations suffered more, rather than simply dividing by population. West Virginia, Kentucky, Ohio, and other hard-hit states received shares larger than their share of the national population; large but less affected states received smaller ones. The formula was the product of negotiation among the states and among the local-government lawyers, and it reflects compromises. Any formula based on historical shipment data rewards states where pills were oversupplied; any formula based on deaths rewards states where the fentanyl wave arrived early and hard. The drafters blended measures to smooth these differences. The virtue of a formula, whatever its imperfections, is that it could be agreed in advance, applied mechanically, and defended as neutral, avoiding a separate negotiation for each state. Allocation within the states and the restriction on uses Once the money reaches a state, it must be divided between the state and its subdivisions. The national agreements set a default, applicable only if a state and its subdivisions did not agree on something else: 15 percent to the state, 15 percent directly to participating subdivisions, and 70 percent to an abatement fund to be spent on opioid remediation across the state, with a role for local governments in deciding how. Most states replaced the default with their own state-subdivision agreement or statute, examined in Chapter 8. The most important innovation in the settlements, and the clearest departure from the tobacco model, is the restriction on uses. The distributor and J&J agreements require that the great bulk of the settlement money, at least 85 percent of payments to participating states over the life of the agreement, be spent on "opioid remediation," defined as care, treatment, and other programs designed to remediate the misuse of opioids or to prevent it. An exhibit to the agreements provides a long, nonexclusive list of approved uses. It begins with "core strategies," such as expanding access to naloxone, increasing the availability of medications for opioid use disorder like buprenorphine and methadone, treatment and recovery services for pregnant and postpartum women and babies with neonatal abstinence syndrome, "warm hand-off" programs that connect overdose survivors with treatment, treatment for incarcerated populations, prevention programs, and syringe service programs. It then lists many other permitted uses, including treatment and recovery services, housing and employment supports for people in recovery, programs connecting people with care, prevention education, law enforcement diversion programs, and data collection and research. A share, limited to 15 percent in the aggregate, can be used to reimburse governments for their past litigation-related costs or for other purposes. The restriction reflects the lesson of tobacco, where settlement money was largely absorbed into general budgets. It also reflects the legal theory of the cases: an abatement remedy in public nuisance is, by definition, directed at eliminating the nuisance, and restricting the money to abatement makes the settlement look like the equitable relief the plaintiffs had sought. The agreements provide for reporting and for oversight, though, as later chapters discuss, the practical enforcement mechanisms are relatively weak. Conduct terms Every major settlement included injunctive relief governing the defendants' future conduct, typically for ten years or more. These terms differ by defendant type. The distributors agreed to establish an independent clearinghouse, overseen by a third party, that would aggregate data from all three companies so that each could see a pharmacy's total opioid orders across distributors, closing a gap that had let pharmacies split orders to evade any single distributor's thresholds. They agreed to maintain controlled-substance compliance programs with defined elements, to use their data to identify and halt suspicious orders, to report them to regulators, and to be monitored for compliance. J&J agreed to stop selling opioids for ten years, not to promote opioids or fund third parties that promote them, and not to lobby on opioid-related legislation. The pharmacy chains agreed to maintain systems for identifying red-flag prescriptions, to give pharmacists authority and support to refuse them, not to base pharmacist compensation on the volume of opioids dispensed, and to report suspicious activity. Several agreements also required the defendants to make internal documents publicly available, feeding an archive maintained by the University of California, San Francisco, and Johns Hopkins University that now holds millions of pages. These terms matter because they reach the conduct that the litigation identified as wrongful, and because, unlike the money, they operate directly on the defendants rather than depending on governments' spending choices. Their long-term effect depends on monitoring and enforcement, which falls largely to the attorneys general. The settlements as private legislation Stepping back, the national settlements look less like the resolution of a lawsuit than like a piece of legislation. They set up a national financing scheme for opioid remediation. They allocated money among the states by formula. They created, in effect, a spending mandate with a list of permitted uses. They imposed a regulatory regime on a sector of the pharmaceutical industry. And they did all this through agreements negotiated by elected attorneys general, local-government lawyers, and corporate defendants, without legislative enactment at the federal level, and with judicial involvement limited to approval of consent judgments in state courts. This is both the achievement and the problem. The achievement is that an enormous amount of money was committed to addressing the epidemic, faster and more coherently than case-by-case adjudication could ever have done. The problem is one of legitimacy and accountability. The agreements' terms were set by lawyers and officials under the pressure of litigation risk, rather than by a legislature responsive to the public. The legal theory underlying them was never validated by the courts that considered it most carefully. And the terms of the settlements were shaped by the interests of the parties at the table, including the plaintiffs' lawyers, whose fees depended on participation. Scholars such as Nora Freeman Engstrom and Robert Rabin, comparing the opioid and tobacco settlements, have noted that litigation can achieve public-health goals that legislatures fail to reach, but at the cost of placing major policy decisions in private negotiations. Whatever one's view, the national settlements are now the governing framework for the financing of opioid remediation in most of the country. The one major defendant they did not reach was the one most closely identified with the epidemic: Purdue Pharma, whose path ran through bankruptcy court. Chapter 7: Bankruptcy, the Sacklers, and the Limits of Release For the manufacturers most exposed to opioid liability, the national settlement model was not the only way out. Several of them used Chapter 11 of the Bankruptcy Code instead, and bankruptcy became the second great arena of the litigation. It offered something that no negotiated settlement could: the power of a federal court to bind every creditor, including those who objected, to a single plan. The most important and most controversial of these cases was that of Purdue Pharma, where the question of whether bankruptcy's binding power could extend to the Sackler family, who owned the company but did not themselves file for bankruptcy, reached the Supreme Court. The Court's answer, in Harrington v. Purdue Pharma L.P. in 2024, reshaped not only the opioid litigation but the use of bankruptcy for mass torts generally. Why bankruptcy Chapter 11 has become a common destination for companies facing mass tort liability, from asbestos manufacturers in the 1980s to makers of defective medical products and, more recently, organizations facing sexual abuse claims. Its appeal lies in several features. The automatic stay halts all litigation against the debtor the moment a petition is filed. The bankruptcy court can estimate and channel all claims into a trust, so that claimants look to the trust for payment and the reorganized company emerges free of them. A plan can be confirmed over the objection of some creditors if the requisite majorities vote in favor and statutory protections are met. And the process centralizes in one court what would otherwise be scattered across thousands. For a company like Purdue, facing more than two thousand lawsuits from states, localities, tribes, hospitals, and individuals, these were compelling advantages. But bankruptcy also offers creditors something: a forum in which the company's finances are exposed, prior transfers can be investigated and clawed back, and the distribution of the estate among creditors is supervised by a court. The Purdue case Purdue filed for Chapter 11 in September 2019 in the Southern District of New York, in White Plains, where the case was assigned to Judge Robert Drain. The filing followed a tentative settlement framework reached with a group of states and local governments, under which the Sackler family, who had owned Purdue since the 1950s, would give up ownership and contribute several billion dollars, and the company would be converted into an entity operated for the public benefit. The central issue from the start was the Sacklers. Investigations by the company's own special committee and by creditors showed that the family had taken roughly $10 billion out of Purdue between 2008 and 2017, a period that began shortly after the company's 2007 guilty plea. Part of that went to taxes and part was reinvested outside Purdue, but the family retained billions. Many of the states and other plaintiffs had sued individual Sacklers directly, alleging personal participation in the company's deceptive marketing. The Sacklers were not bankrupt and did not file. They were willing to contribute to the plan, but only in exchange for a release from all opioid-related civil liability, including the claims of creditors who did not agree. In 2020, in parallel with the bankruptcy, Purdue pleaded guilty to three federal felonies, and the Sacklers separately agreed to pay $225 million to resolve the federal government's civil claims against them, without admitting wrongdoing. The criminal fine and forfeiture against Purdue, nominally over $8 billion, were largely symbolic given the company's assets; the government agreed to accept only a fraction in cash in order to allow the bankruptcy plan to go forward and the value of the company to flow to its creditors. In September 2021, Judge Drain confirmed a plan. The Sacklers would pay about $4.5 billion over roughly nine years; Purdue would be dissolved and its assets transferred to a new company whose profits would go to abatement; the funds would be divided among trusts for states and localities, tribes, hospitals, third-party payors, and individuals; and the Sacklers would receive a broad release of civil claims relating to Purdue and opioids, binding on all creditors, whether or not they consented. The plan had been approved by large majorities of the voting creditors, but a significant group of objectors, including several states, the District of Columbia, and the United States Trustee, the Justice Department official responsible for overseeing bankruptcy cases, argued that the release was illegal. In December 2021, Judge Colleen McMahon of the federal district court vacated the confirmation order, holding that the Bankruptcy Code does not authorize a court to grant non-consensual releases of third parties who are not themselves debtors. The objecting states then negotiated a larger payment from the Sacklers, raising their contribution to between $5.5 and $6 billion, and dropped their objections. In May 2023, the Second Circuit reversed Judge McMahon, holding that such releases are permissible in appropriate circumstances and listing factors to guide courts in deciding whether they are warranted. The United States Trustee sought review in the Supreme Court, which agreed to hear the case and stayed the plan pending its decision. Table 3. The Purdue bankruptcy, from filing to revised plan. Date Event Significance September 2019 Purdue files Chapter 11 Litigation against Purdue stayed; Sackler release sought November 2020 Purdue pleads guilty to three federal felonies Criminal liability established for the company September 2021 Bankruptcy court confirms plan Non-consensual release of Sacklers approved December 2021 District court vacates confirmation Code held not to authorize non-debtor releases May 2023 Second Circuit reverses district court Releases permitted under a multi-factor test June 2024 Supreme Court decides Harrington v. Purdue Pharma Non-consensual third-party releases held unauthorized 2025 Revised plan of about $7.4 billion approved Releases limited to consenting creditors Harrington v. Purdue Pharma The Supreme Court decided the case on June 27, 2024, by a vote of five to four. Justice Gorsuch wrote for the majority, joined by Justices Thomas, Alito, Barrett, and Jackson. Justice Kavanaugh wrote the principal dissent, joined by Chief Justice Roberts and Justices Sotomayor and Kagan. The majority held that the Bankruptcy Code does not authorize a release and injunction that, as part of a Chapter 11 plan, effectively discharges claims against a non-debtor without the consent of the affected claimants. The argument turned on statutory text. The plan's proponents relied on Section 1123(b)(6), which allows a plan to include "any other appropriate provision not inconsistent with the applicable provisions of this title." The majority read that catchall in light of the specific provisions that precede it, all of which concern the debtor's own rights and property, and concluded that it could not be stretched to authorize something so different in kind: extinguishing claims against a party that had not filed for bankruptcy and had not placed its assets at the disposal of creditors. The majority emphasized that the Code's discharge is available only to debtors who submit to the process, including its requirements of disclosure and its limits on what debts can be discharged; the Sacklers sought the benefits of a discharge, including for claims of fraud and willful injury that the Code would not discharge even for a debtor, without those burdens. The Court was careful to note what it was not deciding: it did not address consensual releases, or what counts as consent, and did not call into question plans that had already been consummated. The dissent argued that the decision was devastating for the victims. The plan had been supported by the overwhelming majority of creditors, including most states and individual victims, and by rejecting it, the dissenters said, the Court was depriving victims of a substantial, guaranteed recovery and exposing them to years of uncertain litigation against the Sacklers, in which the first to sue could exhaust the family's reachable assets. The dissent also pointed to the long practice of bankruptcy courts approving such releases in mass tort cases where they were necessary to secure contributions from parties such as insurers and affiliates, and to the practical impossibility of reaching a global settlement if any individual creditor could refuse. The disagreement captured the tension at the center of the entire opioid litigation. The majority insisted on legal form: a court may not extinguish a person's claims against a third party without that person's consent unless Congress has clearly authorized it. The dissent insisted on practical results: a comprehensive resolution that most victims wanted and that delivered billions for abatement. Both were right about something. A regime in which wealthy non-debtors can buy releases from all claimants through someone else's bankruptcy invites abuse, and scholars such as Lindsey Simon had documented the growing use of such releases by non-debtors who had not submitted to the bankruptcy process. But requiring individual consent in a case with hundreds of thousands of claimants makes global settlement much harder. After Harrington The decision sent the parties back to negotiation. In early 2025, Purdue, the Sacklers, and a broad coalition of states and other creditors announced a new agreement. Under it, the Sacklers would pay up to about $6.5 billion over fifteen years and Purdue would contribute about $900 million, for a total of roughly $7.4 billion, significantly more than under the 2021 plan. Crucially, the releases of the Sacklers would apply only to creditors who consented; those who did not would retain their claims against the family. The plan also provided for Purdue to be transformed into a new company, owned by a trust and directed to operate in the public interest, and for the release of a large volume of Purdue documents to the public. The plan was put to a vote, and in November 2025 the bankruptcy judge, Sean Lane, who had succeeded Judge Drain on the case, announced that he would approve it. The revised plan shows both the practical effect of Harrington and its limits. The Sacklers paid more, suggesting that the non-consensual release had been worth something to them and that its unavailability shifted bargaining power toward creditors. Creditors who wished to preserve their claims could do so. But the overwhelming majority of creditors consented, because the plan offered a certain recovery, and because pursuing individual claims against wealthy defendants with assets held in trusts and abroad was costly and uncertain. The outcome looked, in substance, much like the original plan. Mallinckrodt and Endo Two other opioid manufacturers went through bankruptcy with less public attention. Mallinckrodt, one of the largest makers of generic oxycodone, filed for Chapter 11 in Delaware in 2020, with a plan that created a trust to receive $1.6 billion in payments over several years for opioid claimants. The company emerged in 2022, but filed for bankruptcy again in 2023, in a proceeding that sharply reduced the remaining payments owed to the opioid trust. The episode illustrated the risk inherent in any settlement paid over time by a financially weak company: the long payment schedule that makes the headline number possible also exposes it to the company's future insolvency. Endo, maker of Opana, filed in 2022 in the Southern District of New York. Its plan, confirmed in 2024, sold its business to its lenders and established trusts for opioid claimants, including public and private creditors, while a subsidiary pleaded guilty to a federal misbranding charge relating to Opana ER. Rite Aid, a pharmacy chain that faced opioid claims among its many liabilities, also filed for bankruptcy in 2023. What bankruptcy offered, and what it cost For all its controversy, bankruptcy produced real value for opioid creditors. It centralized claims against insolvent or near-insolvent companies that could not have paid all of their creditors in the tort system, ensured equal treatment among similarly situated creditors, and forced disclosure of the companies' finances and the owners' withdrawals. Without bankruptcy, the race to judgment in thousands of courts would likely have left many plaintiffs with nothing. But bankruptcy also carried costs peculiar to mass torts. It stopped the litigation that had been producing public evidence and replaced trial with a negotiated plan, so that the Sacklers were never subjected to a public adjudication of their personal responsibility, though they did testify before Congress in 2020 and before the bankruptcy court. It required years of proceedings and hundreds of millions of dollars in professional fees, paid from the estate. And it created the temptation, which Harrington now forbids, to extend the benefits of bankruptcy to those who had not borne its burdens. Individual victims fared the worst. Personal-injury claimants, those who had become addicted or lost family members to overdose, were a small part of the Purdue plan's economics. The trust for individual claimants was a small fraction of the total, and the payments to individuals were expected to be modest, often in the thousands of dollars. The vast majority of the money went to governments for abatement. That allocation reflects the structure of the litigation as a whole, which is taken up in the next chapter: the opioid settlements were designed to fund public programs, not to compensate the people most directly harmed. Chapter 8: Following the Money Winning or settling a lawsuit usually ends the plaintiff's work. In the opioid litigation, it began a second and in some ways harder task. The settlements committed tens of billions of dollars to be paid over up to eighteen years, into the accounts of fifty states, thousands of counties and cities, and hundreds of tribes, for the purpose of abating a public-health crisis whose character was changing even as the agreements were signed. How that money is divided, governed, and spent will determine whether the litigation achieves anything lasting. This chapter follows the money from the national settlement down to the local program, and examines the choices, disputes, and failures that have marked its first years. The design problem The architects of the settlements faced a problem without a good precedent. Tort damages normally go to the person harmed, who decides how to use them. Abatement orders in traditional nuisance cases direct the defendant to remove a specific condition. Neither model fit a situation in which defendants would pay money to governments to fund an open-ended program of treatment, prevention, and recovery. The tobacco Master Settlement Agreement offered only a cautionary tale. Its payments went into state general funds without meaningful restriction. Public-health organizations tracking the money found year after year that states spent only a small fraction, in most years only a few percent, of their tobacco revenue on tobacco prevention and cessation programs. Some states used it to cover budget shortfalls, fund unrelated projects, or, through securitization, sell off their rights to future payments for a lump sum at a discount. The tobacco money did much good in some places, but as an instrument of tobacco control it underperformed its promise. The opioid settlements responded with three design choices, described in Chapter 6: most of the money was restricted to opioid remediation; local governments received a guaranteed share and a role in decisions; and states were pushed to adopt formal agreements governing the division and use of funds. The result is a system in which the allocation of each state's money depends on that state's particular arrangements. State-local allocation models States took several distinct approaches to dividing and governing the money within their borders. Four broad models can be distinguished, as Table 4 sets out, though many states combine features of more than one. Table 4. Principal models for allocating opioid settlement funds within states. Model Example Who decides spending Principal strength Principal risk Settlement default split National agreement terms (15% state, 15% local, 70% abatement fund) State, localities, and abatement fund governance Applies automatically Leaves governance details unresolved Independent foundation OneOhio Recovery Foundation (55% foundation, 30% local, 15% state) Foundation board with regional representation Insulated from annual budget politics Slower distribution and weaker public accountability Local-majority formula North Carolina memorandum of agreement (80% to local governments) County and municipal governments Money reaches communities closest to the harm Uneven local capacity and oversight State-directed fund with advisory body Statutory opioid settlement funds with advisory councils in several states Legislature and state agencies, advised by experts Statewide planning and coordination Pressure to substitute for existing state spending Ohio's approach, adopted through the OneOhio memorandum of understanding in 2020 and later implemented by state law, directed 55 percent of the money to a new private nonprofit foundation, 30 percent to local governments, and 15 percent to the state. The foundation is governed by a board that includes representatives of regions across the state, as well as state officials, and it distributes money through regional structures. Ohio's model aimed to insulate the largest share of the money from annual legislative appropriation and local political pressures. Its critics, including journalists and some local officials, have raised concerns about how quickly the foundation has distributed money and how transparent its decisions are, since a private foundation is not subject to the same public-records and open-meetings rules as a government agency. North Carolina's memorandum of agreement, by contrast, sent 80 percent of its settlement money directly to local governments, reserved 15 percent for the state, and set aside the remaining 5 percent as an incentive fund to encourage participation. The state provided a detailed menu of approved strategies, drawn from the national list, and required local governments to adopt a process for choosing among them, to report their spending publicly, and to follow specific rules about planning and collaboration. The model trusts local governments, closest to the problem, to decide, while supplying structure and transparency requirements. Other states chose to route money through statutory funds controlled by the legislature, often with advisory councils of public-health experts, people in recovery, and local officials. The advantage is statewide coordination; the risk is that the money becomes one more line in the state budget, subject to the familiar temptation of using it to replace spending that would have happened anyway. What the money can buy The national agreements' list of approved uses, described in Chapter 6, is broad. It includes the evidence-based core of addiction treatment, including medications for opioid use disorder such as buprenorphine and methadone, which reduce overdose mortality substantially; naloxone distribution; treatment for people in jails and prisons, who face a very high risk of overdose death in the weeks after release; programs for pregnant women and newborns; and harm-reduction measures such as syringe service programs. It also includes a wide range of other uses: prevention programs in schools, recovery housing, employment support, first-responder training, and law enforcement diversion programs. The breadth of the list was deliberate: the drafters did not want to lock governments into a set of programs that might become outdated as the epidemic changed, and different communities have different needs. But it left substantial discretion, and discretion has produced controversy. Early public-health guidance tried to shape how that discretion was exercised. In 2021, a group convened by the Johns Hopkins Bloomberg School of Public Health published "Principles for the Use of Funds From the Opioid Litigation," endorsed by dozens of organizations. Its five principles urged governments to spend the money to save lives; to use evidence to guide spending; to invest in youth prevention; to focus on racial equity; and to develop a fair and transparent process for deciding where to spend the funding. The principles have been widely cited by state and local officials, but they are not binding. The spending disputes As money began to flow in 2022 and 2023, journalists and researchers began to track how it was being used. KFF Health News, in collaboration with local newsrooms and with the Johns Hopkins researchers, investigated the choices of state and local governments; the Opioid Settlement Tracker, maintained by the lawyer and researcher Christine Minhee, compiled the terms of state agreements and flagged questionable uses; and several state auditors and legislative committees began their own reviews. The picture they found was mixed. Many governments spent the money on programs closely aligned with the evidence: expanding access to medications for opioid use disorder, funding naloxone distribution, supporting treatment in jails, and building recovery services. Others spent on items whose connection to abatement was contested or tenuous. Reports documented the use of settlement funds for law enforcement equipment and vehicles, for jail construction and operations, for general public-safety programs, and for filling gaps in existing budgets. Some of these uses could be defended under the broad list of approved uses; others were difficult to reconcile with it. In some jurisdictions, money sat unspent for long periods, because governments lacked plans, staff, or confidence about what they were permitted to do. The disputes reflect a deeper disagreement about what abatement means. Public-health advocates emphasize treatment and harm reduction, and point to evidence that medications for opioid use disorder and naloxone reduce deaths. Others, particularly some local officials and law enforcement agencies, see the epidemic partly as a problem of drug trafficking and crime, and view policing and incarceration as legitimate abatement. Harm-reduction programs such as syringe services remain politically contested in some places, even though the settlement agreements expressly permit them. The settlements allow room for these disagreements, and the resulting choices vary with local politics. The other persistent concern is supplantation: the use of settlement money to replace existing government spending rather than to add to it. If a state cuts its own addiction treatment budget and fills the gap with settlement funds, the program continues but the settlement produces no net increase in treatment. Supplantation is hard to prove and hard to prevent, and most agreements do not directly prohibit it. The tobacco experience shows how easily earmarked money becomes general revenue over time. The lawyers' share Any account of where the money went must include the lawyers. Most local governments retained private law firms on contingency, typically promising a percentage of any recovery, and the attorneys general used a mix of their own staff and outside counsel. On the scale of recoveries involved, conventional contingency percentages would have produced fees in the billions and diverted a large share of abatement money to counsel. The national settlements addressed this by creating separate funds for attorneys' fees and costs, paid by the defendants over time alongside the abatement payments, and by providing for a fee panel to allocate them among the many firms that had worked on the cases. In exchange, counsel who drew from these funds agreed to limits on what they could collect under their individual contingency contracts, and the MDL court imposed an assessment on recoveries by cases in the federal litigation to compensate the lead lawyers whose work benefited everyone. The resulting fees are still very large in absolute terms; the combined fee funds in the national settlements run to billions of dollars over their lives. But the structure keeps most fees out of the abatement money itself, and it has made the fee arrangements more visible than in most mass litigation. The fee structure has its critics. Some argue that it rewarded lawyers for signing clients up to the settlements, since fees depended on participation, and so gave counsel a stake in persuading governments to accept terms that might not have been in their interest. Others argue that without the contingency-fee bar, few local governments could have afforded to sue at all, and that the lawyers bore substantial risk in pursuing a theory that, as the trial record shows, often lost. Both points have force. The opioid fee arrangements are, at a minimum, an improvement on the tobacco settlement, whose fee awards to a small number of firms became notorious. Oversight and accountability The settlement agreements themselves contain relatively limited enforcement mechanisms for spending restrictions. They require reporting of spending on non-remediation purposes and, in some cases, provide for audits, but the parties with standing to enforce them are principally the defendants and the settling governments themselves, and neither has strong incentives to police how governments spend the money. The defendants have little interest once they have paid, and the governments are the ones spending. In practice, oversight depends on state-level mechanisms: the terms of state-subdivision agreements, state statutes, advisory councils, state auditors, and public reporting requirements. These vary widely. Some states require detailed public reporting by every recipient; others require little. Research by public-health groups has found that transparency in many states is limited, making it difficult for the public to know how the money is being spent. Journalism and civil society have filled some of the gap. Reporting on questionable uses has prompted some governments to change course, and advocacy by people in recovery and families who lost members to overdose has pressed officials to align spending with evidence. Some states have strengthened their reporting requirements in response. The long payment schedules create continuing opportunities for correction, but also continuing opportunities for drift. Who is left out The most fundamental feature of the distribution system is who does not receive money. The settlements were designed to fund governments, not to compensate individuals. People who became addicted, families who lost members, and children who grew up in foster care because their parents were addicted receive no direct share of the national settlements, although some may benefit from the programs they fund. Individual claimants did receive some compensation through the bankruptcy trusts described in the previous chapter, but those amounts were small relative to the harm and to the total sums involved. This outcome follows from the legal theory. Public nuisance, the vehicle of the governmental claims, is a remedy for harm to the public, not to individuals, and abatement is a forward-looking remedy aimed at the condition rather than compensation for past injury. The individual claims that might have produced compensation had largely failed, for the reasons explained in Chapter 2. The system that emerged thus reflects a choice, largely made by the structure of the law, to address the epidemic as a public problem requiring public investment rather than as the sum of individual injuries requiring compensation. There are reasons to defend that choice. Money spent on treatment and prevention may do more good, over time, than the same money divided into individual payments, and the population of people harmed by the epidemic is too large and too diffuse to compensate adequately in any case. But it is a choice with costs. It leaves the most directly harmed people without a remedy, and it makes the value of the settlements depend on the quality of government decision-making, which, as the spending disputes show, is uneven. Chapter 9: Accountability Beyond Money The title of this book pairs corporate accountability with mass torts and public nuisance, and the pairing deserves scrutiny. It is easy to measure the opioid litigation by its dollar totals, and those totals are large. It is harder to say whether the companies and individuals responsible for the epidemic were held accountable in any fuller sense: whether wrongdoing was established, whether the people who made the decisions faced consequences, whether conduct changed, and whether the next company contemplating a similar course will be deterred. This chapter examines the forms of accountability the litigation and its parallel proceedings produced, and their limits. What accountability might mean Accountability has several components, and they do not always travel together. There is the finding of responsibility: a public, authoritative determination that a person or company did wrong. There is sanction: a cost imposed because of the wrong, sufficient to express condemnation and to deter. There is repair: compensation or restoration for those harmed. There is reform: changes in conduct that prevent recurrence. And there is truth: a public record of what happened and why. The opioid litigation delivered these in uneven measure. It produced enormous repair payments, though largely to governments rather than individuals. It produced some reform through injunctive terms. It produced a substantial public record. It produced relatively few authoritative findings of responsibility and relatively few sanctions on individuals. The national settlements, like most civil settlements, included no admission of liability; the defendants denied wrongdoing, and continue to do so. Criminal accountability The criminal law is the traditional instrument for authoritative findings of wrongdoing and for sanctions aimed at culpable individuals. Its role in the opioid crisis was significant but narrow. Purdue pleaded guilty twice, in 2007 and 2020, and the second plea involved three felonies. These are authoritative admissions of criminal conduct by the company. But the 2007 plea was followed by further misconduct, suggesting that the penalty had not deterred, and the 2020 plea was largely symbolic in financial terms, because the government subordinated much of its claim to the bankruptcy plan. No member of the Sackler family was criminally charged. The Department of Justice's 2020 civil settlement with the Sacklers, for $225 million, resolved claims under the False Claims Act and related statutes without any admission, and the Justice Department's own announcement made clear that it had not resolved all potential criminal liability. No criminal charges against any Sackler have been brought. The most successful individual prosecutions came at the extremes. John Kapoor, founder of Insys Therapeutics, and several other Insys executives were convicted in federal court in Boston in 2019 of racketeering conspiracy for a scheme that bribed doctors, through sham speaker fees, to prescribe the company's fentanyl spray and deceived insurers into paying for it. Kapoor was sentenced to sixty-six months in prison. Insys itself pleaded guilty and filed for bankruptcy. In 2019, federal prosecutors in Manhattan brought the first criminal drug trafficking charges against a pharmaceutical distributor and its executives, charging Rochester Drug Cooperative, then one of the country's largest regional distributors, and its former chief executive, Laurence Doud, with conspiring to distribute controlled substances. The company entered a deferred prosecution agreement; Doud was convicted at trial in 2023. Thousands of physicians and pharmacists have been prosecuted over the years for operating so-called pill mills, and state medical boards have revoked many licenses. McKinsey & Company provides the most recent example. In December 2024, McKinsey entered a deferred prosecution agreement with the Justice Department and agreed to pay $650 million to resolve criminal and civil investigations into its work for Purdue, admitting that it had helped Purdue with its efforts to sell OxyContin through strategies that contributed to misbranding. A former McKinsey partner, Martin Elling, pleaded guilty to obstruction of justice for destroying records relating to the Purdue work. McKinsey had earlier announced that it would stop advising on opioid-specific business. Several features limit criminal law's reach. Corporate criminal liability in the United States is broad in theory but usually resolved by plea or deferred prosecution agreement, often with modest fines relative to the company's size and with the company surviving. Individual executives in large organizations are hard to prosecute, because decisions are diffused across committees and layers of management, and prosecutors must prove individual knowledge and intent beyond a reasonable doubt. The crimes that fit best, such as misbranding under the food and drug laws, carry limited penalties for individuals unless fraud can be proved. And the large distributors and pharmacy chains, whose conduct was at the center of the civil litigation, have faced civil and administrative enforcement but no criminal charges comparable to those in the Rochester Drug Cooperative case. Civil and regulatory enforcement Between the criminal law and the private litigation lies the regulatory apparatus. The DEA brought administrative actions against distributors and pharmacies throughout the 2000s and 2010s, including registration suspensions and civil penalties, but its enforcement was episodic and its political support fluctuated. In December 2020, the Justice Department filed a civil suit against Walmart under the Controlled Substances Act, alleging that the company's pharmacies had filled thousands of invalid prescriptions and that its distribution centers had failed to report suspicious orders. The FDA imposed a risk evaluation and mitigation strategy on extended-release and long-acting opioids in 2012, requiring manufacturers to fund prescriber education, and later extended it to immediate-release opioids. It approved abuse-deterrent formulations and, in 2017, requested the withdrawal of the reformulated Opana ER from the market because its abuse by injection was linked to outbreaks of HIV and hepatitis C. The CDC published guidelines for prescribing opioids for chronic pain in 2016, revised in 2022. Regulatory responses changed the environment in which the defendants operated. Opioid prescribing in the United States peaked around 2012 and has fallen substantially since, driven by prescriber education, state prescription drug monitoring programs, guidelines, insurance restrictions, and the reputational effects of the litigation itself. But regulatory enforcement is prospective and partial; it does not assign responsibility for the harm already done. Reform through settlement terms The conduct terms of the national settlements, described in Chapter 6, are the litigation's most direct contribution to reform. The distributor clearinghouse, the pharmacy red-flag requirements, and J&J's withdrawal from the opioid market address the practices the plaintiffs identified as wrongful. They are enforceable by the states in court, and they last for a decade or more. Their limits are also clear. By the time they took effect, prescription opioids were no longer the main driver of overdose deaths; illicit fentanyl was, and the settlement terms have little bearing on the illicit market. Opioid prescribing had already fallen sharply. Some of the reforms codified practices that the defendants had already adopted under regulatory and litigation pressure. And the terms' effectiveness depends on monitoring and enforcement by the states, which have limited resources for the task. There is also a real concern about overcorrection. Clinicians and patient advocates have documented cases in which patients with legitimate chronic pain, some on long-term opioid therapy, found it difficult to obtain their prescriptions filled or were abruptly tapered, as pharmacies and prescribers became wary of scrutiny. The FDA and the CDC have issued warnings against abrupt discontinuation, and the 2022 CDC guideline explicitly cautioned against rigid application of dosage thresholds. The injunctive terms were designed to target inappropriate dispensing, not legitimate care, but distinguishing the two is the very judgment that the litigation showed to be difficult. The public record The litigation's most durable contribution to accountability may be the public record it created. Discovery in the MDL and in state cases produced millions of documents, many of which became public through court filings, trials, and the document-disclosure terms of the settlements. The Opioid Industry Documents Archive, maintained by the University of California, San Francisco, and Johns Hopkins University, now makes a large part of that record available to researchers, journalists, and the public. The ARCOS data release allowed anyone to see where pills went. Congressional investigations, particularly by the House Energy and Commerce Committee and the House Oversight Committee, which questioned members of the Sackler family in 2020, added to the record. Journalists and authors turned that record into a public narrative. Books such as Barry Meier's Pain Killer, Sam Quinones's Dreamland, Beth Macy's Dopesick, Eric Eyre's Death in Mud Lick, and Patrick Radden Keefe's Empire of Pain documented the conduct of the companies and families involved, and several were adapted for television. The public understanding of the opioid epidemic as a crisis caused in significant part by corporate conduct, rather than solely by individual failings or illicit drug trafficking, owes a great deal to this combination of litigation and journalism. The record has also had reputational consequences that no court ordered. Museums and universities, including the Louvre, the Metropolitan Museum of Art, the Tate galleries, and Tufts University, removed the Sackler name from galleries and buildings between 2019 and 2023. Activists, notably the artist Nan Goldin and the group she founded, drew attention to the family's philanthropy as a form of reputation laundering. Reputational sanctions of this kind are informal and uneven, but for a family that had built its public identity on philanthropy, they were not trivial. The unfinished defendants Accountability has also been uneven across the supply chain. The national settlements reached the manufacturers, distributors, and chain pharmacies that were the first targets of the litigation. Later rounds of litigation turned to parties further from the pill bottle. Pharmacy benefit managers, the companies that administer prescription drug benefits for insurers and employers and that negotiate with manufacturers over which drugs are covered and on what terms, became defendants in the later phases of the MDL and in suits by state attorneys general. The theory against them is that they had access to vast quantities of dispensing data that could have revealed patterns of abuse, that they designed formularies and utilization rules that facilitated high-volume opioid prescribing, and in some cases that they accepted payments from manufacturers in exchange for favorable coverage of opioids. The largest benefit managers, including Express Scripts and OptumRx, deny the claims and argue that they merely administered coverage decided by their clients and that they, too, relied on prescribers' judgment. The federal MDL court selected bellwether cases against them, and those proceedings continued through 2025 and into 2026. Whether they end in a further round of national settlements will say much about how far public nuisance-style theories can travel from the core of the supply chain. Consultants and marketing agencies, such as McKinsey and Publicis Health, were reached through a combination of attorney general investigations and criminal enforcement rather than through the MDL's nuisance claims. Their cases matter because they extend accountability to firms that did not make or handle any opioid but helped design the strategies used to sell them. The legal theory in these cases rests less on nuisance than on consumer protection and on complicity in the manufacturer's misconduct. Prescribers are the most obvious unfinished defendants. The physicians who wrote prescriptions were the legally necessary link in the chain, and the defendants argued throughout that responsibility belonged with them. But the plaintiffs in the governmental cases generally did not sue doctors, who could not have paid judgments of the relevant size, and whose conduct was more naturally addressed through licensing boards and criminal prosecution of the most egregious. The result is a peculiar asymmetry: the parties whose decisions were closest to each patient were largely outside the civil litigation, while the companies at a distance paid the bulk of the money. Deterrence The final question is whether the litigation will deter future corporate conduct of the kind it targeted. The answer depends on how companies weigh the expected costs of such conduct against its expected profits, and on whether decision-makers expect to bear the costs personally. On the monetary side, the settlements are large in absolute terms, but they must be measured against the revenues the companies earned from opioids over two decades, spread over many years, and in many cases partly deductible or covered by insurance. The distributors and pharmacy chains remain among the largest companies in the United States. For companies of that size, the settlements are a significant but survivable cost. Purdue, by contrast, was destroyed as an independent company, and the Sacklers lost ownership, though they kept a large part of their wealth. On the individual side, deterrence is weaker. Few executives faced personal consequences. The Sackler family's experience, in which billions were withdrawn from the company, criminal charges were never brought, and a civil settlement was eventually reached on terms that left the family wealthy, may teach future actors that the personal risks of aggressive conduct are manageable. A further lesson may be less intended. Companies may learn that the most important variable is the legal theory that can be brought against them, and that public nuisance, after its setbacks in Oklahoma, Ohio, and West Virginia, is uncertain enough to be worth fighting. Or they may learn that even an uncertain theory, pressed in enough courts by enough governments, produces settlements in the tens of billions. Which lesson prevails depends on what happens next in the courts, which is the subject of the conclusion. Conclusion: The Opioid Model and Its Inheritors The opioid litigation ended, for most defendants, not with a verdict but with a check, a set of conduct rules, and a release. Looking back across it, the pattern is clear enough to state plainly. The governments that sued did not persuade the courts that looked hardest at their legal theory. They persuaded the defendants that they could not afford to find out who was right. The result was one of the largest transfers of money in the history of American civil litigation, directed not to the people most directly harmed but to public programs, governed by private agreements whose terms have the reach of legislation. That outcome has three consequences worth drawing out, because they matter for what comes next. Public nuisance after opioids The first concerns public nuisance itself. The tort emerged from the opioid litigation in a curious condition: more prominent than ever, and doctrinally weaker in several important states. The highest courts of Oklahoma and Ohio, and a federal judge in West Virginia after a full trial, rejected its application to the distribution of lawful products, each for somewhat different reasons. The American Law Institute has cautioned against it. Yet it underpinned settlements of historic size, and it continues to be pleaded. The tort's inheritors are already in court. School districts, counties, and states sued the e-cigarette maker Juul and its investor Altria on public nuisance and related theories, and Juul paid more than a billion dollars in settlements with states, school districts, and other plaintiffs in 2022 and 2023. Hundreds of school districts and many states have sued the operators of major social media platforms, alleging that their products were designed to be addictive to young people and have created a youth mental-health crisis; the federal cases were consolidated in a multidistrict litigation in the Northern District of California, with parallel proceedings in California state court. Water utilities sued the manufacturers of PFAS chemicals, and 3M agreed in 2023 to pay about $10.3 billion to settle claims by public water systems. Cities, counties, and states have sued fossil fuel companies over the costs of climate change on nuisance and consumer protection theories. Each of these efforts borrows elements of the opioid model: governmental plaintiffs, public nuisance or similar claims, a large and diffuse harm, a centralized proceeding, and the hope of a national settlement. Whether they succeed will depend on the same variables that decided the opioid cases. State law matters: in a state whose courts, like Oklahoma's, have confined nuisance to property and criminal conduct, or whose legislature, like Ohio's, has folded nuisance claims about products into a product liability statute, these claims will struggle. The nature of the defendant's conduct matters: the opioid plaintiffs did best where they could point to the violation of a specific legal duty, such as the pharmacist's corresponding responsibility, rather than to the lawful sale of a product that later caused harm. And the scale of exposure matters: the settlement pressure that the opioid litigation generated came from the combination of thousands of plaintiffs, uncertain law, and potentially unlimited abatement remedies. The lesson for courts is that the uncertainty of public nuisance law has itself been a source of leverage, and that clarifying the law, in either direction, would reduce it. The opioid litigation never produced a decisive ruling from any court with national reach on whether public nuisance extends to the marketing and distribution of lawful products, and it could not have, because the question is one of state law. State supreme courts and legislatures are the bodies that can settle it, and several have done so. Where they have not, the leverage remains. The settlement as governance The second consequence concerns the settlements as instruments of public policy. The national opioid settlements created a framework that allocates money by formula, restricts its use, and regulates industry conduct, without any federal statute. This is a genuine achievement of legal engineering, and in the absence of legislative action it may have been the only way to direct resources at the scale required. But it also raises questions that the opioid experience has not answered. The first is democratic accountability. The terms of the settlements were set by attorneys general, local-government lawyers, and corporate defendants. The attorneys general are elected, and many of the spending decisions are made by elected local officials. But the basic architecture, including the list of permitted uses and the allocation formula, was agreed in negotiation, not debated in a legislature. If future mass litigation follows the same model, a significant amount of public-health policy will be made through this kind of private ordering. The second is durability. Payments stretched over eighteen years depend on the continued solvency of the payers, as the second Mallinckrodt bankruptcy demonstrated. They also depend on sustained political attention to spending restrictions, which history suggests tends to fade. The tobacco settlement's drift into general revenue took place gradually, over decades. The opioid settlements' stronger restrictions will be tested in the same way. The third is the relationship between the money and the problem. The epidemic that the settlements address is not the epidemic that the litigation was about. The lawsuits concerned prescription opioids, but by the time the money arrived, most overdose deaths were caused by illicit fentanyl, increasingly mixed with other substances such as xylazine and stimulants. The permitted uses are broad enough to address that shift, and much of the money is being spent on treatment, naloxone, and other measures that work regardless of the drug involved. The sharp decline in overdose deaths in 2024 suggests that some combination of treatment expansion, naloxone availability, changes in the drug supply, and other factors is having an effect, though how much of it is attributable to settlement spending is not yet known and may never be precisely measured. What a reader should take from all this The third consequence is practical, and it concerns anyone who wants the settlement money to do what it was meant to do: residents of a county receiving funds, members of an advisory council, local officials, clinicians, families, and people in recovery. The single most important fact about the opioid settlement money is that most of its spending decisions are being made locally, over many years, by officials who have wide discretion and who respond to public attention. In most states, there is a formal process, an advisory body, or a public-reporting requirement that gives residents a point of entry. The Johns Hopkins principles, the list of core strategies in the settlement agreements, and the growing body of research on effective interventions provide a benchmark for judging proposals. Medications for opioid use disorder, naloxone distribution, treatment in jails and at release, and services for pregnant women and families have the strongest evidence behind them. Spending that substitutes for existing budgets, or that funds purposes only loosely connected to addiction, deserves scrutiny. The money is large enough, and will flow for long enough, that sustained attention can change outcomes. For those concerned with corporate accountability, the lesson is sobering but not hopeless. The litigation did not produce the authoritative judgments of wrongdoing, or the individual sanctions, that many hoped for. It did produce a public record of unusual depth, enforceable conduct rules, the destruction of Purdue as an independent company, and billions of dollars for repair. The principal gaps lay in the weakness of individual accountability and the absence of compensation for individuals, both of which reflect limits of the legal tools available rather than failures of effort. Closing those gaps would require changes that litigation alone cannot deliver: stronger and more consistent regulatory enforcement, criminal law better adapted to diffuse corporate decision-making, and legislative choices about how to fund the response to public-health harms caused by lawful industries. The opioid litigation will be studied for a long time as a model, both of what aggregate litigation can accomplish and of what it cannot. Its central achievement was to force the companies that profited from the sale of opioids to pay a share of the cost of the harm. Its central limitation was that it did so through a legal theory that the courts never fully accepted and a settlement process that substituted negotiation for judgment. Whether that trade was worth making is a question the settlement money itself will help answer, one payment and one program at a time, over the next two decades. Notes Introduction 1. State ex rel. Hunter v. Johnson & Johnson, 2021 OK 54, 499 P.3d 719 (Okla. 2021). 1. Harrington v. Purdue Pharma L.P., 603 U.S. 204 (2024). Chapter 1 1. Jane Porter and Hershel Jick, "Addiction Rare in Patients Treated with Narcotics," New England Journal of Medicine 302 (1980): 123. 1. Centers for Disease Control and Prevention, National Center for Health Statistics, provisional and final drug overdose death data (National Vital Statistics System), 1999 to 2024. 2. Controlled Substances Act, 21 U.S.C. § 801 et seq.; 21 C.F.R. § 1301.74(b) (suspicious orders); 21 C.F.R. § 1306.04(a) (corresponding responsibility of pharmacists). 3. Masters Pharmaceutical, Inc. v. Drug Enforcement Administration, 861 F.3d 206 (D.C. Cir. 2017). 4. Substance Use-Disorder Prevention that Promotes Opioid Recovery and Treatment for Patients and Communities (SUPPORT) Act, Pub. L. No. 115-271 (2018). 5. Ensuring Patient Access and Effective Drug Enforcement Act of 2016, Pub. L. No. 114-145. 6. United States v. Purdue Frederick Co., 495 F. Supp. 2d 569 (W.D. Va. 2007) (accepting plea agreement). 7. U.S. Department of Justice, "Opioid Manufacturer Purdue Pharma Pleads Guilty to Fraud and Kickback Conspiracies," press release, November 24, 2020. 8. U.S. House of Representatives, Committee on Energy and Commerce, Red Flags and Warning Signs Ignored: Opioid Distribution and Enforcement Concerns in West Virginia (2018). 9. Eric Eyre, reporting in the Charleston Gazette-Mail, awarded the 2017 Pulitzer Prize for Investigative Reporting; Washington Post analysis of the DEA ARCOS database, 2019. Chapter 2 1. Restatement (Second) of Torts § 402A, comment k (1965). 1. Foister v. Purdue Pharma L.P., 295 F. Supp. 2d 693 (E.D. Ky. 2003). 2. Price v. Purdue Pharma Co., 920 So. 2d 479 (Miss. 2006). 3. City of Flagstaff v. Atchison, Topeka & Santa Fe Railway Co., 719 F.2d 322 (9th Cir. 1983). 4. Ganim v. Smith & Wesson Corp., 780 A.2d 98 (Conn. 2001). 5. Master Settlement Agreement (1998), administered through the National Association of Attorneys General. 6. Campaign for Tobacco-Free Kids and partner organizations, annual reports on state tobacco settlement spending, Broken Promises to Our Children. Chapter 3 1. W. Page Keeton et al., Prosser and Keeton on the Law of Torts, 5th ed. (West, 1984), § 86. 1. Restatement (Second) of Torts §§ 821B, 821C (1979). 2. City of Chicago v. Beretta U.S.A. Corp., 821 N.E.2d 1099 (Ill. 2004). 3. Protection of Lawful Commerce in Arms Act, 15 U.S.C. §§ 7901 to 7903 (2005). 4. State v. Lead Industries Ass'n, 951 A.2d 428 (R.I. 2008). 5. People v. ConAgra Grocery Products Co., 17 Cal. App. 5th 51 (2017). 6. Restatement (Third) of Torts: Liability for Economic Harm § 8 (2020). 7. Donald G. Gifford, "Public Nuisance as a Mass Products Liability Tort," University of Cincinnati Law Review 71 (2003): 741. Chapter 4 1. State ex rel. Hunter v. Purdue Pharma L.P., No. CJ-2017-816 (Cleveland County Dist. Ct., Okla.), judgment after trial, 2019; reversed, State ex rel. Hunter v. Johnson & Johnson, 2021 OK 54. 1. People v. Purdue Pharma L.P. (Orange County Super. Ct., Cal.), tentative and final statements of decision, 2021. 2. City of Huntington v. AmerisourceBergen Drug Corp. and Cabell County Commission v. AmerisourceBergen Drug Corp. (S.D. W. Va. 2022) (memorandum opinion and order after bench trial). 3. In re National Prescription Opiate Litigation (Track Three, Lake and Trumbull Counties), MDL No. 2804 (N.D. Ohio), jury verdict November 2021; abatement order August 2022. 4. In re National Prescription Opiate Litigation, Supreme Court of Ohio, decided December 2024 (answering certified question from the Sixth Circuit); Ohio Rev. Code § 2307.71(A)(13). 5. City and County of San Francisco v. Purdue Pharma L.P. (N.D. Cal. 2022) (findings of fact and conclusions of law as to Walgreens). Chapter 5 1. 28 U.S.C. § 1407. 1. In re National Prescription Opiate Litigation, 290 F. Supp. 3d 1375 (J.P.M.L. 2017). 2. In re National Prescription Opiate Litigation, 927 F.3d 919 (6th Cir. 2019) (ARCOS data and protective order). 3. In re National Prescription Opiate Litigation, 332 F.R.D. 532 (N.D. Ohio 2019) (certifying negotiation class), reversed, 976 F.3d 664 (6th Cir. 2020). 4. In re National Prescription Opiate Litigation, 956 F.3d 838 (6th Cir. 2020) (mandamus). 5. Francis E. McGovern and William B. Rubenstein, "The Negotiation Class: A Cooperative Approach to Class Actions Involving Large Stakeholders," Texas Law Review 99 (2020): 73. 6. Elizabeth Chamblee Burch, Mass Tort Deals: Backroom Bargaining in Multidistrict Litigation (Cambridge University Press, 2019). Chapter 6 1. Distributor Settlement Agreement and Janssen Settlement Agreement (2021), with exhibits, published at nationalopioidsettlement.com. 1. Settlement agreements with Teva, Allergan, CVS, Walgreens, and Walmart (2022), published by the settling states and the national settlement administrator. 2. Nora Freeman Engstrom and Robert L. Rabin, "Pursuing Public Health Through Litigation: Lessons from Tobacco and Opioids," Stanford Law Review 73 (2021): 285. Chapter 7 1. In re Purdue Pharma L.P., 633 B.R. 53 (Bankr. S.D.N.Y. 2021). 1. In re Purdue Pharma L.P., 635 B.R. 26 (S.D.N.Y. 2021). 2. In re Purdue Pharma L.P., 69 F.4th 45 (2d Cir. 2023). 3. Harrington v. Purdue Pharma L.P., 603 U.S. 204 (2024). 4. 11 U.S.C. § 1123(b)(6). 5. Lindsey D. Simon, "Bankruptcy Grifters," Yale Law Journal 131 (2022): 1154. Chapter 8 1. OneOhio Memorandum of Understanding (2020) and OneOhio Recovery Foundation governing documents. 1. North Carolina Memorandum of Agreement Between the State of North Carolina and Local Governments on Proceeds Relating to the Settlement of Opioid Litigation (2021). 2. Johns Hopkins Bloomberg School of Public Health, Principles for the Use of Funds From the Opioid Litigation (2021). 3. Opioid Settlement Tracker, maintained by Christine Minhee. 4. KFF Health News, reporting series on opioid settlement spending, 2022 to 2025. Chapter 9 1. United States v. Kapoor (D. Mass.), jury verdict 2019. 1. U.S. Attorney's Office for the Southern District of New York, announcements concerning Rochester Drug Cooperative and Laurence Doud, 2019 and 2023. 2. U.S. Department of Justice, announcement of McKinsey & Company deferred prosecution agreement, December 2024. 3. United States v. Walmart Inc., complaint filed December 2020 (D. Del.). 4. Centers for Disease Control and Prevention, "CDC Clinical Practice Guideline for Prescribing Opioids for Pain," MMWR Recommendations and Reports 71, no. 3 (2022). 5. Opioid Industry Documents Archive, University of California, San Francisco, and Johns Hopkins University. Further Reading Patrick Radden Keefe, Empire of Pain: The Secret History of the Sackler Dynasty (Doubleday, 2021).** The definitive narrative history of the Sackler family and Purdue Pharma, drawing heavily on litigation documents. Essential background for the bankruptcy and accountability questions. Beth Macy, Dopesick: Dealers, Doctors, and the Drug Company that Addicted America (Little, Brown, 2018).** A ground-level account of the epidemic's early years in Appalachian Virginia, including the 2007 federal prosecution of Purdue. Sam Quinones, Dreamland: The True Tale of America's Opiate Epidemic (Bloomsbury, 2015).** Traces the convergence of prescription opioid marketing and heroin distribution networks; the best account of how the first wave fed the second. Eric Eyre, Death in Mud Lick: A Coal Country Fight Against the Drug Companies That Delivered the Opioid Epidemic (Scribner, 2020).** The Pulitzer-winning reporter's account of the pill shipments to West Virginia and of the litigation that exposed them. Barry Meier, Pain Killer: An Empire of Deceit and the Origin of America's Opioid Epidemic, 2nd ed. (Random House, 2018). The first major investigation of OxyContin marketing, originally published in 2003 and updated. Nora Freeman Engstrom and Robert L. Rabin, "Pursuing Public Health Through Litigation: Lessons from Tobacco and Opioids," Stanford Law Review 73 (2021).** A careful comparison of the two great governmental litigation campaigns and what each achieved. Donald G. Gifford, "Public Nuisance as a Mass Products Liability Tort," University of Cincinnati Law Review 71 (2003).** The leading early critique of product-based public nuisance claims, written before the opioid cases but anticipating their central problems. Elizabeth Chamblee Burch, Mass Tort Deals: Backroom Bargaining in Multidistrict Litigation (Cambridge University Press, 2019).** A critical study of how MDL settlements are negotiated and whose interests they serve. Johns Hopkins Bloomberg School of Public Health, Principles for the Use of Funds From the Opioid Litigation (2021).** A short, practical guide for officials and residents deciding how settlement money should be spent. The Opioid Industry Documents Archive (UCSF and Johns Hopkins University) and nationalopioidsettlement.com. The primary sources: millions of pages of internal company documents, and the full text of the national settlement agreements with their exhibits, including the list of approved uses and the state allocation percentages. Hashtags: #TheOpioidLitigation #OpioidLitigation #OpioidCrisis #OpioidEpidemic #MassTorts #MassTortLitigation #PublicNuisance #PublicNuisanceLaw #CorporateAccountability #CorporateLiability #PharmaceuticalLitigation #PharmaceuticalLaw #PharmaceuticalAccountability #TortLaw #TortReform #ProductLiability #PrescriptionDrugLitigation #PrescriptionOpioids #OpioidManufacturers #DrugDistributors #PharmacyLiability #PharmaceuticalMarketing #DeceptiveMarketing #MisleadingDrugPromotion #CorporateMisconduct #CorporateNegligence #DrugDiversion #ControlledSubstancesAct #DEARegulations #SuspiciousOrderMonitoring #PharmacistResponsibility #OpioidSupplyChain #OxyContin #PurduePharma #SacklerFamily #JohnsonAndJohnson #McKesson #CardinalHealth #AmerisourceBergen #CVSHealth #Walgreens #Walmart #TevaPharmaceuticals #OpioidSettlements #NationalOpioidSettlements #SettlementAgreements #SettlementFunds #SettlementAllocation #OpioidAbatement #AbatementRemedies #PublicHealthLitigation #PublicHealthAccountability #PublicHealthLaw #AddictionTreatment #OpioidUseDisorder #OverdosePrevention #NaloxoneAccess #AddictionRecovery #HarmReduction #CommunityRecovery #MassLitigation #MultidistrictLitigation #MDL2804 #BellwetherTrials #ClassActionLitigation #StateAttorneysGeneral #GovernmentLitigation #CivilLiability #CorporateBankruptcy #BankruptcyLaw #ThirdPartyReleases #PurduePharmaBankruptcy #HarringtonVPurduePharma #SupremeCourtLitigation #LegalAccountability #RegulatoryEnforcement #FDARegulation #DrugPolicy #HealthcareRegulation #ConsumerProtectionLaw #ProximateCausation #LearnedIntermediaryDoctrine #MunicipalCostRecovery #EconomicLossRule #InjunctiveRelief #LegalSettlements #TobaccoLitigation #CorporateTransparency #LitigationDiscovery #ARCOSData #PublicHealthFunding #LegalPrecedents #FutureOfMassTorts
- Tech Policy and Antitrust: Navigating the Digital Markets Act (DMA)
Download the Book (PDF): Introduction On 7 March 2024, six of the largest technology companies in the world changed how their products behaved in Europe on the same day. iPhone users in the European Union who installed iOS 17.4 could, for the first time, download an app store that Apple did not run. Android and Chrome users began seeing screens asking them to choose a search engine and a browser rather than receiving one by default. Meta had published, the day before, a technical document explaining how a rival messaging service could connect to WhatsApp. Microsoft had started letting European Windows users uninstall Edge and remove Bing from the search box on the taskbar. None of these companies did this because it wanted to. They did it because a regulation adopted in Brussels in 2022, the Digital Markets Act, said that the six months it had given them to comply were over. Two and a half years later, the picture is messier and more interesting than either the law's supporters or its critics predicted. The European Commission has issued its first fines under the Act: €500 million against Apple and €200 million against Meta in April 2025, and €890 million against Google in July 2026, split between self-preferencing in search results and restrictions on how app developers may steer customers away from Google Play. Apple has rewritten its European business terms three times, and the latest version, which took effect on 1 October 2026, replaced the per-install fee that made developers furious with a flat commission on sales. WhatsApp has built the machinery to exchange end-to-end encrypted messages with outside apps and began rolling it out in late 2025, though the first announced partners were two small European start-ups rather than Signal, Telegram or Apple. Third-party smartwatches can receive and answer iPhone notifications in Europe because the Commission wrote down, feature by feature, what Apple had to open and when. At the same time, alternative app stores remain marginal, at least one has shut down, and Apple has held back new features from European users while arguing that the law makes their phones less safe. This booklet is about what happens when a legal obligation meets a piece of engineered software controlled by a company that would rather not change it. Its argument can be stated in one sentence. A walled garden is not opened by a prohibition but by a specification: the Digital Markets Act changes markets only where its general obligations are translated into concrete interfaces, prices and timelines precise enough to be built and verified, and the gatekeeper's control over that translation is where the real contest takes place. That claim explains much of what has happened since 2024. The Act tells Apple that it must allow developers to inform users of cheaper offers elsewhere "free of charge". It does not say whether a link may lead to a web view or must open the browser, whether the warning screen that precedes it may say "unsafe", or whether a commission on purchases made within seven days of tapping the link is a charge. The Act tells Meta that a rival messenger must be able to exchange text, images, voice notes, videos and files with WhatsApp users and that the level of security, including end-to-end encryption, must be preserved. It does not tell Meta which protocol to use, how to verify a rival's users, what to do with spam, or who holds the metadata. The Act tells Google not to favour its own services in search rankings. It does not say what a neutral results page looks like. Every one of these gaps is filled by someone, and in the first instance that someone is the gatekeeper, because it writes the code, sets the price and decides the release schedule. The Commission's work, and much of this booklet, consists of watching the gatekeeper fill those gaps and deciding whether the result complies. Seen this way, several things that look like failures or scandals become intelligible. The long sequence of Apple fee structures is not just stubbornness; it is the predictable result of an obligation that prohibits some charges without saying what an acceptable charge is. The modest uptake of messaging interoperability is not proof that nobody wanted it; it reflects a design in which the rival must adopt WhatsApp's protocol, sign WhatsApp's agreement and accept WhatsApp's security conditions, which is a high bar for any service that has its own users and reputation to protect. The Commission's turn, from 2024 onwards, towards formal "specification" decisions, which spell out interfaces and deadlines in the detail of an engineering requirements document, is the institution learning that ordering a result is not enough; it must describe one. The booklet proceeds in eight chapters. The first explains why Europe decided that ordinary competition enforcement was too slow for digital platforms, using the cases that shaped that judgement: Google Shopping, Google Android and Apple's music streaming rules. The second lays out the machinery of the Act itself, from the thresholds that make a company a gatekeeper to the fines and remedies that back the obligations. The next four chapters each take one front where a walled garden is being opened. Chapter 3 is about messaging interoperability: what it took, at the protocol level, to let an outside app talk to WhatsApp without breaking end-to-end encryption, and why the result looks the way it does. Chapter 4 is about distribution: sideloading, alternative app marketplaces, notarization and the eligibility rules that decide who may run a store. Chapter 5 is about money: anti-steering rules, alternative billing, and the long negotiation over what Apple and Google may charge developers who do business outside their stores. Chapter 6 is about the operating system as a set of capabilities, from smartwatch notifications to the hooks that let an AI assistant act on a user's behalf, and about the Commission's method of specifying them one by one. Chapter 7 steps back to the argument that runs underneath all four: security. Gatekeepers resist most obligations in the name of user safety, privacy and the integrity of their systems, and the Act allows them to do so where measures are "strictly necessary and proportionate". How a regulator should test such claims, and how a gatekeeper might abuse them, is one of the most consequential questions in the field. Chapter 8 then turns to enforcement as it is actually practised: the cycle of compliance reports, workshops, preliminary findings and fines; the role of appeals; the April 2026 review that judged the Act fit for purpose; the extension of the regime towards cloud computing and AI; and the spread of similar laws to the United Kingdom and Japan amid open hostility from the United States government. The conclusion argues what follows from all of this for regulators, for companies on both sides of the gate, and for anyone trying to judge whether the experiment is working. A word on scope. The Digital Markets Act is one of several European laws aimed at large platforms. The Digital Services Act deals with illegal content, systemic risks and transparency; the General Data Protection Regulation governs personal data; the AI Act governs artificial intelligence systems. These matter here only where they collide with the DMA, as when a data-sharing obligation meets data protection law. The booklet also leaves aside most of the Act's obligations on advertising transparency and the treatment of business users on marketplaces such as Amazon's, not because they are unimportant but because the questions the booklet is built around, how one opens a technical system on legal command, are sharpest on mobile platforms, messaging and search. The reader assumed here is intelligent and curious but not a specialist in either competition law or software engineering. Legal terms are explained when they first appear, and so are technical ones. The facts are stated as they stand in early October 2026. Several matters remain open: Google's period to comply with the July 2026 decisions has only just elapsed, the Commission is still deciding whether to designate Amazon's and Microsoft's cloud services, Apple's appeals against the fine and the interoperability decisions are pending before the EU courts, and the newest Apple terms have been in force for only a few days. Where an outcome is not yet known, the booklet says so rather than guessing. One more point is worth making at the outset, because it cuts against a common way of talking about this subject. Debates about the DMA are often framed as a contest between regulation and innovation, or between Europe and American technology firms. Those framings are not wrong, but they hide the part that matters most for understanding outcomes. The decisive questions are usually narrow and technical: which API a third-party app may call, in what order screens appear, which transactions a fee attaches to, how long a link-out counts as a referral, what a warning message says, who can verify an identity key. Policy is made in those details. A reader who comes away able to see where the details are, and why they decide so much, will understand the Digital Markets Act better than one who knows only the headline fines. Chapter 1: Why Europe Stopped Waiting The Digital Markets Act was born out of impatience. By the late 2010s, European competition officials had spent more than a decade pursuing the largest technology companies under the ordinary rules of antitrust law, and they had won most of the cases they brought. Yet the markets they were trying to protect had barely moved. The companies paid the fines, changed their conduct at the margin, appealed, and kept their positions. The lesson that policymakers drew was not that the companies were above the law, but that the law worked on the wrong timescale and asked the wrong question. This chapter explains that lesson through the cases that taught it, and sets out the bargain the DMA struck in response: faster, simpler rules in exchange for giving up much of the nuance that makes antitrust fair. The case that took fourteen years The emblematic case is Google Shopping. In the mid-2000s Google began promoting its own comparison shopping service in its general search results while the algorithms that ranked rival comparison sites pushed many of them far down the page. Small comparison sites complained to the European Commission; the British site Foundem, run by a husband and wife, became the best known. The Commission opened a formal investigation in November 2010. It adopted a decision in June 2017, finding that Google had abused its dominant position in general search and imposing a fine of €2.42 billion. Google appealed. The General Court upheld the decision in November 2021. The Court of Justice dismissed Google's final appeal in September 2024. From the opening of the investigation to the final judgment, fourteen years passed. During that period the comparison shopping market that the case was meant to protect largely ceased to exist in the form it had taken in 2010. Many of the rivals that complained were diminished or gone. Google's remedy, an auction in which rival comparison services could bid for places in the shopping unit at the top of results, was introduced in 2017 and has been criticized ever since by those same rivals as preserving Google's control over where and how they appear. Whatever one thinks of the merits, the outcome made a simple point vivid: a legal process that takes longer than a product cycle cannot restore competition in a market that tips in a few years. Google Android repeated the pattern on a different front. In July 2018 the Commission fined Google €4.34 billion for tying its search app and Chrome browser to the Play Store in its licensing agreements with phone makers, for paying manufacturers and mobile operators to pre-install Google Search exclusively, and for preventing manufacturers from selling devices running forks of Android. The General Court largely upheld the decision in September 2022, trimming the fine to €4.125 billion. By then, Google's position in mobile search was settled, and the choice screen it introduced in Europe to address the decision changed user behaviour only at the margin. The case against Apple's App Store rules for music streaming apps shows the same rhythm in the 2020s. Spotify complained in March 2019 that Apple's rules prevented it from telling iPhone users that a subscription was cheaper if bought on Spotify's website. In March 2024 the Commission fined Apple more than €1.8 billion for imposing those anti-steering provisions on music streaming providers. Five years for one category of app, with an appeal to follow, while the same rules applied to every other category of digital content. These cases had something else in common besides their duration. Each required the Commission to define a relevant market, establish that the company was dominant within it, prove that a particular practice was abusive, and show that it had or was likely to have anticompetitive effects. Each invited an extended debate over efficiencies, justifications and counterfactuals. That apparatus exists for good reasons: it is the protection firms have against being punished for competing hard. But it also means that each case addresses one practice in one market, and the next practice in the next market starts from zero. When the Commission's 2019 AdSense fine against Google was annulled by the General Court in September 2024, partly for failures in the Commission's assessment of the relevant contracts, it underlined how fragile even a successful-looking case could be after years of litigation. The precedent nobody wanted to repeat There was an older case that the drafters of the DMA had in mind, and it is the one most relevant to this booklet's argument. In March 2004 the Commission found that Microsoft had abused its dominance in PC operating systems in two ways: by tying Windows Media Player to Windows, and by refusing to supply rivals in the work group server market with the information they needed for their products to interoperate with Windows. It fined Microsoft €497 million and ordered it to disclose complete and accurate interface specifications on reasonable terms. The order looked straightforward. Its execution was not. What counts as complete and accurate documentation of a protocol? What is a reasonable royalty for it? Who decides whether a rival can actually build a working product from what was disclosed? The Commission appointed a monitoring trustee with technical expertise, and the dispute over the quality of Microsoft's documentation and the level of its licence fees ran for years. In February 2008 the Commission imposed a periodic penalty of €899 million on Microsoft for charging unreasonable prices for the interoperability information, an amount the General Court later reduced to €860 million. The Court of First Instance upheld the original decision in September 2007. The Microsoft case is where the central problem of this booklet first appeared in European law. A regulator can order a dominant firm to open its system. But the firm controls the specification, the documentation, the pricing and the timetable, and every one of those can be used to make compliance formally present and practically useless. The lesson the Commission took from Microsoft was not that interoperability remedies fail; the work group server market did become more open. It was that such remedies need to be specified in enough detail to be verified, and that a regulator who leaves the specification to the regulated firm should expect to spend years arguing about it. A related lesson came from the browser choice screen. In 2009 Microsoft committed to show European Windows users a screen offering a choice of web browsers. In 2013 the Commission fined it €561 million because, for over a year, a Windows update had silently dropped the screen for millions of users. The episode showed that even a simple, agreed remedy depends on the gatekeeper's code continuing to implement it, and that regulators need some way to notice when it stops. The diagnosis of 2019 By 2019 a consensus had formed among economists and policymakers on both sides of the Atlantic that digital platforms posed a structural problem that case-by-case enforcement was ill suited to solve. Three reports published that year carried particular weight. The Commission's own special advisers, Jacques Crémer, Yves-Alexandre de Montjoye and Heike Schweitzer, delivered Competition Policy for the Digital Era in April. The British government's Digital Competition Expert Panel, chaired by Jason Furman, published Unlocking Digital Competition in March. The Stigler Center at the University of Chicago published its committee report on digital platforms later in the year. The reports differed in emphasis, but they converged on a description of the markets. Digital platforms benefit from strong network effects: a messaging service is more valuable the more people use it, an app store more valuable the more apps it carries and the more users those apps can reach. They enjoy economies of scale and scope in data: a search engine that sees more queries can rank results better, and data gathered in one service improves another. They often sit in two-sided or multi-sided markets where one side, usually users, pays nothing and the other side, businesses, pays for access to the first. And their positions, once established, are protected by switching costs and default effects that make it very hard for an equally good or even better rival to dislodge them. From this description followed a prescription. The reports argued that markets with these features tend to tip towards a single winner, that the winner then becomes a gatekeeper between businesses and their customers, and that competition for the market, the hope that a new entrant would one day displace the incumbent, should be supplemented by rules ensuring competition within the platform and on its edges. The Furman report proposed a code of conduct for firms with "strategic market status", backed by a specialist unit. Germany acted first in law: in January 2021 it added section 19a to its competition act, allowing the Federal Cartel Office to designate firms of "paramount significance for competition across markets" and prohibit specified practices. Alphabet, Meta, Amazon, Apple and Microsoft were all designated under it in the years that followed. The bargain of ex ante regulation The Commission published its proposal for the Digital Markets Act on 15 December 2020, alongside the Digital Services Act. The legislative process was quick by European standards. The Parliament and the Council reached political agreement in March 2022, the Regulation was signed in September 2022, it entered into force on 1 November 2022 and its rules began to apply on 2 May 2023. The central design choice was to regulate ex ante, meaning before any harm is proven in an individual case, rather than ex post, meaning after the fact. Instead of asking in each case whether a dominant firm's conduct in a defined market harmed competition, the Act identifies a small set of firms by size and reach, calls them gatekeepers, and imposes on them a list of obligations that apply automatically. There is no need to define a relevant market or prove dominance. There is no need to show anticompetitive effects in each instance. And, crucially, there is no general defence of efficiency: a gatekeeper cannot argue that self-preferencing in a particular case actually benefits consumers. The only escape hatches are narrow ones: suspension for exceptional economic circumstances, exemption on grounds of public health or public security, and the security and integrity carve-outs discussed in Chapter 7. This is a genuine bargain, and it is worth being clear about what was given up. Antitrust's case-by-case method is slow precisely because it tries to distinguish harmful conduct from beneficial conduct that looks similar. Self-preferencing can be a way of excluding rivals; it can also be a way of offering users an integrated product they prefer. An app store's ban on steering can be a way of extracting monopoly rents; it can also be a way of funding the review process and the developer tools that make the store valuable. By turning these questions into rules, the DMA accepted that it would sometimes prohibit conduct that a full analysis would have permitted. The drafters judged that the cost of occasional over-inclusion was lower than the cost of a decade of delay. Reasonable people still disagree about that judgement, and much of the opposition to the Act from the companies concerned and from the United States government rests on rejecting it. There is a further subtlety. The DMA is formally not a competition law. Its stated aims are contestability and fairness in digital markets, and its legal basis is the internal market article of the EU Treaties rather than the competition chapter. It applies "without prejudice" to the competition rules in Articles 101 and 102 of the Treaty, so the Commission and national authorities can still bring antitrust cases against the same companies for the same practices, subject to the general principle, clarified by the Court of Justice in its 2022 bpost judgment, that an undertaking should not be punished twice for the same facts without proper coordination between proceedings. In practice the two tools now run side by side. The Commission accepted commitments from Apple under Article 102 in July 2024 to open the iPhone's near-field communication chip to third-party payment apps, and in June 2026 it imposed interim measures under antitrust law requiring Meta to keep WhatsApp's business interface open to rival AI chatbots, while DMA proceedings were running on other fronts. What the drafters chose not to do The shape of the Act is also defined by what its drafters left out, and three omissions matter for everything that follows. The first is price regulation. Nowhere does the DMA set a maximum commission for app stores, a ceiling on advertising fees or a price for interoperability. Where it touches price at all, it does so indirectly: steering must be possible "free of charge", interoperability with operating system features must be provided "free of charge", and access to app stores must be on "fair, reasonable and non-discriminatory" conditions. European legislators were wary of turning the Commission into a utility regulator setting tariffs for software platforms, a role that would have required cost models, rate cases and a permanent apparatus of the kind that telecoms regulators maintain. The choice was understandable. As Chapter 5 shows, it also meant that the most commercially important question, how much a gatekeeper may charge for access to its users, was left to be settled through enforcement rather than legislation. The second omission is structural separation. Some advocates had argued that the cleanest way to stop a platform favouring its own services was to forbid it from owning them: to separate Google's search engine from its shopping and travel businesses, or Apple's App Store from its apps. The DMA does not do this. It regulates conduct, leaving the integrated structure of the companies intact, and reserves structural remedies for the extreme case of systematic non-compliance described in Chapter 2. Conduct rules are less disruptive than break-ups, but they require continuous supervision, because an integrated company always has an incentive and an opportunity to tilt its platform towards its own products in ways no rule fully anticipates. The third omission is a general duty of interoperability. The Act requires interoperability only in two places: for messaging services in Article 7 and for operating system features in Article 6(7). It does not require social networks to interoperate, as some proposals had suggested, and it does not require app stores to share their catalogues or reviews. The drafters chose targeted obligations they believed could be implemented over a broad principle whose application would be uncertain. That choice explains why so much of the action under the Act concerns mobile operating systems and messaging: those are the places where the legislators wrote down that walls must have doors. Each of these omissions was a reasonable bet in 2022. Together they meant that the DMA's success would depend on how well its targeted, conduct-based, price-silent obligations could be turned into concrete requirements, which is the subject of the rest of this booklet. Rules that still need writing The bargain had a hidden cost that became visible only once the Act began to apply. Rules are fast only if they are self-executing, that is, if the obligated firm can read them and know exactly what to do. Some of the DMA's obligations come close. A gatekeeper must not require business users to subscribe to another of its services as a condition of access; must allow users to uninstall pre-installed apps; must not combine personal data across services without consent. Others are framed at a level of generality that leaves the decisive choices open. "Free of charge" steering, "effective" interoperability, "fair, reasonable and non-discriminatory" access conditions, ranking that is "transparent, fair and non-discriminatory": each of these phrases describes an outcome without specifying the means. The drafters knew this. Article 8 of the Act gives the Commission the power, on its own initiative or at a gatekeeper's request, to adopt decisions specifying the measures a gatekeeper must take to comply with certain obligations. The power was initially seen as a fallback. It has become one of the Commission's main instruments. The first specification decisions, adopted against Apple in March 2025, ran to dozens of pages on matters such as notification forwarding to smartwatches and the timelines for answering developers' interoperability requests. In 2026 the Commission used the same tool to tell Google how Android must expose features to rival AI assistants and how Google must share search data. In each case, what began as a one-line obligation in the Regulation ended as something closer to an engineering requirements document. That trajectory is the theme of the chapters that follow. The DMA was meant to escape the slowness of antitrust by replacing analysis with rules. It has succeeded in compressing timelines: the first fines came within about a year of the investigations opening, rather than seven. But it has not escaped the problem the Microsoft case revealed twenty years earlier. Opening a technical system requires someone to describe, in detail, what open means for that system. Where the Commission has done that work, markets have begun to move. Where it has left the description to the gatekeeper, it has usually found itself back in a long negotiation over the meaning of a phrase. Chapter 2: The Machinery of the Act A reader who opens the Digital Markets Act expecting a short list of prohibitions will find instead a piece of administrative machinery: a procedure for deciding who is covered, a set of obligations sorted by how much interpretation they need, a set of tools for turning general obligations into specific ones, and a graduated scale of sanctions. Understanding how these parts fit together is the precondition for understanding every fight described later in this booklet, because those fights usually turn on which part of the machine a dispute has entered. This chapter walks through the machinery in the order a gatekeeper experiences it. Who is a gatekeeper The Act regulates "gatekeepers", a term it defines functionally and then makes operational through numerical thresholds. Functionally, a gatekeeper is an undertaking that has a significant impact on the internal market, provides a "core platform service" that is an important gateway for business users to reach end users, and enjoys an entrenched and durable position in its operations or is foreseeably about to. The list of core platform services is closed. It comprises online intermediation services such as app stores and marketplaces, online search engines, online social networking services, video-sharing platform services, number-independent interpersonal communications services (the legal term for messaging apps that do not rely on telephone numbers to route calls), operating systems, web browsers, virtual assistants, cloud computing services and online advertising services. A company that does not provide one of these services cannot be a gatekeeper, however large it is. Each functional criterion is presumed met when a company crosses quantitative thresholds. Significant impact is presumed where the undertaking achieved annual turnover in the European Economic Area of at least €7.5 billion in each of the last three financial years, or where its average market capitalisation or equivalent fair market value was at least €75 billion in the last financial year, and it provides the same core platform service in at least three Member States. The gateway criterion is presumed met where the service had at least 45 million monthly active end users established or located in the EU and at least 10,000 yearly active business users established in the EU in the last financial year. Entrenchment is presumed where the user thresholds were met in each of the last three financial years. A company that meets the thresholds must notify the Commission within two months, and the Commission must designate it within 45 working days of a complete notification. The company may present arguments that, despite meeting the thresholds, it does not satisfy the functional criteria, but the Act limits what may count: the arguments must be "sufficiently substantiated" to "manifestly" call the presumption into question, and arguments about economic efficiencies are not admissible at this stage. In the other direction, the Commission can designate a company that does not meet the thresholds through a qualitative assessment following a market investigation. The Commission designated the first six gatekeepers on 6 September 2023: Alphabet, Amazon, Apple, ByteDance, Meta and Microsoft, covering 22 core platform services. It added Apple's iPadOS in April 2024 after a qualitative assessment, since the tablet operating system fell short of the business user threshold but was judged an important gateway anyway, and designated Booking in May 2024 for its online travel intermediation service. Table 1 shows the designations as they stood in early October 2026. Table 1. Designated gatekeepers and core platform services, October 2026. Gatekeeper Core platform services designated First designated Source Alphabet Search, Play, Maps, Shopping, YouTube, Android, Chrome, online advertising 6 Sep 2023 European Commission Amazon Marketplace, advertising 6 Sep 2023 European Commission Apple App Store, iOS, Safari; iPadOS added 29 Apr 2024 6 Sep 2023 European Commission ByteDance TikTok 6 Sep 2023 European Commission Meta Facebook, Instagram, WhatsApp, Messenger, Meta Ads 6 Sep 2023 European Commission Microsoft Windows, LinkedIn 6 Sep 2023 European Commission Booking Booking.com 13 May 2024 European Commission The list is as interesting for what it leaves out. In February 2024 the Commission decided not to designate Apple's iMessage, Microsoft's Bing search engine, its Edge browser or its advertising service, accepting that, despite meeting the thresholds, they were not important gateways. It concluded in October 2024 that the social network X should not be designated. In April 2025 it removed Facebook Marketplace from Meta's list after Meta showed it had too few business users. In February 2026 it found that Apple Ads and Apple Maps, which Apple had notified as meeting the thresholds in November 2025, should not be designated either. ByteDance challenged its designation for TikTok in the General Court and lost in July 2024. The result is a list of 23 services run by seven companies, most of them American. Critics in the United States read that as targeting; the Commission's answer is that the thresholds are neutral and the companies above them happen to be American because the industry's largest firms are. The omission of iMessage matters for Chapter 3. Because the Commission found that iMessage was not a gateway for business users, Apple has no obligation under the DMA to make it interoperable with WhatsApp or anything else. The messaging interoperability obligation therefore applies in Europe to WhatsApp and Messenger, and the most popular messaging service in some other markets is outside it. Three tiers of obligation Once designated, a gatekeeper has six months to comply with the obligations in Articles 5, 6 and 7 for each listed service. The three articles are not arbitrary groupings. They are sorted roughly by how much the obligation depends on the specifics of the service. Article 5 contains obligations the drafters considered self-executing, capable of being complied with on the basis of the text alone. A gatekeeper must not combine personal data from one core platform service with data from its other services, or from third parties, without the user's consent. It must not stop business users from offering the same products at different prices or conditions through other channels, the so-called parity or most-favoured-nation clauses. It must allow business users, free of charge, to communicate and promote offers to end users acquired through the platform and to conclude contracts with them, which is the anti-steering obligation at the heart of Chapter 5. It must allow end users to access content, subscriptions and features they bought elsewhere through a business user's app. It must not require end users or business users to use the gatekeeper's identification service, web browser engine or payment service. It must not make access to one of its services conditional on subscribing to another. And it must provide advertisers and publishers with information on prices and fees. Article 6 contains obligations "susceptible of being further specified". These are the ones where the right answer depends heavily on the technology. A gatekeeper must not use non-public data generated by its business users to compete against them. It must let users uninstall pre-installed apps and change defaults, and must show choice screens for search engines, browsers and virtual assistants. It must allow the installation and effective use of third-party apps and app stores on its operating system, and allow them to be set as defaults. It must not rank its own products more favourably than similar third-party products, and must apply transparent, fair and non-discriminatory conditions to ranking. It must allow interoperability with the same hardware and software features of its operating system or virtual assistant that its own services use. It must provide effective data portability, give business users access to the data their activities generate, and give rival search engines access to anonymised ranking, query, click and view data on fair terms. It must apply fair, reasonable and non-discriminatory general conditions of access to its app store, search engine and social network, and must not make it disproportionately hard to terminate a service. Article 7 stands alone. It deals with interoperability of messaging services and sets out a phased timetable, with specific functionalities required at specific dates after designation. It is the most technically detailed obligation in the Act, and still leaves the design choices to the gatekeeper. Two features of the obligations shape how they are fought over. The first is the security carve-out. Articles 6(4), 6(7) and 7 allow the gatekeeper to take measures that are "strictly necessary and proportionate" to protect the integrity of its hardware or operating system or the security of its service, provided it duly justifies them. Each carve-out is a door through which the gatekeeper can bring back restrictions it would otherwise have to drop, and Chapter 7 is devoted to how that door is policed. The second is the anti-circumvention rule in Article 13. A gatekeeper must not undermine compliance through contractual, commercial, technical or any other means, including by degrading the quality of a service or making it unduly difficult for users to exercise their rights. The rule is the Commission's main weapon against compliance that is formally present but practically useless: a link-out that requires five confirmation screens, a choice screen that loads slowly, an interoperability interface that works only half the time. From general rule to specific measure The Act's designers expected gatekeepers to read the obligations and comply. They built in three mechanisms for when that did not happen or when the answer was genuinely unclear. The first is the compliance report. Every gatekeeper must publish, within six months of designation and annually thereafter, a detailed report describing the measures it has taken to comply with each obligation, and a non-confidential summary of it. It must also submit an independently audited description of the techniques it uses to profile consumers. The reports are public, and third parties, developers and rivals, comb through them for gaps. The Commission holds public compliance workshops after each round, at which gatekeepers present their solutions and face questions from stakeholders. These workshops have become a distinctive feature of DMA practice: a public, adversarial, technical forum where an Apple engineer may be asked by a rival browser maker why a particular API is unavailable. The reports also create a paper trail. Because each gatekeeper must describe in writing how it complies with each obligation, and update that description every year, a later change in conduct can be compared with what the company said it was doing. When a measure that was presented as compliant turns out to work differently in practice, the report becomes evidence. Developers and rivals have used each annual round, and the public workshops that follow it, to point to gaps between the description and their experience. The second is specification under Article 8. Where an obligation in Article 6 or 7 is unclear in its application to a particular service, the Commission can open proceedings and adopt a decision specifying the measures the gatekeeper must implement. A gatekeeper can request such a decision itself, as a way of obtaining legal certainty. The Commission must conclude these proceedings within six months, first communicating preliminary findings within three months and consulting third parties on the proposed measures. A specification decision is not a finding of infringement. It is a binding instruction about what compliance means, and once adopted it removes much of the room for argument. The third is the regulatory dialogue that surrounds everything else. Much of what actually happens under the DMA happens through requests for information, informal meetings, draft solutions sent to the Commission for feedback, and revised proposals. The formal record shows the outcomes. The informal process determines most of them, a point taken up in Chapter 8. Sanctions and the threat behind them Where compliance fails, the Commission can open a non-compliance investigation, issue preliminary findings, and then adopt a non-compliance decision ordering the gatekeeper to cease the conduct and explain how it will comply. The Act asks the Commission to endeavour to adopt such decisions within twelve months of opening proceedings. A non-compliance decision may impose a fine of up to 10 percent of the gatekeeper's total worldwide turnover in the preceding financial year, rising to 20 percent where the gatekeeper has committed the same or a similar infringement within the previous eight years. Fines for procedural failures, such as supplying incorrect information or obstructing an inspection, are capped at 1 percent. Separately, the Commission may impose periodic penalty payments of up to 5 percent of average daily worldwide turnover for each day a gatekeeper fails to comply with a decision. Daily penalties are a more potent instrument than lump sums, because they make continued non-compliance more expensive than compliance by the day. Behind all of this sits Article 18. If the Commission has adopted at least three non-compliance decisions against a gatekeeper within eight years, it may open a market investigation into systematic non-compliance. If that investigation confirms the pattern, the Commission may impose any behavioural or structural remedy that is proportionate and necessary, including in principle the divestiture of a business, and may temporarily prohibit acquisitions in the affected areas. No gatekeeper has yet reached three non-compliance decisions. Apple and Meta each have one; Google, after July 2026, has two, issued the same day. The structural option is remote, but its existence frames every negotiation. The fines imposed so far sit far below the ceilings. Apple's €500 million in April 2025 was a small fraction of one percent of its annual revenue, as was Google's €890 million in July 2026. The Commission has explained its figures by reference to the gravity, duration and recurrence of the breaches, and some critics have accused it of timidity under American political pressure. A more charitable reading is that the Commission treats the first fines as signals and relies on the threat of periodic penalties and of escalation to Article 18 to change behaviour. Whether that is right will become clearer as the first decisions are tested on appeal and as the companies' later compliance is assessed. Who else is in the room The Commission is the sole enforcer of the DMA's obligations. Within it, responsibility is shared between the competition directorate and the directorate for communications networks, content and technology, an arrangement that brings together antitrust lawyers and economists with officials who understand telecoms regulation and technical standards. A high-level group of European regulators, including telecoms, data protection, consumer protection and audiovisual bodies, advises on consistency with neighbouring regimes. National competition authorities can investigate possible breaches and report to the Commission but cannot enforce the DMA themselves. National courts, however, can. Because the obligations are directly applicable, a business harmed by a gatekeeper's breach can sue in a national court for an injunction or damages, and the Act includes provisions on cooperation between courts and the Commission. Private enforcement has been slower to develop than public enforcement, but it is a reason why gatekeepers cannot treat the Commission as their only audience: a German or Dutch court applying Article 5 or 6 in a private dispute is a second reader of the same text. Finally, the Act was designed to evolve. Article 12 lets the Commission update the obligations through delegated acts following a market investigation, and Article 19 allows market investigations into whether new services or practices should be brought in. Article 53 required a review of the whole Regulation by May 2026. The Commission published that review at the end of April 2026, concluding that the Act remained fit for purpose. Chapter 8 returns to what the review did and did not change, and to the open question of how far a law built around app stores and search engines can reach into cloud computing and artificial intelligence. The machinery described here is impressive on paper. Its defining weakness is the one already identified. Most of the obligations that matter for opening a walled garden sit in Articles 6 and 7, which require specification; and until the Commission specifies, the gatekeeper does. The next four chapters show what that means on four different fronts. Chapter 3: Messaging Interoperability and the Encryption Problem Of all the walled gardens the Digital Markets Act tries to open, messaging is the one where the economic case is clearest and the engineering is hardest. The economic case is simple. A messaging service is valuable almost entirely because of who else uses it. Nobody switches to a better messenger if none of their friends are on it, so the incumbent's advantage comes not from the quality of its product but from the size of its network. If a rival's users could message the incumbent's users directly, that advantage would shrink to whatever the incumbent's product was actually worth. Email works this way: nobody needs a Gmail account to write to a Gmail user. Telephone networks work this way. Modern messaging does not, and Article 7 of the Act sets out to change that for the services it designates. The engineering is hard because modern messaging is end-to-end encrypted, and end-to-end encryption, done properly, depends on both ends being trustworthy and on a reliable way to learn the other end's keys. A walled garden is in this case partly a security architecture. Opening it without weakening the security of billions of conversations is a genuine design problem, not a pretext, although, as Chapter 7 discusses, genuine design problems can still be used as pretexts. What Article 7 requires Article 7 applies to number-independent interpersonal communications services designated under the Act, which in practice means WhatsApp and Messenger. It obliges the gatekeeper, upon request and free of charge, to make certain basic functionalities interoperable with the service of any other provider that offers or intends to offer such services in the EU. The functionalities are phased. From the start, end-to-end text messaging between two individual users must work, together with the sharing of images, voice messages, videos and other attached files between them. Within two years of designation, the same must work in groups: text messaging within groups of individual users and the sharing of images, voice messages, videos and files in those groups. Within four years, voice and video calls between two individuals and between a group and an individual must work. The Commission can extend these deadlines where the gatekeeper shows the extension is necessary to ensure effective interoperability and preserve security. The Article contains three further requirements that shape everything else. The level of security, including end-to-end encryption where applicable, that the gatekeeper provides to its own users must be preserved across the interoperable services. The gatekeeper must publish a reference offer setting out the technical details and general terms and conditions of interoperability, and must respond to any reasonable request within three months by making the functionalities available. And end users must remain free to decide whether to use the interoperable functionalities at all. The gatekeeper may collect and exchange only the personal data strictly necessary to provide interoperability, and may take strictly necessary and proportionate measures to ensure that third-party providers do not endanger the integrity, security and privacy of its services. Table 2 sets out the phased obligations against what Meta has said and done. Table 2. Article 7 messaging interoperability: legal timetable and Meta's implementation for WhatsApp. Functionality Legal timing Meta's stated plan Position by October 2026 One-to-one text, images, voice notes, video, files From designation, compliance by March 2024 Reference offer published 6 March 2024 Third-party chats announced November 2025 with BirdyChat and Haiket Group text and file sharing Within two years of designation Groups promised for 2025 Available "once our partners are ready" One-to-one and group voice and video calls Within four years of designation Calling planned for 2027 Not yet due Sources: Regulation (EU) 2022/1925, Article 7; Meta announcements of September 2024 and November 2025. How Meta built it Meta published the WhatsApp reference offer on 6 March 2024, the day before the compliance deadline, together with an engineering description of its design. The design is the clearest available example of how a gatekeeper fills the gaps an obligation leaves open, and so it is worth describing carefully. WhatsApp and Messenger both use the Signal Protocol for end-to-end encryption. In that protocol, each user's device generates long-term identity keys and a supply of one-time "prekeys", and uploads the public halves to the service's server. When someone wants to start a conversation, their device fetches the recipient's public keys from the server, uses them to establish a shared secret, and from then on derives fresh keys for every message so that compromising one key does not expose past or future messages. The server relays encrypted messages and holds public keys, but it cannot read message contents. The difficulty for interoperability lies in two places. The first is key distribution: a WhatsApp user who wants to message a user of another service has to obtain that user's keys from somewhere, and has to trust that the keys really belong to that user rather than to an attacker who inserted their own. Within WhatsApp, Meta's server is the trusted directory. Across two services, someone has to vouch for identities on the other side. The second is the protocol itself: the two clients must agree on how messages are formatted, encrypted, acknowledged and delivered, including attachments, receipts and error cases. Meta's answer was to make the third party join WhatsApp's system rather than build a bridge between two systems. Third-party clients connect to WhatsApp's servers using WhatsApp's own client-to-server protocol. Transport between client and server is encrypted using the Noise Protocol Framework, as it is for WhatsApp's own apps. Messages are structured as protocol buffers, encrypted with the Signal Protocol and carried inside XML stanzas derived from XMPP, the older open messaging standard that WhatsApp's internal protocol grew out of. The third party authenticates its users to WhatsApp using a token scheme based on OpenID, with tokens cryptographically signed by the third party to prove that a given user identifier belongs to it. Meta said it preferred that third parties use the Signal Protocol too, but would consider other encryption protocols if the provider could demonstrate that they offered the same security guarantees. Meta also offered a variant in which the third party places its own proxy server between its clients and WhatsApp's servers, so that its apps do not connect to Meta directly. Meta's engineers noted the costs plainly: with a proxy in the middle, WhatsApp would no longer have a direct connection to both clients, and the proxy would see all the chat metadata passing through it. This architecture has real security merits. It avoids the server-side bridges that some earlier interoperability proposals relied on, in which a message would be decrypted by one service and re-encrypted for the other, destroying end-to-end encryption. It keeps WhatsApp's existing spam and abuse defences in the loop, since WhatsApp's servers still see the delivery metadata they use to detect bulk messaging. And it puts the encryption into the clients, where it belongs. Meta's engineers were also candid about its limits. Without control of both endpoints, they wrote, Meta could not guarantee what a third-party provider did with the messages it sent or received, and so could not make the same privacy promise for interoperable chats that it made for chats between WhatsApp users. Preserving privacy in an interoperable system, they wrote, was a shared responsibility. Notice what the design does to the balance of power. The rival must implement WhatsApp's protocol, sign WhatsApp's agreement, connect to WhatsApp's servers or route through a proxy that WhatsApp's terms govern, and accept WhatsApp's judgement about whether its encryption is good enough. Each of these choices can be defended on security grounds. Together they mean that the gatekeeper does not merely open a door; it writes the building code for anyone who wants to use it. The user's side of the door On the user's side, the design choices were equally consequential. Article 7 requires that users remain free to decide whether to use interoperable functionalities, and Meta built interoperability as an opt-in. In its September 2024 update, Meta described an onboarding flow in which users would receive a notification explaining third-party chats, could turn the feature on, and could choose which third-party apps they wanted to receive messages from. They could keep third-party chats in a separate folder or merge them into a single inbox. Meta also said it would support reactions, direct replies, typing indicators and read receipts, features beyond the bare minimum the Act requires. In November 2025, announcing the first partners, Meta said users in Europe would see a notification in WhatsApp's Settings tab over the coming months and could turn third-party chats on or off at any time. The opt-in is a reasonable reading of the law and also a significant brake on adoption. A rival messenger's pitch is weaker if its users can reach only those WhatsApp users who have found a setting and switched it on. The user interface also emphasizes the differences between WhatsApp chats and third-party chats, which is honest, since the privacy guarantees do differ, but which also signals caution to every user who sees it. These are the kinds of choices the Act leaves to the gatekeeper. Each is defensible. Each shapes the market. Why the big players stayed away The first two services Meta announced were BirdyChat and Haiket, both small European start-ups. BirdyChat, a messenger aimed at work conversations, has since rebranded as Balsio. Haiket is a voice-first messaging app. None of the large, established services, Signal, Telegram, Threema, Apple's iMessage or Google Messages, announced that it would connect. There are good reasons for this, and they say more about the structure of Article 7 than about the demand for interoperability. For a service like Signal, whose entire proposition is that it collects as little metadata as possible and is run by a non-profit that answers to no advertiser, joining WhatsApp's network on WhatsApp's terms would mean exposing its users' metadata to Meta's servers and accepting Meta's contractual conditions. Its users chose it precisely to avoid that. For a service with its own large network, such as Telegram, interoperability offers little: its users already have WhatsApp installed if they want it, since most people use more than one messenger, and connecting would hand WhatsApp's users easy access to its community. For Apple, which is not obliged to open iMessage, connecting to WhatsApp would invite pressure to reciprocate. And for any service, implementing a rival's proprietary protocol creates an ongoing engineering dependency on that rival's release schedule and goodwill. The beneficiaries Article 7 can realistically serve are therefore new entrants with a distinctive product and no network, for whom access to WhatsApp's users is the only way to get started. That is a real and worthwhile group, but it is a narrower one than the law's supporters imagined. The Commission's April 2026 review noted that new messaging applications had emerged because of the interoperability requirements, which is true, and modest. The other way to open a network Article 7 took one route to interoperability: oblige the dominant service to open its own protocol. There is another route, which is to agree on a shared open standard that every service implements. The history of the alternative is instructive. The Internet Engineering Task Force chartered a working group in 2022 called MIMI, for More Instant Messaging Interoperability, explicitly with the DMA in mind. Its work builds on Messaging Layer Security, an IETF standard published as RFC 9420 in July 2023, which provides efficient end-to-end encrypted group key agreement for groups ranging from two to thousands of members. MLS was designed by engineers from many companies, including some of the gatekeepers, and it solves problems, such as adding and removing members of large encrypted groups, that the Signal Protocol's group mechanisms handle less efficiently. In principle, an open standard for message formats and delivery on top of MLS would let any provider talk to any other on equal terms, without one of them writing the building code. A parallel story played out in SMS's successor. Rich Communication Services, the carrier-backed standard for richer text messaging, had for years lacked end-to-end encryption in its common profile, and Apple did not support it. Apple added RCS to the iPhone in 2024 with iOS 18, and in March 2025 the GSMA published a new version of the RCS Universal Profile that specifies end-to-end encryption based on MLS, with Apple saying it would add support in future software updates. Here a shared standard moved because of market pressure and the threat of regulation in several jurisdictions, rather than through a specific legal obligation in Europe. Standards are slow, and the DMA's timetable did not wait for MIMI to finish. But the contrast clarifies what Article 7 traded away. By placing the obligation on the gatekeeper and allowing it to define the interface through its reference offer, the Act secured something concrete within months, a working, encrypted, documented interface, at the cost of letting the gatekeeper's architecture become the de facto standard. A requester can ask for changes, and the Commission can in principle use Article 8 to specify the measures, but the default is the gatekeeper's design. A brief episode from outside the DMA shows what happens with no obligation at all. In December 2023 a start-up called Beeper released Beeper Mini, an Android app that reverse-engineered Apple's iMessage protocol so that Android users could send end-to-end encrypted iMessages to iPhone users. Within days Apple changed its systems to block it, citing the security risks of unauthorized access to its servers. Beeper tried several workarounds before giving up. Whatever one thinks of reverse-engineering a company's servers, the episode illustrates the baseline: without a legal obligation, a dominant messaging service can close an unauthorized door as fast as someone opens it. Article 7 changes that baseline, which is its real achievement. Groups, calls and the hardest parts still to come The one-to-one stage was the simplest. Group messaging across providers is harder because groups have state: who is a member, who added whom, who can change the name, what happens when someone leaves. In WhatsApp's architecture, encrypted group messages use "sender keys" that each member distributes to every other member, and the server keeps track of membership. When some members of a group are on a different service, both services must agree on how membership changes are authorized and propagated, how keys are rotated when someone leaves, and what each side's server is allowed to know. Meta said in 2024 that it would add group support in 2025. When it announced its first partners in November 2025, it said groups with third-party users would become available once partners were ready to support them, which places the burden of the next step on the requesters. Calls are harder still, because real-time audio and video depend on low-latency connections between clients, often negotiated through relay servers, and the encryption must be established in the call setup. Meta has said calling will come in 2027, consistent with the four-year deadline. The design choices for calls will repeat the pattern of the first stage: Meta will publish an interface, and requesters will either accept it or ask the Commission to intervene. The lesson of this chapter is that messaging interoperability under the DMA has worked as an engineering project and has, so far, done little to the market. Meta built a secure, documented interface roughly on time and went somewhat beyond the minimum. The market effect is small because the obligation lets the gatekeeper set the terms of entry, because the users the gatekeeper holds must opt in to be reachable, and because the services most able to challenge WhatsApp have little reason to join its network on its terms. Whether that counts as success depends on what one expected. If the aim was to make it possible for a new messenger to reach WhatsApp's users without asking Meta's permission in the commercial sense, it has been met. If the aim was to dissolve WhatsApp's network advantage in Europe, it has not, and the Act as written was unlikely to achieve it. Chapter 4: Opening the App Store For most of the iPhone's history, there was exactly one way to put software on it: the App Store. Apple reviewed every app, signed every binary, distributed every update and took a commission on every sale of digital goods. The arrangement was, from Apple's point of view, the foundation of the iPhone's security and its reputation for quality. From the point of view of developers and competition authorities, it made Apple the sole gatekeeper between a large share of Europe's smartphone users and every company that wanted to sell them software. Article 6(4) of the Digital Markets Act was written to end that monopoly of distribution. This chapter examines what it took to do so, why the result has so far been modest, and what that teaches about opening a platform whose owner controls not just the rules but every step of the user's path. What the obligation says Article 6(4) requires a gatekeeper to allow and technically enable the installation and effective use of third-party software applications and app stores on its operating system, and to allow them to be accessed by means other than the gatekeeper's own services. Users must be able to set downloaded apps and stores as defaults, and the gatekeeper must not prevent a third-party store from prompting users to make it the default. The gatekeeper may take measures to ensure that third-party apps and stores do not endanger the integrity of its hardware or operating system, provided they are strictly necessary, proportionate and duly justified, and may apply measures and settings other than default settings that let users protect their security effectively. On Android, the obligation changed relatively little. Android has always allowed users to install apps from outside Google Play, and other stores, such as Samsung's Galaxy Store and the open-source F-Droid, have existed for years. The friction on Android lay elsewhere: in the warnings shown when a user enables installation from an unknown source, in Google's agreements with phone makers about which apps are pre-installed, and in the fact that Google Play's position made it the store developers could not do without. On iOS, by contrast, the obligation required Apple to build a distribution system from scratch. How Apple built it Apple's first answer, which shipped with iOS 17.4 in March 2024, had three components. First, alternative app marketplaces: approved developers could run their own app stores, which users would install from the marketplace's website and which could then install and update other apps. Second, notarization: every iOS app distributed in the EU, whether through the App Store or elsewhere, would pass through an Apple review process focused on baseline platform integrity rather than the App Store's content and business rules. Apps that passed would be signed by Apple and could be installed. Third, a new set of business terms, described in Chapter 5, including a Core Technology Fee of €0.50 for each first annual install beyond one million a year, charged to developers who chose the new terms whether or not their apps were distributed through the App Store. In iOS 17.5, two months later, Apple added web distribution, allowing qualifying developers to offer apps for download directly from their own websites. Each component had a defensible rationale and each became a point of conflict. Notarization meant Apple continued to review, and could refuse, every app installed on an iPhone in Europe, though on narrower grounds than App Store review. Apple's stated checks covered accuracy, functionality, safety, security and privacy: whether the app was what it claimed to be, whether it worked, whether it contained malware or tried to escape its sandbox, and whether it collected data without permission. Those are reasonable checks, and Android's Play Protect does something similar after the fact. But they also preserved for Apple a veto point over every app, and the boundaries of "safety" and "accuracy" are matters of judgement. The eligibility rules decided who could run a marketplace or distribute from the web. In 2024 Apple required a prospective marketplace operator either to provide a stand-by letter of credit of €1,000,000 from a highly rated financial institution or to have been a member of the Apple Developer Program for two continuous years with an app that had more than one million first annual installs in the EU in the preceding year. Web distribution required a similar track record. Apple argued that marketplaces took on responsibilities for content moderation, fraud, refunds and user support, and that it needed some assurance that operators could meet them. Small developers and open-source projects argued that the rules excluded precisely the newcomers the DMA was meant to help. The user journey decided how hard it was to use the result. To install a marketplace, a user had to visit the operator's website, authorize the marketplace in Settings, and pass through a sequence of system screens describing what the marketplace was and warning that apps it installed were not reviewed by Apple. Apple described these screens as informing users. Developers called them scare screens. In both cases the screens did what screens do: they reduced the number of people who completed the process. The first stores, and one that closed Several marketplaces launched despite the friction. AltStore PAL, a small open-source project that had long distributed apps to iPhones through developer workarounds, opened in April 2024 as one of the first approved marketplaces. Setapp Mobile, a subscription marketplace run by the Ukrainian company MacPaw, entered beta in May 2024 and opened to the public that September at €9.49 a month. Epic Games, which had fought Apple in American courts since 2020 over the App Store's rules, opened the Epic Games Store on iPhone in the EU in August 2024, bringing Fortnite back to iOS in Europe. Epic's path is a small history of the DMA in itself. In March 2024 Apple terminated the developer account of Epic's Swedish subsidiary, which Epic intended to use to run its European store, citing Epic's past conduct and public criticism of Apple's terms. After the Commission made inquiries, and Epic's founder publicly accused Apple of violating the Act, Apple reinstated the account within days. Apple also initially rejected Epic's marketplace for reasons related to the design of its install buttons, before approving it. Each step was small. Together they showed how many discretionary decisions sat in Apple's hands even after the obligation applied. In January 2026 MacPaw announced that it was shutting Setapp Mobile, citing Apple's "still-evolving and complex terms". The closure was a public setback for the idea that the DMA would produce a thriving market of iOS app stores, and it set off a public argument between Apple and the Commission over who was to blame: the store's own limited appeal, or the conditions under which it had to operate. Both explanations contain truth. An app store needs users to attract developers and developers to attract users, the same chicken-and-egg problem every platform faces, and a store that users must take deliberate steps to install, behind warning screens, with no pre-installation and no presence on the home screen, starts with very few of either. The Commission's verdict and Apple's revisions In June 2024 the Commission opened an additional investigation into Apple's terms for alternative distribution. In April 2025, alongside the anti-steering fine discussed in Chapter 5, it sent Apple preliminary findings on those terms. It identified three problems. The Core Technology Fee disincentivized developers from using alternative distribution, since choosing the new business terms exposed them to a new fee. The eligibility requirements were overly strict and hampered developers' ability to distribute through alternative channels. And Apple made it overly burdensome and confusing for users to install apps through those channels. Apple, the Commission said, had failed to show that these measures were strictly necessary and proportionate. Apple's revisions came in stages. In June 2025 it announced that iOS 18.6 would bring an updated installation experience for alternative marketplaces, that a new interface would allow apps to be downloaded from within other apps later that year, and that it would move to a single business model by replacing the Core Technology Fee with a commission on sales. The terms that took effect on 1 October 2026 completed that transition. The per-install fee is gone, replaced for apps distributed through alternative marketplaces or the web by a Core Technology Commission of 5 percent on sales of paid apps and digital goods and services. Marketplace operators earning less than €10 million in global revenue, and less than €1 million in lifetime revenue from their EU marketplace app, are exempt from the commission on download or subscription fees for access to apps. And eligibility to run a marketplace or distribute from the web now requires meeting any one of seven criteria: a moderate financial stability score from Dun & Bradstreet, being publicly traded or owned by a public company, having venture funding from an established investment firm, having completed a financial audit by a licensed accountant, being an approved government, educational or non-profit entity, providing a $1,000,000 stand-by letter of credit, or having one million first annual installs worldwide. These are substantial changes. The new eligibility list opens marketplace operation to a much wider range of companies than the original rules did. The switch from a per-install fee to a per-sale commission removes the most frightening feature of the 2024 terms, the risk that a free app that suddenly became popular would incur fees bearing no relation to its revenue. A Commission spokesperson welcomed the August 2026 announcement and said the Commission would monitor implementation. Whether the changes fully answer the April 2025 preliminary findings is a matter the Commission had not formally resolved by early October 2026. Why the fee structure mattered so much The change from a per-install fee to a per-sale commission deserves a closer look, because it shows how a single design parameter can decide whether an obligation has any practical effect. Consider a hypothetical developer of a popular app that is free to download and earns money from a minority of users who buy an optional subscription. Suppose the app reaches three million first annual installs in the EU in a year and earns €1 million in revenue there. Under the 2024 alternative terms, the Core Technology Fee would have charged €0.50 for each first annual install above one million, that is, €0.50 on two million installs, or €1 million. The fee would have consumed the developer's entire European revenue, before any commission on sales. Under the October 2026 terms, if the same developer distributed the app through an alternative marketplace or from its own website, Apple's Core Technology Commission of 5 percent on its €1 million of sales would come to €50,000. The example is constructed, but the arithmetic is Apple's, and it explains why developers treated the original fee as a deterrent. A per-install fee charges for reach regardless of revenue, which punishes exactly the business model, free apps with a paying minority, on which much of the mobile economy runs. It also exposes a developer to a risk it cannot control: an app that goes viral can incur a large bill before it has earned anything. Apple softened the fee over time, exempting developers with no revenue and giving small developers a period free of it, but the structure remained. A commission on sales, by contrast, scales with what the developer actually earns. Whatever one thinks of its level, it does not make distribution outside the App Store a gamble. The fee also shows why a regulator cannot assess openness by looking at permissions alone. In March 2024 the formal right to distribute outside the App Store existed. The fee attached to exercising that right made it uneconomic for a large class of developers. The Commission's April 2025 finding that the fee disincentivized alternative distribution was a finding about a price, reached under an obligation that says nothing about prices. Browser engines: permission without uptake The DMA's effect on web browsers on iOS illustrates a different failure mode, in which permission is granted but the conditions make it too costly to use. Before the DMA, Apple required every browser on iOS to use WebKit, its own browser engine. Chrome, Firefox and Edge on the iPhone were in effect Safari with different interfaces. Article 5(7) prohibits a gatekeeper from requiring business users to use its web browser engine, and in March 2024 Apple began allowing browsers in the EU to use alternative engines through a framework called BrowserEngineKit. The conditions were demanding. A browser using its own engine had to be a separate app distributed only to EU users, had to meet a list of security requirements, and could not easily be developed and tested by engineers outside the EU. For a company like Google or Mozilla, maintaining a separate engine port, a separate app and a separate release process for one region was a large investment with uncertain return. For well over a year after the obligation applied, no major browser vendor shipped a version of its iOS browser to European users with its own engine. The rule had changed. The market had not, because the gatekeeper's implementation made exercising the right expensive. This pattern, in which a gatekeeper complies with the letter of an opening obligation while surrounding it with costs, conditions and frictions that deter use, is the central practical problem of the DMA. It is not always bad faith. Each condition can usually be justified by reference to security, quality or legal risk. But each condition also subtracts from the effectiveness the Act requires, and the Act's anti-circumvention rule, which prohibits degrading the conditions or quality of services or making it unduly difficult for users to exercise their rights, exists precisely to catch the cumulative effect. The Android mirror It would be wrong to read the app store story as purely about Apple. Google's Android shows the same tension from the other direction. Android has long allowed sideloading, but Google has steadily tightened the conditions around it. In August 2025 Google announced that it would require developers distributing Android apps outside Google Play on certified devices to verify their identity with Google, beginning in a handful of countries in September 2026 and extending globally in 2027, citing the prevalence of malware in sideloaded apps. After criticism from developers and advocates of open software, it said it would provide an option for experienced users to install apps from unverified developers. The policy is not a DMA measure and applies worldwide. But it raises exactly the question Article 6(4) poses: when does a gatekeeper's measure to protect users from malicious software become a gate on who may distribute software at all? Verification of developer identity is a reasonable anti-malware tool. It also gives the operating system owner a registry of every developer who may reach its users outside its store, and a lever over each of them. Under the DMA, such a measure applied to Android in Europe would have to be strictly necessary and proportionate to protect the integrity of the operating system. Whether a mandatory registry passes that test is an open question that the Commission will eventually have to answer. What distribution really requires The experience of the first two and a half years points to a conclusion that was not obvious in 2022. Allowing third-party stores to exist is not the same as allowing them to compete. Competition in distribution requires that a rival store can reach users with friction comparable to the incumbent's, that developers can list in it without paying the incumbent a fee that makes the rival uneconomic, and that the incumbent's review and eligibility gates are narrow enough that the rival, not the incumbent, decides what it sells. Each of these conditions is a specification problem. Each concerns screens, fees, checklists and timelines that the Act does not describe and that the gatekeeper designs in the first instance. The Commission has moved, through preliminary findings and pressure, towards specifying more of these details. The October 2026 Apple terms are the product of that pressure. Whether they produce a viable market of iOS app stores will depend on things no regulator controls, including whether any store can offer users something they cannot get from the App Store. But the obstacles that the gatekeeper's design placed in the way are lower than they were, and that is the most the law can do. Chapter 5: Billing, Steering and the Price of Access If Chapter 4 was about who may hand software to a user, this chapter is about who gets paid when the user spends money inside it. It is the front on which the Digital Markets Act has produced the most visible conflict, the first fines and the most frequent rewriting of corporate terms. It is also the front where the gap between a general legal obligation and a specific implementation is widest, because the obligation in question, that developers must be able to steer users to other offers "free of charge", says nothing explicit about the one thing everyone actually cares about: the commission. Why steering matters The economics are worth setting out plainly. For more than a decade, Apple and Google charged developers a commission of 30 percent on sales of digital goods and services made through their app stores, reduced to 15 percent for small developers and for subscriptions after their first year. Developers selling digital content had to use the store's own in-app payment system, so the commission could be collected. And they were forbidden to tell users, inside the app, that the same subscription or item could be bought more cheaply on the web. This last rule, the anti-steering provision, is what made the commission stick. A developer who could say "subscribe on our website and pay 20 percent less" would have shifted many customers out of the store's payment system, and the store would have had to lower its commission to keep them. The anti-steering rule is therefore not a side issue. It is the enforcement mechanism of the commission. Remove it and the commission becomes contestable: the store can still charge what it likes on purchases that go through its system, but it must compete with the developer's own checkout for each customer. Most of the fight described in this chapter is a fight over whether, after the anti-steering rule is formally removed, the store may recapture the commission by other means, for instance by charging a fee on purchases made after a user follows a link out of the app. What the Act says, and what it does not Two provisions of Article 5 govern this ground. Article 5(4) requires a gatekeeper to allow business users, free of charge, to communicate and promote offers to end users acquired through its core platform service or through other channels, and to conclude contracts with those end users whether or not they use the gatekeeper's service to do so. Article 5(7) prohibits a gatekeeper from requiring business users to use its payment service, or technical services supporting payment such as in-app purchase systems, in the context of services they provide through the gatekeeper's platform. Read together, these say that a developer must be able to tell users about cheaper offers elsewhere, must be able to link them there, and cannot be forced to use the store's payment system. They do not say that the store must stop charging a commission. Article 6(12), which requires fair, reasonable and non-discriminatory general conditions of access to app stores, assumes that access to an app store may be priced. Apple has argued throughout that it is entitled to be paid for the technology, tools and distribution it provides, and that removing payment from the purchase flow does not remove its contribution to the sale. The question the Act leaves unresolved is where "free of charge" steering ends and a legitimate charge for platform services begins. The question is not academic. If a store may charge the same commission on a sale that follows a link-out as on a sale through its own payment system, steering becomes pointless: the developer pays the commission either way and adds its own payment processing costs on top. If the store may charge nothing on such sales, every developer has an incentive to route every purchase out of the app, and the store's revenue from those apps falls to zero while it continues to distribute, update and review them. Between those extremes lies a range of fees, and the Act provides no number. Apple's sequence of answers Apple's response to this ambiguity has passed through three distinct designs, summarized in Table 3. Each reflects the same strategy: comply with the formal requirement to allow steering and alternative payments, while setting fees that limit how much the developer can save by using them. Table 3. Apple's business terms for apps in the EU, 2024 to 2026. Terms App Store sales Steered or alternatively processed sales Apps outside the App Store Source Standard terms before the DMA 30%; 15% reduced rate Steering largely prohibited Not permitted Apple Alternative terms, March 2024 17%; 10% reduced; plus 3% for Apple payment processing Permitted under alternative terms €0.50 per first annual install above one million Apple June 2025 terms 2% initial acquisition fee plus 5% or 13% store services fee Plus 5% Core Technology Commission Per-install fee pending transition Apple; 9to5Mac Unified terms, 1 October 2026 26% with Apple payments; 15% reduced 20% (10%) in-app alternative payment; 15% (10%) on link-out sales within 7 days 5% Core Technology Commission Apple The March 2024 design created two parallel sets of terms. Developers could stay on the old terms, with the old 30 and 15 percent commissions and no ability to steer or use other payment systems, or switch to the new terms, which lowered the App Store commission to 17 percent (10 percent at the reduced rate) plus 3 percent if they used Apple's payment processing, allowed alternative payment providers and linking out, and added the Core Technology Fee of €0.50 for each first annual install beyond one million. The fee was the sting. For a developer with a popular free app and modest revenue, it could exceed what the old commission would have cost. Many developers, and especially those with large free user bases, stayed on the old terms. The Commission had opened a non-compliance investigation into Apple's steering rules in March 2024, three weeks after the compliance deadline. In June 2024 it sent Apple preliminary findings. In April 2025 it adopted a non-compliance decision, finding that Apple's restrictions prevented app developers from fully benefiting from alternative distribution channels and that consumers could not fully benefit from cheaper offers, and fined Apple €500 million. It ordered Apple to remove the technical and commercial restrictions on steering within sixty days or face periodic penalty payments. Apple's June 2025 response, announced on the day it said it was making changes required by the Commission and stating that it disagreed and would appeal, replaced the earlier structure for steering apps with a layered set of fees. An initial acquisition fee of 2 percent applied to sales to new users in their first six months after download. A store services fee applied at either 5 percent, for a limited tier of services covering distribution, trust and safety and app management, or 13 percent for the full set of App Store services, reduced to 10 percent for small businesses and for subscriptions after the first year. For apps that communicated and promoted offers through actionable links, a Core Technology Commission of 5 percent applied to the resulting sales. Developers could now steer to any destination they chose, including through in-app web views. Apple said it would move all developers to a single business model, with the per-install fee replaced by a per-sale commission, by the start of 2026. The terms that finally unified the model were announced in August 2026, after what Apple described as close collaboration with the Commission, and took effect on 1 October 2026. They abolish the initial acquisition fee, the store services fee and the Core Technology Fee. Within the App Store, the commission is 26 percent on sales processed through Apple's in-app purchase system, 20 percent on sales processed through an alternative payment provider within the app, and 15 percent on sales made within seven days after a user taps an out-of-app link, each falling to 15, 10 and 10 percent respectively for small businesses, certain partner programmes and subscriptions after their first year. Apps distributed through alternative marketplaces or the web pay a 5 percent Core Technology Commission on sales of paid apps and digital goods and services. Developers may for the first time offer alternative payment options alongside Apple's own in-app purchase rather than instead of it, subject to new child-safety protections. A worked example Consider a European developer selling a subscription at €10 a month, in its first year, outside any reduced-rate programme. Before the DMA, Apple took €3.00 from each payment through the App Store, and the developer could not tell users that a web subscription existed. Under the October 2026 terms, the developer has three options within the App Store. If it uses Apple's in-app purchase, Apple takes €2.60. If it processes payment itself or through a payment company inside the app, Apple takes €2.00, and the developer pays its own payment provider's fees on top. If it sends the user to its website through a link and the user subscribes within seven days, Apple takes €1.50, and again the developer pays its own processing costs. In the subscription's second year, the three figures fall to €1.50, €1.00 and €1.00. If the developer distributes the app through an alternative marketplace instead, Apple's charge is €0.50, plus whatever the marketplace itself charges. Three things follow from these numbers. The first is that steering is now worth something. The difference between €2.60 and €1.50 per month, less the developer's own payment costs, is a real saving at scale, and a developer can now tell users about it inside the app. The second is that Apple still collects a substantial share of steered revenue, so the price advantage developers can offer users is smaller than the headline removal of anti-steering rules suggests. The third is that the seven-day attribution window matters a great deal. A user who taps a link, leaves, and subscribes on the website ten days later generates no commission under these terms. The window is a negotiated number. It reflects a judgement about how long a purchase can fairly be attributed to the app, and it is exactly the kind of detail that decides the economic effect of the rule. Google's version, and what the fine revealed Google Play's history runs in parallel. Google had long allowed some alternative billing in certain jurisdictions under pressure from regulators in South Korea and elsewhere, typically with a commission only a few percentage points below its standard rate. Under the DMA it introduced programmes allowing developers in Europe to offer alternative billing and to steer users to external offers, with fees on the resulting transactions. The Commission opened a non-compliance investigation into Google Play's steering rules in March 2024, sent preliminary findings in March 2025, and adopted a decision on 23 July 2026, fining Google €430 million. The decision found that Google prevented app developers from freely communicating and promoting offers and restricted the channels through which they could steer users. It also found that the level of the steering-related fees Google charged, and the length of the period during which it charged them, went beyond what is compliant with the DMA. Google was given sixty days to comply, under threat of periodic penalty payments of up to 5 percent of its average daily worldwide turnover. That period expired in late September 2026, and the Commission had not announced by early October whether it considered Google's revised terms compliant. The Google decision is significant for what it implies about the Commission's reading of Article 5(4). The Commission did not say that any fee on steered sales was prohibited. It said that the level of the fees and the duration of the charging period went too far. That is the language of price regulation. In effect, the Commission has accepted that a gatekeeper may charge something on transactions that follow steering, while reserving to itself the judgement of how much and for how long. The phrase "free of charge" in the Regulation has become, in enforcement, a ceiling to be negotiated, and the numbers in Apple's October 2026 terms, including the seven-day window, are best read as the product of that negotiation. Who gets the saving A further question sits beneath the fee negotiations and receives less attention than it deserves: if developers pay the gatekeeper less, do users pay less? The DMA's stated purpose is fairness and contestability, and its supporters often describe lower prices for consumers as the end result. But a reduction in the commission is first a transfer from the platform to the developer. Whether any of it reaches the user depends on the developer's pricing decisions and on how much competition the developer faces. Economic reasoning suggests mixed outcomes. Where a developer faces strong competition, for instance among music streaming services, it has reason to pass savings on, or to use them to offer discounts for web purchases that draw users out of the store's payment system. Where a developer has a distinctive product with few substitutes, it may keep the saving as margin. Many developers set a single price across platforms and countries for simplicity, and a reduction in European commissions may not change prices at all. The steering obligation makes a different kind of competition possible: a developer can now show users a cheaper web price inside the app, which makes the store's commission visible to the user in a way it never was. That visibility is itself a form of discipline on the store, since a user who sees that the same subscription costs less on the web may ask why. Reliable public evidence on pass-through under the DMA is still thin, and the Commission's review of April 2026 did not quantify it. This is a gap worth closing. A regime that justifies itself partly by consumer benefit should be able to say whether consumers have benefited, and the data to answer the question, prices for the same digital goods across channels and over time, is not hard to collect. The view from across the Atlantic The European approach is clearer when set beside the American one. In the United States, the same issue has been fought through private litigation. In the case brought by Epic Games, a federal court in 2021 ordered Apple to allow developers to include links to outside purchasing options. Apple complied by permitting links but charging a commission of 27 percent on resulting purchases. In April 2025 the court found that Apple had wilfully violated the injunction, barred it from charging any commission on purchases made through such links, and referred the matter for possible criminal contempt proceedings. Apple appealed, and the question of whether it may charge anything at all on such purchases has continued to be litigated. The two systems arrived at different answers to the same question. An American court, applying an injunction in a single case, prohibited the commission on link-outs outright. The European Commission, applying a regulation across all developers, has accepted a reduced commission with a limited attribution window. The difference is not only legal. A court enforcing an injunction can ask whether a party evaded its order. A regulator administering an ongoing obligation must decide what an acceptable business model looks like, and has chosen to allow the gatekeeper to keep one. What has been settled and what has not After two and a half years, developers in Europe can tell iPhone and Android users about cheaper offers, link to them, and process payments themselves. Those are real changes, and some large developers, music and video streaming services among them, have used them. What has not happened is a collapse of app store commissions to the cost of payment processing, which is what some advocates expected. The DMA did not set a price, and the gatekeepers have used that silence to preserve most of their revenue on the transactions they can still see. Whether that outcome is a failure depends on what one thinks the commission pays for. If it pays mainly for payment processing, the remaining charges are rents protected by a reluctant regulator. If it pays for a platform that took decades and many billions to build and still requires continual investment, a reduced but positive charge on steered sales is a reasonable settlement. The Act did not decide between these views. It created a process in which the Commission would, case by case and number by number, decide for it. That process, with all its slowness and opacity, is now where the price of access to Europe's smartphone users is set. Chapter 6: Specifying the Operating System The deepest wall in a mobile ecosystem is not the app store. It is the operating system itself, and in particular the set of capabilities the operating system grants to the platform owner's own products but not to anyone else's. An Apple Watch can display and answer an iPhone's notifications, pair by being held near the phone, switch audio automatically between devices and share the phone's Wi-Fi credentials. For years, a Garmin or Samsung watch connected to an iPhone could do only some of these things, or could do them only clumsily, because the interfaces it needed were private to Apple. The same pattern holds on Android for Google's own services, and it is now repeating itself with AI assistants, where the question is whether a rival assistant can be summoned by voice, see what is on the screen and act inside apps the way the platform's own assistant can. Article 6(7) of the Digital Markets Act addresses this wall. Its application to Apple between 2024 and 2026 is the most fully worked example of the argument this booklet makes, because it shows a regulator abandoning the hope that a general obligation would be self-executing and instead writing, feature by feature, what the gatekeeper must build and by when. The obligation Article 6(7) requires a gatekeeper to allow providers of services and hardware, free of charge, effective interoperability with, and access for the purposes of interoperability to, the same hardware and software features accessed or controlled through its operating system or virtual assistant as are available to its own services and hardware. Business users and alternative providers of services offered together with or in support of core platform services must likewise have free and effective access to the same operating system, hardware or software features that the gatekeeper uses itself. The gatekeeper may take strictly necessary and proportionate measures to ensure that interoperability does not compromise the integrity of the operating system, virtual assistant, hardware or software features, provided it duly justifies them. The key phrases are "the same" features and "effective" interoperability. The obligation is not to provide some interface that does something similar. It is to provide access to the same capability the gatekeeper's own products enjoy, in a way that actually works. That is a demanding standard, and it collides with how operating systems are built. Platform owners routinely use private interfaces for their own products, sometimes because the interface is unfinished, sometimes because it relies on privileged access that would be risky to expose generally, and sometimes because keeping it private is a competitive advantage. Distinguishing among these reasons is hard, and the gatekeeper knows its code far better than the regulator does. From requests to specification Apple's first response to Article 6(7), in March 2024, was a request process. Developers in the EU could submit interoperability requests through a form, and Apple would assess them. The process was opaque to requesters: they did not know how their request was being evaluated, how long it would take, or why it might be refused. Apple, for its part, emphasized that each request required engineering work and a security review, and publicly singled out Meta, saying it had filed more requests than any other company and that granting them could give it access to sensitive user data, such as the contents of notifications and the networks a phone had joined. In September 2024 the Commission opened two specification proceedings under Article 8, the first time it had used the tool. One concerned a set of features used by connected devices such as smartwatches, headphones and televisions. The other concerned the process by which Apple handled interoperability requests in general. In December 2024 it published Apple's proposed measures for consultation, inviting developers and device makers to comment. On 19 March 2025 it adopted two decisions. The first decision specified measures for nine iOS connectivity features, with a timetable for each. The second specified how Apple must handle future requests: improved access to technical documentation on features not yet available to third parties, timely communication and updates to developers on the status of their requests, and a more predictable timeline for reviewing them. Apple challenged the decisions in the EU's General Court, arguing that they would slow its innovation and expose users' data. The form of the first decision deserves attention. It is not a finding that Apple infringed the law. It is a set of engineering requirements: for each feature, what must be available, to whom, with what limitations, and by which iOS release or date. It is, in other words, the specification the Act itself did not contain, written by the regulator after a public consultation in which the people who would use the interfaces could say what they needed. The Commission's own summary describes it as providing legal certainty for Apple and a clear path for developers. A less diplomatic description is that the Commission concluded the general obligation would not be honoured in the way it intended unless it described the result in detail. What has been opened By May 2026 the Commission could point to a substantial list of capabilities that Apple had opened, some in the EU only and some worldwide. Table 4 summarizes the main items as the Commission described them. Table 4. Selected iOS capabilities opened to third parties under Article 6(7), as reported by the European Commission in May 2026. Capability What it enables Release or deadline Availability Notifications to accessories Third-party watches can show and act on iPhone notifications Beta framework in iOS 26.3; deadline 1 June 2026 EU only Proximity pairing Simplified pairing of third-party devices, as with Apple's own Deadline 1 June 2026 EU Automatic audio switching Audio moves between devices without manual selection Deadline 1 June 2027 EU Wi-Fi network information Accessories can join the iPhone's Wi-Fi network iOS 26.4 EU only NFC payment sessions Rings or bracelets can process payments without the phone present iOS 26.0 EU only Wi-Fi Aware and background execution Fast peer-to-peer transfer; accessories keep working when the app is closed iOS 26 Worldwide Source: European Commission, DMA factsheet on smartphone interoperability and data portability, 11 May 2026. The list repays reading. Some features were opened only in the EU, which is what the law requires. Others, including high-bandwidth peer-to-peer Wi-Fi connections for casting and file transfer, the ability for accessories to keep working in the background, a framework for third-party alarm apps and improvements for password managers, were released worldwide. Once an interface has been designed, documented, secured and shipped, maintaining a separate version for Europe can cost more than releasing it everywhere. This is a small but real instance of the "Brussels effect", the tendency of European rules to shape global products because companies find it simpler to adopt one standard than several. Data portability moved in the same way. Apple and Google built a cross-platform transfer tool that lets users move contacts, calendars, messages, photos, documents, Wi-Fi networks, passwords and data from third-party apps between an iPhone and an Android phone, and Apple added a two-way interface allowing rival browsers to import bookmarks, history, passwords and extensions from Safari. These are worldwide features. The DMA's obligation to provide effective data portability applies in Europe, but the engineering, once done, travelled. Anatomy of an opening It helps to see, in concrete terms, what turning a private capability into a public interface involves, because the steps explain both why it takes time and where a reluctant gatekeeper can slow it down. Take notification forwarding. On the iPhone, the Apple Watch receives notifications through a privileged channel built into the operating system: the phone decides which notifications to forward, transmits them over a secure link, and lets the watch send back actions such as replying or dismissing. To open this to a third-party watch, the platform owner must first decide what the interface exposes. Every notification, or only those the user selects? Full content, or only a summary? The ability to reply, or only to dismiss? Each choice trades usefulness against privacy. It must then design a permission model, so that the user explicitly grants the third-party device access and can revoke it. It must build the interface as a public framework, which is different from a private one: it must be stable across operating system versions, documented well enough for outsiders to use, and robust against misuse by an app that requests access deceptively. It must test the framework with real third-party devices, release it in beta to developers, collect their feedback and fix problems before general release. And it must keep maintaining it, because every future change to how notifications work must now preserve the public interface or deprecate it with notice. In the EU case there is one more step: geography. Many of the features Apple opened are available only in the EU. That requires the operating system to know whether a given user is entitled to them, which Apple determines from the user's account and location, with allowances for travel. Building and maintaining two versions of a feature, one for European users and one for everyone else, is itself an engineering cost, and part of why some features were released worldwide instead. None of these steps is a pretext. Each is how competent platform engineering is done. But each is also a place where timing and scope are decided by the gatekeeper unless someone else decides them. A beta can last a release cycle or two. Documentation can be thorough or thin. A permission prompt can be neutral or alarming. A framework can expose the full capability or a reduced version of it. The Commission's specification decisions matter precisely because they fix these choices, or at least their outer limits, in advance: which feature, at what level of functionality, by which release. The cost Apple says Europe pays Apple's response to this regime has been to withhold some new features from European users. In September 2025 it said that Live Translation with AirPods, iPhone Mirroring and some Maps features, including Visited Places and Preferred Routes, had been delayed or not launched in the EU, and it attributed this to the DMA. Its reasoning follows from the structure of Article 6(7). If a new feature for Apple's own devices relies on an operating system capability, the obligation may require Apple to make the same capability available to third-party devices. Apple says it cannot always do so safely or on the same schedule, and that it would rather delay the feature in Europe than ship it to its own products alone and be found in breach. There are two ways to read this. On Apple's reading, the DMA imposes a tax on integration: every time Apple's products work together in a new way, it must engineer a version that works with everyone else's products, and Europeans get new features late or not at all. On the Commission's reading, the delays are a choice. Apple could design new features with third-party access in mind from the start, as the law expects, and the act of withholding features from European users is itself a form of pressure on the regulator. Both readings contain something true. Designing for openness from the beginning is cheaper than retrofitting it, but it is not free, and a company used to building for its own ecosystem first will feel the change as friction. The delays are also strategically convenient for a company lobbying, as Apple did in its submission to the Commission's review of the Act, for the law to be substantially changed. What matters for this booklet is that the conflict over feature delays is a conflict over specification by other means. The Act says "the same features". Apple's delays are a way of controlling when a feature becomes "available to its own services" and therefore when the obligation attaches. The Commission's answer, so far, has been to specify timetables for named features and a process for future ones, narrowing the room for such timing games without eliminating it. Android, AI and the next front The Commission's second use of specification against a mobile operating system came in 2026 and concerned something that barely existed when the Act was drafted: AI assistants. In January 2026 it opened proceedings to specify how Google must comply with Article 6(7) for Android, focusing on the features used by Google's own AI services such as Gemini, and with Article 6(11) on sharing search data. It consulted on draft measures in April 2026 and adopted them on 16 July 2026. The Android measures are the AI-era equivalent of smartwatch notifications. Third-party AI assistants must be able to be activated by voice with their own wake words, as Google's assistant is with "Hey Google", including with the screen locked and without excessive battery drain. They must be able to perform actions in apps on the user's behalf, such as booking a taxi, sending an email or sharing photos, through the user's preferred apps. They must be able to provide reply suggestions in chats and access location-based information. The draft measures also described contextual access points through which a rival service could receive information about what is on the user's screen, matching capabilities Google offers through features such as Circle to Search. The Commission said the measures included safeguards for privacy, device integrity and security. The search data measures address a different but related advantage. Article 6(11) requires Google to give rival search engines access, on fair, reasonable and non-discriminatory terms, to anonymised ranking, query, click and view data generated by Google Search. The July 2026 specification makes three choices that matter. AI chatbots offering search functions qualify as eligible recipients. The data must be anonymised by a multi-layered method developed with privacy experts and aligned with draft joint guidelines on the interplay between the DMA and the GDPR. And Google may pre-assess cybersecurity and data protection risks before sharing, with a pricing formula and access process the Commission said were fair and transparent. The Commission stated that Google should share the same data it collects to optimise its own search services. These decisions push the DMA into contested territory. An assistant that can read the screen and act in apps holds extraordinary power over the user's data and attention. Opening that capability to rivals widens competition; it also widens the number of companies holding that power, some of them far less scrutinized than Google. Search data, even anonymised, can sometimes be re-identified, and the decision to include AI chatbots among the recipients means the data will feed the training and tuning of systems whose outputs are hard to audit. The Commission's choice to specify safeguards in detail, rather than leave them to Google, is consistent with the lesson of Apple. But it also means the Commission now owns the design, and will own the consequences if it goes wrong. What specification teaches The Apple and Google specification proceedings mark the point at which the DMA's enforcers accepted the argument of this booklet in practice. A general obligation to provide "effective interoperability" with "the same features" did not produce effective interoperability on its own. A public process that named features, set dates, required documentation and consulted the people who would use the interfaces did produce it, at least in part, within about eighteen months. Specification has costs. It is slow relative to the pace of product development, with each proceeding taking at least six months and implementation stretching over two years for some features. It requires the regulator to acquire genuine engineering competence, or rely on those it consults. It shifts some responsibility for design outcomes from the company to the state. And it invites the gatekeeper to treat the specification as a ceiling: to open exactly what was named and nothing more. But the alternative has been tried, in the Microsoft case two decades ago and in the first year of the DMA, and it did not work better. The operating system is a commons only to the extent that someone writes down, in detail, which parts are shared. Chapter 7: The Security Argument Every fight described in the previous four chapters had a security dimension, and in almost every one the gatekeeper made a security argument. Apple said that alternative app stores and sideloading would expose iPhone users to malware, fraud and inappropriate content. Meta said that interoperable messaging could not carry the same privacy guarantees as WhatsApp. Apple said that opening notifications and Wi-Fi information to third-party devices would leak sensitive data. Google said that apps installed from outside Google Play carried far more malware than those inside it. These arguments are not invented. Each describes a real risk. They are also, unavoidably, arguments for keeping control in the hands of the party making them. This chapter asks how a regulator, or a citizen trying to judge the debate, should tell the difference between a security measure and a gate. What the Act allows The Digital Markets Act anticipates the argument. Article 6(4), on third-party apps and stores, lets the gatekeeper take measures to ensure that they do not endanger the integrity of its hardware or operating system, and lets it apply measures and settings, other than default settings, that enable users to protect their security effectively. Article 6(7), on interoperability with operating system features, lets the gatekeeper take measures to ensure that interoperability does not compromise the integrity of the operating system, virtual assistant, hardware or software features. Article 7, on messaging, lets the gatekeeper take measures to ensure that third-party providers do not endanger the integrity, security and privacy of its services. Each permission is qualified in the same way: the measures must be strictly necessary and proportionate, and the gatekeeper must duly justify them. Three features of this language matter. The first is that the burden lies on the gatekeeper. It is not for the rival or the regulator to prove that a restriction is unnecessary; it is for the gatekeeper to show that it is necessary. The Commission's April 2025 preliminary findings against Apple's alternative distribution terms turned on exactly this point: Apple, the Commission said, had failed to demonstrate that its measures were strictly necessary and proportionate. The second is that "integrity" is narrower than "security" in its ordinary sense. Article 6(4) speaks of the integrity of the hardware and operating system, not of protecting users from every bad decision. The third is the reference to settings other than default settings. The Act contemplates that a user may choose to take a risk, and that the gatekeeper's role is to make protections available, not to make the choice for the user. Real risks It is worth dwelling on why the security arguments cannot simply be dismissed. A smartphone carries a person's banking credentials, health data, location history, private messages and photographs, and authenticates them to their employer and their government. A single successful attack can be devastating. The centralized review model of the App Store has real security value: it lets Apple scan every app before installation, revoke malicious apps from every device at once, and enforce rules on data collection backed by the threat of removal. Google's own data, cited when it announced developer verification for Android in August 2025, suggested that apps from internet sideloading sources carried many times more malware than apps from Google Play. Fraudsters do build fake banking apps and persuade victims to install them from websites; this is one of the commonest forms of mobile fraud in parts of Asia. Messaging interoperability raises its own risks. End-to-end encryption protects message contents from the service provider, but it does nothing to protect them from a malicious or careless client at the other end. If a WhatsApp user messages someone on a third-party app that stores messages unencrypted on its servers, or that is compromised, the WhatsApp user's message is exposed. Spam and abuse are harder to control when the sender's account is held by another service whose verification standards may be weaker. Meta's engineers were right to say that they could not make the same promise for interoperable chats. Operating system interoperability raises risks of data leakage. Notification content is among the most sensitive data a phone handles: one-time passwords, private messages, medical reminders. Forwarding it to a third-party watch means trusting the watch maker's software and servers with it. Wi-Fi network history can reveal where a person lives, works and travels. A rival AI assistant with access to the screen and the ability to act inside apps holds, by design, as much power as the platform's own. Real pretexts And yet the history of platform security arguments includes many that served commercial ends. The test that most clearly separates the two is consistency: does the gatekeeper apply the same standard to itself and to its partners as to its rivals? Apple's treatment of browser engines is the clearest example. For years Apple required every iOS browser to use WebKit, justifying the rule partly on security grounds: a browser engine is a large attack surface, and Apple could patch WebKit for every browser at once. Yet the security of browser engines is a field in which Google and Mozilla have deep expertise, and both ship their own engines safely on every other platform. Under the DMA, Apple allowed alternative engines in Europe but surrounded them with requirements, discussed in Chapter 4, that made shipping one unattractive. The security case for some conditions was real; the case that only Apple could be trusted to build a safe engine was weak. Apple's own Mac is a second example. Since macOS Catalina in 2019, Apple has allowed Mac users to install software from outside the Mac App Store, provided it is signed and notarized by Apple, a model close to the one it now applies in Europe on iOS. Apple argues that the iPhone is different: more personal, more widely used, a more attractive target. Its software chief, Craig Federighi, told the court in the Epic Games trial in May 2021 that Apple had a level of malware on the Mac that it did not find acceptable, and that the iPhone's more restrictive model was intended to avoid that. The argument is not frivolous. But it concedes that the notarization model works well enough for a computer that holds much of the same data, and it frames the question as one of degree rather than principle. A third example comes from privacy rather than security. Apple's App Tracking Transparency framework, introduced in 2021, requires apps to ask users' permission before tracking them across other companies' apps and websites. It is popular with users and privacy advocates. It also attracted competition complaints, on the grounds that Apple applied its rules more strictly to third-party apps than to its own advertising and that the design of the prompts disadvantaged rivals. France's competition authority fined Apple €150 million over the framework's implementation in March 2025, and in August 2026, following agreements with several European competition authorities, Apple announced changes to the framework in the EU, including an alternative version of the permission prompt that is the only version available in Germany, France, Italy, Poland and Romania. The episode shows privacy measures working, at the same time, as genuine protections and as competitive tools. Neither aspect cancels the other. Safety, content and who sets the standards A distinct strand of the argument concerns not malware but content. The App Store has always applied editorial rules: no pornography, limits on gambling, rules on apps aimed at children, rules on misleading claims. These are not security measures in the sense of protecting the integrity of the operating system. They are judgements about what kinds of software Apple is willing to distribute. Under the DMA, an alternative marketplace may apply different judgements, and Apple's notarization, which is limited to baseline checks of integrity, functionality, safety, security and privacy, does not import the App Store's editorial rules. The consequences became visible in February 2025, when AltStore PAL began distributing an app for viewing pornography, which its developer promoted as the first such app available on the iPhone with Apple's sign-off. Apple responded that it had not approved the app, that it was deeply concerned about the risks such apps posed, especially to children, and that notarization under the EU rules did not let it refuse an app on these grounds. Apple's September 2025 statement on the DMA's effects cited the episode among its arguments. The answer the Act itself supplies is that nothing in it prevents Apple from offering parental controls, content restrictions and other protections that users can switch on, and that what it prevents is Apple acting as the sole arbiter of what lawful software European adults may install. Both sides are right about something. The DMA does shift responsibility for content standards from one company to many, and some of the new decision-makers will set lower standards than Apple did. That is the nature of opening a distribution channel: it brings the content standards of the open web, which are many and varied, to a device whose owner had previously applied one. At the same time, a platform owner's content rules are not neutral safety measures. They have excluded categories of legal software for commercial and reputational reasons as well as protective ones, and their application has been inconsistent over the years. The Act's design, which lets users choose a marketplace, allows protections that do not depend on default settings, and limits the gatekeeper's mandatory review to integrity, is a deliberate choice to treat adult users as entitled to make decisions the platform owner would not make for them. The child-safety provisions that Apple added to its October 2026 terms for alternative payments show how the issue can be handled within that design. Instead of forbidding alternative payments to protect children, Apple attached safeguards to them. That is the shape of a measure the Act contemplates: a specific protection, targeted at a specific risk, applied without closing the door it guards. A test for security claims From the cases so far, a practical test can be assembled, and the Commission's decisions suggest it has been converging on something like it. First, the gatekeeper must identify a specific threat, not a general one. "Sideloading is dangerous" is not a threat. "Apps installed outside the store can request accessibility permissions and use them to read banking passwords" is. A specific threat can be checked against evidence and matched to a specific countermeasure. Second, the measure must address that threat. A per-install fee does not reduce malware. An eligibility rule requiring a million-euro letter of credit protects against a marketplace that cannot pay refunds, a commercial risk, but not against malware, since malicious actors can be well funded. The April 2025 preliminary findings did not accept that Apple's fee and eligibility rules were justified by integrity concerns, and that is the right result under this test. Third, there must be no less restrictive measure that would achieve the same protection. Notarization that checks for malware is less restrictive than full App Store review, and Apple adopted it. A warning screen shown once is less restrictive than a sequence of screens. A setting the user can switch on is less restrictive than a prohibition. The Act's reference to measures "other than default settings" points directly at this kind of alternative. Fourth, the gatekeeper must apply the measure consistently. If a restriction on third-party access to a feature does not also apply to the gatekeeper's own products in comparable ways, or if it is waived for favoured partners, it is unlikely to be strictly necessary. Fifth, and most important for the argument of this booklet, the measure should be specified publicly and in advance. A security requirement that a third party can read, implement and test against is a standard. A security requirement that the gatekeeper applies case by case, on reasons it does not disclose, is a gate. Apple's notarization criteria became more defensible as they were published in more detail. Meta's interoperability security requirements are defensible partly because they are written into a reference offer that any requester can read. The Commission's specification decisions for Apple and Google, which include security and privacy safeguards written by the regulator after consultation, take the logic one step further: the safeguards are no longer the gatekeeper's to define. Privacy law as a second referee The DMA does not operate alone on these questions. The General Data Protection Regulation governs any processing of personal data, including the data that the DMA requires gatekeepers to share or make portable. Article 6(11), on search data, requires anonymisation precisely because sharing raw query logs would breach data protection law. Article 7 limits the personal data the gatekeeper may collect for messaging interoperability to what is strictly necessary. Article 5(2) makes the combination of personal data across a gatekeeper's services conditional on user consent within the meaning of the GDPR. Article 5(2) produced the DMA's other first fine. In November 2023 Meta introduced a model in the EU under which users of Facebook and Instagram had to choose between consenting to the combination of their data for personalised advertising and paying a monthly subscription for an ad-free service. In April 2025 the Commission found that this binary "consent or pay" choice did not comply, because it did not give users a specific choice to opt for a service that used less of their personal data and was otherwise equivalent, and fined Meta €200 million. Meta had already introduced, in November 2024, a free option showing less personalised ads, which the Commission was assessing. In December 2025 the Commission announced that Meta had committed to offer EU users, from January 2026, an effective choice between consenting to share all their data and seeing fully personalised advertising, and opting to share less personal data for an experience with more limited personalised advertising. The Commission said it would gather evidence on the uptake and impact of the new model. The interplay between the two regimes is complicated enough that the Commission and the European Data Protection Board launched a public consultation on joint guidelines in October 2025, publishing contributions in March 2026. The guidelines matter because each regime can be used to resist the other. A gatekeeper asked to share data under the DMA can cite the GDPR as a reason to share less; a gatekeeper asked to obtain consent under the GDPR can design the consent so that it serves its commercial interest. Coordinated guidance narrows both openings. Who decides The underlying question in this chapter is not whether security matters. It is who decides how much security is enough, and on what evidence. Before the DMA, the answer on mobile platforms was the platform owner, with no external check beyond the market and occasional antitrust cases. The DMA does not replace the gatekeeper's judgement with the regulator's on every question. It requires the gatekeeper to justify its judgement, in terms that can be tested, whenever that judgement restricts the openness the Act requires. That arrangement has a cost. Gatekeepers have security teams of a size and expertise no regulator can match, and a regulator that overrides a well-founded security judgement could cause real harm. The Commission has responded by consulting widely, relying on published standards where they exist, and building safeguards into its specification decisions. It has also, notably, accepted many of the gatekeepers' security measures, including notarization, opt-in interoperability and pre-assessment of risks before search data is shared. The cases where it has pushed back are mostly cases where the measure did not address the threat claimed for it. The right conclusion is neither that security arguments are pretexts nor that they are trumps. It is that a security measure earns deference when it is specific, effective, minimal, consistent and public. Measures that meet that standard should survive the DMA. Measures that do not are the gate the Act was written to open. Chapter 8: Enforcement as Negotiation The image most people have of the Digital Markets Act comes from its fines: €500 million for Apple, €200 million for Meta, €890 million for Google. The fines are real, but they are the visible tip of a process that runs mostly out of sight, through information requests, draft proposals, workshops, preliminary findings and revisions. Enforcement of the DMA is best understood as a structured negotiation over specifications, with fines as one of several ways of moving it along. This chapter describes how that negotiation has actually run, what the first decisions show, how the Act is being extended to cloud computing and AI, and how it has fared in a political environment more hostile than its drafters expected. The first cycle The compliance deadline for the first six gatekeepers fell on 7 March 2024. Eighteen days later, on 25 March 2024, the Commission opened five non-compliance investigations: into Alphabet's rules on steering in Google Play, into Alphabet's self-preferencing in Google Search, into Apple's rules on steering in the App Store, into Apple's choice screen and default settings, and into Meta's "consent or pay" model. In June 2024 it opened a further investigation into Apple's terms for alternative distribution, including the Core Technology Fee. Table 5 sets out where each stood by early October 2026. Table 5. Principal DMA non-compliance proceedings and outcomes, March 2024 to October 2026. Gatekeeper Issue Opened Outcome Fine Apple App Store anti-steering March 2024 Non-compliance decision, 23 April 2025 €500 million Meta Consent or pay for data combination March 2024 Non-compliance decision, 23 April 2025 €200 million Apple Browser choice screen and defaults March 2024 Closed after changes, 23 April 2025 None Apple Terms for alternative distribution June 2024 Preliminary findings, April 2025 None to date Alphabet Self-preferencing in Google Search March 2024 Non-compliance decision, 23 July 2026 €460 million Alphabet Google Play anti-steering March 2024 Non-compliance decision, 23 July 2026 €430 million Alphabet Demotion of media publishers' content in Search November 2025 Investigation open None to date Source: European Commission DMA announcements. Two things stand out. The first is speed. The Apple and Meta decisions came about thirteen months after the investigations opened, close to the twelve-month target in the Act and far quicker than any comparable antitrust case. The Google decisions took longer, about twenty-eight months, which reflects both the difficulty of the search case and, many observers believe, the political caution discussed below. The second is the mix of outcomes. One investigation closed without a fine because Apple changed its conduct: it redesigned its browser choice screen, added a menu for changing default apps for calling, messaging, keyboards, password managers and translation, and allowed users to uninstall pre-installed apps including Safari. The Commission described this as the result of early and proactive engagement. Closure without sanction was the reward for that, and the signal to other gatekeepers was deliberate. How the negotiation runs Between the opening of an investigation and its outcome, the parties exchange a great deal. The Commission sends requests for information. The gatekeeper submits documents and proposals, often draft versions of new terms or interfaces, and asks for feedback. Rivals and business users submit complaints and evidence. The Commission holds compliance workshops at which the gatekeeper presents its measures and takes questions from interested parties in public. Preliminary findings set out the Commission's provisional view and give the gatekeeper a formal chance to respond. Throughout, the gatekeeper can adjust its conduct, and the Commission can signal what would satisfy it. This process has advantages. It allows the regulator to learn from the people who build and use the systems in question, it gives the gatekeeper a chance to propose solutions that fit its architecture, and it often produces change faster than a contested decision would. Apple's October 2026 terms, announced after what both sides described as close dialogue, are an example: whatever their merits, they arrived without a further decision and without periodic penalties. The process also has costs. It is opaque: the public sees the outcomes, not the bargaining, and third parties often learn of proposals only when they are announced. It favours the party with the most engineers and lawyers. It invites gatekeepers to make incremental concessions, each just enough to defer a decision, and to treat each concession as an opening position for the next round. Apple's three successive fee structures between 2024 and 2026 can be read as good-faith convergence or as a long delay in reaching terms the Commission would accept. Probably they were both. Circumvention by design: the Booking example The negotiation is not confined to the mobile platforms. Booking's experience shows how the same dynamic plays out where the obligation looks entirely self-executing. Article 5(3) prohibits a gatekeeper from preventing business users from offering the same products to end users at different prices or conditions through other channels. For an online travel agency, this means hotels must be free to offer lower prices on their own websites or on rival platforms than on Booking.com. Booking removed the parity clauses from its general delivery terms and partner agreements in July 2024, soon after its designation, and dropped price parity as an eligibility criterion for its Genius, Preferred and Preferred Plus programmes across the European Economic Area. That was not the end of the matter. A platform can discourage lower prices elsewhere without any contractual clause, simply by rewarding hotels whose prices elsewhere are no lower: ranking them higher, making them eligible for promotional programmes, or labelling their offers in the hotel's dashboard as "competitive". In September 2026, following further dialogue with the Commission, Booking confirmed that external prices were no longer a factor in eligibility for its Booking Sponsored Benefit programme for any property in the EEA, that external prices were not used in the default ranking of listings, and that it had removed "competitive" and "non-competitive" labels from the interface hotels use to manage their listings, so as not to signal what prices Booking expected. The episode is a small illustration of a large point. A contractual prohibition can be removed in a day. The same effect, produced by ranking signals, programme criteria and interface labels, is a matter of product design, and finding it requires the regulator to look inside the system. The Commission took more than two years after Booking's designation to complete that work, and it did so through dialogue rather than a decision. The anti-circumvention rule in Article 13 is what makes such dialogue possible: it extends the obligation from the contract to the code. The Google search decision and the problem of neutrality The search self-preferencing case shows the specification problem in its starkest form. Article 6(5) prohibits a gatekeeper from treating its own services more favourably in ranking than similar third-party services, and requires transparent, fair and non-discriminatory ranking conditions. The Commission's July 2026 decision found that Google displayed its own services for shopping, hotels, transport and sports with enhanced prominence at the top of results pages, with visual enhancements and filters, while similar third-party services did not get the same prominence. The difficulty is that there is no neutral search results page waiting to be uncovered. Every design privileges someone. From 2024 onwards Google tested various redesigns in Europe. Some removed specialized features, such as embedded maps or flight information. Others gave space to units listing comparison sites. Different groups of businesses lobbied for opposite results. Comparison and aggregator sites wanted Google's own specialized units replaced by units featuring them. Hotels, airlines and other direct suppliers, who sell to customers without paying an intermediary, worried that designs favouring aggregators would push them down the page, and some reported traffic losses during tests. Users, meanwhile, had grown used to seeing answers rather than links. The decision does not prescribe a page design. It orders Google to apply fair and non-discriminatory treatment to third-party services in its results, notes that Google has begun testing modifications to how it presents shopping, hotels, flights and sports content, and says the Commission will monitor implementation and continue assessing changes involving AI Overviews. In other words, the most difficult specification question in the DMA, what a fair results page looks like, has been left to the next round of the negotiation, with the fine as an incentive to settle it. The open investigation into Google's demotion of news publishers' content under its policy on site reputation abuse adds a further dimension: whether a search engine's quality rules can themselves be a form of unfair treatment. Appeals and the courts Every non-compliance decision can be challenged before the General Court and then on points of law before the Court of Justice. Apple has challenged both the €500 million fine and the March 2025 interoperability decisions, and Meta has contested its decision as well. Cases in the General Court commonly take two years or more, and further appeals add time. ByteDance's challenge to its designation, dismissed by the General Court in July 2024, was the first significant court ruling on the Act, and it upheld the Commission. The appeals matter for more than the money. They will settle questions the Act leaves open: how strictly "free of charge" must be read, how much deference the Commission owes gatekeepers' security justifications, and whether specification decisions can impose detailed engineering requirements. A court that read the obligations narrowly could unwind much of what the Commission has negotiated. A court that endorsed the Commission's approach would strengthen its hand in every future negotiation. Until then, both sides negotiate in the shadow of uncertain law. Extending the machine In July 2025 the Commission opened a consultation on the first review of the Act, and in August 2025 it gathered views on how the DMA could support contestable markets in the AI sector. It published the review on 28 April 2026, concluding that the DMA remained fit for purpose and had opened new opportunities for businesses and developers while giving users more control over their devices and data. It pointed to data portability between services and devices, alternative browsers and search engines being chosen as defaults, connected device makers gaining interoperability, alternative app stores launching and new messaging applications emerging. The review did not rewrite the Act. The more consequential extensions have come through the tools already in it. In November 2025 the Commission opened three market investigations into cloud computing: two to assess whether Amazon Web Services and Microsoft Azure should be designated as gatekeepers despite not meeting the quantitative thresholds for cloud services, and one to assess whether the Act's existing obligations are adequate to address practices in the cloud sector. On 25 June 2026 the Commission informed Amazon and Microsoft of its preliminary view that both services should be designated, describing them as the largest and second-largest cloud computing services in the EU and pointing to lock-in, high switching costs and the growing role of AI tools and partnerships in cloud procurement. A final decision was pending in early October 2026. AI has entered the DMA through the existing obligations rather than new ones. The Android interoperability measures of July 2026 are about AI assistants. The search data measures make AI chatbots eligible recipients. And the Commission has used ordinary antitrust law alongside the DMA, imposing interim measures on Meta in June 2026 to preserve rival AI chatbots' access to WhatsApp. The pattern is that AI services are being regulated as users of gatekeeper platforms, entitled to access, rather than as gatekeepers themselves. Whether a general-purpose AI assistant will one day itself be designated as a core platform service is a question the review left open. Politics and the wider world The DMA was drafted at a time of transatlantic consensus that big technology firms needed new rules. That consensus did not survive 2024. In February 2025 the President of the United States issued a memorandum directing his administration to examine foreign regulations, including European digital rules, that it regarded as unfairly targeting American companies. Senior officials publicly attacked the DMA as discriminatory, and in August 2025 the President threatened tariffs against countries whose digital rules burdened American technology firms. The Commission insisted throughout that its digital rules were not part of trade negotiations. Critics noted, however, that the Google decisions came later than expected and that the first fines were small relative to the companies' revenues, and asked whether pressure had played a part. The Commission denies it; outsiders cannot know. What is clear is that the European model has spread regardless. The United Kingdom's Digital Markets, Competition and Consumers Act 2024 created a regime under which the Competition and Markets Authority can designate firms with strategic market status and impose tailored conduct requirements, and in October 2025 the CMA designated Google in general search and Apple and Google in mobile platforms. Japan's Mobile Software Competition Act, which took full effect in December 2025, imposes obligations on smartphone operating systems and app stores closely resembling those in the DMA. The Commission signed a cooperation arrangement with the Japan Fair Trade Commission in July 2025 and hosted a workshop with partner countries on digital market regulation in June 2026. Other jurisdictions, including Brazil, have debated similar laws. The British model differs in a way that bears on this booklet's argument. Instead of fixed obligations applying to all gatekeepers, it lets the regulator design conduct requirements for each designated firm after investigation. That is slower to start but builds specification into the law from the beginning. The DMA began with uniform rules and has moved, through Article 8, towards tailored specifications. The two regimes are converging from opposite directions on the same insight: obligations must eventually be written at the level of the specific system they govern. What enforcement has shown Two and a half years of enforcement support a few firm conclusions. The DMA is faster than antitrust: decisions have come in one to two and a half years rather than seven to fourteen. The fines so far are modest and have functioned as signals rather than as punishments matched to the gains from non-compliance. Most change has come through negotiation in the shadow of sanctions, and the instruments that have produced the clearest change, the Apple interoperability specifications, the Android AI measures and the search data measures, are those in which the Commission wrote down the required result in detail. The obligations that remain most contested, steering fees and search neutrality, are those where it has not. The escalation ladder also matters more than its use so far suggests. Alphabet now has two non-compliance decisions, both adopted on the same day, and a third investigation open into its treatment of news publishers in search. A third decision within eight years would allow the Commission to open a market investigation into systematic non-compliance under Article 18, with structural remedies and a temporary ban on acquisitions among the possible outcomes. Nobody expects the Commission to order the break-up of Google over a search results page. But the possibility changes the arithmetic of every negotiation that follows, because each further finding of non-compliance now carries consequences beyond its own fine. The same logic will apply to Apple and Meta as their records lengthen. Periodic penalty payments perform a similar function at shorter range: a gatekeeper that has received a non-compliance decision and then fails to comply faces daily penalties, so the cost of delay rises with each day rather than arriving as a single sum years later. A regulator administering this system needs more than legal power. It needs engineers who can read an interface specification, economists who can judge a fee, and the political backing to hold a position against the world's largest companies and their government. The DMA supplied the power. Whether the Commission sustains the rest will determine whether the specifications keep coming. Conclusion The Digital Markets Act was sold as a way to replace slow, case-by-case antitrust with clear rules that the largest platforms would simply have to follow. Two and a half years of application show that the rules were clear about direction and silent about design, and that the silence was filled first by the gatekeepers and then, where the Commission chose to engage, by the regulator. The fights that mattered were not about whether Apple must allow alternative app stores or whether WhatsApp must talk to other messengers. They were about the screens a user must pass, the fee attached to a link, the protocol a rival must implement, the date by which a notification interface must ship and the anonymisation method applied to a query log. If the argument of this booklet is right, several things follow. The first concerns how the success of the Act should be judged. Headline indicators are misleading. The number of alternative app stores, the size of fines, and the count of companies that have requested interoperability measure activity, not openness. The questions that reveal whether a walled garden has really been opened are narrower and more technical. How much friction does a user face in exercising a right the Act grants, compared with the friction of using the gatekeeper's own option? What share of a steered sale does the gatekeeper still collect, and for how long after the link is tapped? How long after the gatekeeper's own product gains a capability does a rival's product gain the same one? How much does it cost, in engineering and compliance, for a newcomer to connect? These measures are harder to collect and less dramatic to report, but they are the ones that track whether the obligations have been translated into specifications that work. A regulator serious about outcomes should publish them, and the review cycle built into the Act is the natural place to do so. The second concerns the regulator's own role. The Commission began by treating specification as a fallback and has ended by relying on it for the most important openings it has achieved. That shift should be embraced rather than apologized for. A detailed, public, consulted specification is better for everyone than a general obligation followed by years of argument over its meaning. It gives the gatekeeper certainty, gives challengers something to build against, and gives courts something to review. But embracing specification has consequences. The Commission must recruit and retain technical expertise on a scale closer to a standards body than a competition authority. It must accept that when it writes a design, it shares responsibility for that design's failures, including security failures. And it must be disciplined about scope, specifying what is needed to make an obligation effective and no more, because a regulator that designs too much will slow the products it means to open. The third concerns the gatekeepers. The companies that have fared best under the Act are those that engaged early and designed solutions with the obligation in mind, as Apple did with its browser choice screen and default settings, closing an investigation without a fine. The companies that have fared worst have treated each obligation as a constraint to be minimized and each concession as an opening position. The rational strategy has shifted. Retrofitting openness feature by feature, under regulatory pressure, is expensive and generates delays and adverse publicity. Designing interfaces with third-party access in mind from the start, documenting them and publishing the security requirements they must meet, costs less over time and leaves less for the regulator to specify. Gatekeepers that continue to withhold features from Europe, rather than build them to be open, are making a bet that political pressure will weaken the Act. That bet looked better in 2025 than it does after the review of April 2026 and the Google decisions of July 2026. The fourth concerns the challengers the Act was meant to help. The DMA can lower barriers; it cannot create demand. The early experience shows that openness benefits most those who have a distinctive product and lack only access: a voice-first messenger that needs WhatsApp's users to reach anyone, a games company with a large audience and a grievance about commissions, a watch maker that needs iPhone notifications to sell watches. It benefits much less those who hoped that removing the gate would bring users flooding out of the garden. Users stayed with defaults when the defaults were good, and the gatekeepers' products remain good. Challengers who want to use the Act well should invest in what openness exposes them to, which is competition on the merits, and should also invest in the open standards, such as Messaging Layer Security and the IETF's work on messaging interoperability, that would one day free them from building against a gatekeeper's proprietary interface. The fifth concerns the security argument, which will only grow in importance as AI assistants gain the ability to see screens and act inside apps. The test proposed in Chapter 7, that a security measure deserves deference when it is specific, effective, minimal, consistent and public, should become the explicit standard for every carve-out the Act allows. Gatekeepers that meet it should keep their measures. Those that cannot should lose them. And the regulator should hold itself to the same standard when it writes safeguards into its own specifications. The sixth concerns the world beyond Europe. The DMA's specifications are already shaping global products, because features designed for European compliance, such as cross-platform data transfer, peer-to-peer Wi-Fi interfaces and browser data import, have been released worldwide. Britain and Japan have adopted regimes built on the same diagnosis, and Britain's design builds in tailored conduct requirements from the start, the destination the DMA reached by experience. The United States government has attacked the European approach as discriminatory, and its pressure will continue. But the engineering cannot easily be undone. Once an interface exists and developers depend on it, removing it costs the gatekeeper goodwill everywhere, not only in Europe. Finally, the experience of the DMA says something about regulation in technical domains generally. The old choice between leaving markets alone and ordering specific outcomes does not describe what happened here. What emerged instead is a form of continuous co-design, in which a public authority and a private company, under the eyes of rivals and users, negotiate the specifications of systems that billions of people use. Its legitimacy depends on transparency, because negotiations conducted in private over public obligations invite suspicion from all sides. Its effectiveness depends on competence, because a regulator that cannot read an interface specification cannot judge one. And its success depends on a recognition that the Microsoft case taught twenty years ago and the DMA has had to learn again: a walled garden is opened not by declaring that it must be open, but by writing down, precisely, where the gates go. Notes 1. Regulation (EU) 2022/1925 of the European Parliament and of the Council of 14 September 2022 on contestable and fair markets in the digital sector (Digital Markets Act), Official Journal of the European Union, L 265, 12 October 2022. Source for the thresholds, obligations, specification procedure, fines and review provisions discussed throughout, in particular Articles 3, 5, 6, 7, 8, 13, 18, 30, 31 and 53. 1. European Commission, Digital Markets Act website (digital-markets-act.ec.europa.eu), gatekeeper designations and news archive, 2023 to 2026. Source for Table 1, the non-designation decisions, and the dates of proceedings in Table 5. 1. European Commission, "Commission finds Apple and Meta in breach of the Digital Markets Act," press release IP/25/1085, 23 April 2025; and "Commission closes investigation into Apple's user choice obligations and issues preliminary findings on rules for alternative apps," IP/25/1086, 23 April 2025. 1. European Commission, "Commission fines Google €890 million for breaches of the Digital Markets Act," press release IP/26/1670, 23 July 2026. 1. European Commission, "Commission provides guidance under Digital Markets Act to facilitate development of innovative products on Apple's platforms," IP/25/816, 19 March 2025; and "Commission starts first proceedings to specify Apple's interoperability obligations," IP/24/4761, 19 September 2024. 1. European Commission, "Factsheet: How the DMA is making smartphones better: interoperability and data portability case studies," 11 May 2026. Source for Table 4. 1. European Commission, "Commission opens proceedings to assist Google in complying with interoperability and online search data sharing obligations," 27 January 2026; and "Commission provides guidance to Google for AI interoperability on Android and sharing of Google Search data," IP/26/1634, 16 July 2026. 1. European Commission, "Meta commits to give EU users choice on personalised ads under Digital Markets Act," 8 December 2025. 1. European Commission, "Review highlights Digital Markets Act remains fit for purpose and has positive impact," 28 April 2026; "Commission launches market investigations on cloud computing services," 18 November 2025; "Commission reaches preliminary position that Amazon's and Microsoft's market leading cloud services should be designated under the DMA," IP/26/1444, 25 June 2026. 1. Dick Brouwer, Meta Engineering, "Making messaging interoperability with third parties safe for users in Europe," engineering.fb.com, 6 March 2024; Meta, "An update on how we're building safe and secure third-party chats for users in Europe," September 2024; Meta, "Messaging interoperability: WhatsApp enables third-party chats for users in Europe," 14 November 2025. Sources for Chapter 3 and Table 2. 1. Apple Developer, "Updates for apps in the European Union," 26 June 2025; "Changes for apps in the European Union," 18 August 2026; and the support page "Apps in the EU," describing business terms effective 1 October 2026. Sources for Table 3 and Chapters 4 and 5. Percentages for the June 2025 terms as reported by 9to5Mac, "Apple announces sweeping App Store changes in the EU," 26 June 2025. 1. Apple, "The Digital Markets Act's impacts on EU users," Apple Newsroom, September 2025. Source for the features Apple delayed in the EU. 1. 9to5Mac, "Setapp Mobile 3rd-party app store for iPhone fails," 15 January 2026, reporting MacPaw's closure announcement. 1. Apple Developer, "Updates to App Tracking Transparency in the European Union," 16 September 2026. 1. R. Barnes, B. Beurdouche, R. Robert, J. Millican, E. Omara and K. Cohn-Gordon, "The Messaging Layer Security (MLS) Protocol," RFC 9420, Internet Engineering Task Force, July 2023. 1. Commission Decision of 27 June 2017, Case AT.39740 Google Search (Shopping); Court of Justice, Case C-48/22 P, Google and Alphabet v Commission, judgment of 10 September 2024. Commission Decision of 18 July 2018, Case AT.40099 Google Android; General Court, Case T-604/18, judgment of 14 September 2022. 1. Commission Decision of 24 March 2004, Case COMP/C-3/37.792 Microsoft; Court of First Instance, Case T-201/04, judgment of 17 September 2007. 1. Court of Justice, Case C-117/20, bpost, judgment of 22 March 2022. 1. Jacques Crémer, Yves-Alexandre de Montjoye and Heike Schweitzer, Competition Policy for the Digital Era (European Commission, 2019); Digital Competition Expert Panel, Unlocking Digital Competition (HM Treasury, March 2019). Further Reading The Digital Markets Act, Regulation (EU) 2022/1925 (EUR-Lex). The text itself is shorter and more readable than its reputation suggests. Articles 5 to 8 repay close reading, and the recitals explain the reasoning behind each obligation. European Commission, DMA website and gatekeeper compliance reports (digital-markets-act.ec.europa.eu). The primary record of designations, decisions, specification measures and consultations. The non-confidential compliance reports that each gatekeeper publishes every March are the best single source for how the companies themselves describe their implementation. Jacques Crémer, Yves-Alexandre de Montjoye and Heike Schweitzer, Competition Policy for the Digital Era (European Commission, 2019).** The report that framed the European diagnosis of digital platform markets, covering network effects, data and the limits of traditional enforcement. Digital Competition Expert Panel, Unlocking Digital Competition (HM Treasury, 2019).** The Furman report, which proposed the code-of-conduct model that became the United Kingdom's strategic market status regime. Useful as a contrast with the DMA's uniform obligations. Stigler Committee on Digital Platforms, Final Report (Stigler Center, University of Chicago, 2019).** An American counterpart to the European and British reports, with substantial chapters on market structure, data and the political power of platforms. Anu Bradford, The Brussels Effect: How the European Union Rules the World (Oxford University Press, 2020) and Digital Empires: The Global Battle to Regulate Technology (Oxford University Press, 2023).** The standard accounts of why European rules shape global products and of the contest between American, Chinese and European models of digital regulation. Cory Doctorow, The Internet Con: How to Seize the Means of Computation (Verso, 2023).** A polemical but well-informed case for interoperability as the central remedy for platform power, including the argument for adversarial interoperability that the Beeper episode illustrates. Dick Brouwer, "Making messaging interoperability with third parties safe for users in Europe" (Engineering at Meta, 2024). Meta's own account of the WhatsApp and Messenger interoperability architecture. Essential for understanding the design choices discussed in Chapter 3, whatever one thinks of them. R. Barnes and others, "The Messaging Layer Security (MLS) Protocol," RFC 9420 (IETF, 2023). The open standard for end-to-end encrypted group messaging that underpins the standards-based alternative to gatekeeper-defined interoperability. Technical, but its introduction is accessible. Hashtags: #TechPolicy #DigitalMarketsAct #DMA #EUdigitalMarketsAct #Antitrust #CompetitionLaw #DigitalCompetition #EuropeanUnionLaw #EURegulation #TechnologyRegulation #DigitalRegulation #PlatformRegulation #BigTechRegulation #DigitalEconomy #DigitalMarkets #CompetitionPolicy #AntitrustEnforcement #EuropeanCommission #Gatekeepers #DigitalGatekeepers #CorePlatformServices #MarketContestability #FairDigitalMarkets #ExAnteRegulation #ExPostEnforcement #MarketDominance #AbuseOfDominance #PlatformPower #NetworkEffects #PlatformEconomics #DigitalMonopolies #SelfPreferencing #AntiSteering #AppStoreRegulation #AppStoreCompetition #AlternativeAppStores #AlternativeAppMarketplaces #AppDistribution #Sideloading #DeveloperRights #AppDeveloperEconomy #PlatformFees #AppStoreCommissions #AlternativeBilling #DigitalPayments #BrowserChoice #SearchEngineChoice #SearchEngineCompetition #SearchRanking #OperatingSystemInteroperability #MessagingInteroperability #WhatsAppInteroperability #EndToEndEncryption #SignalProtocol #MessagingLayerSecurity #OpenStandards #OpenAPIs #PlatformInteroperability #SoftwareEcosystems #WalledGardens #DataPortability #DataAccess #ConsumerChoice #ConsumerProtection #PrivacyAndCompetition #GDPR #DigitalServicesAct #EUAIAct #Cybersecurity #PlatformSecurity #SecurityAndPrivacy #FairAccess #FRAND #RegulatoryCompliance #DMACompliance #ComplianceMonitoring #RegulatoryEnforcement #AntitrustFines #RegulatoryRemedies #StructuralRemedies #BehavioralRemedies #Article5DMA #Article6DMA #Article7DMA #Article8DMA #Article13DMA #DigitalMarketGovernance #TechnologyGovernance #RegulatoryPolicy #DigitalSingleMarket #DigitalInnovation #PlatformAccountability #CompetitionAndInnovation
- Silicon Photonics Co-Packaging (Removing the Data Center Bottleneck)
Download the Book (PDF): Introduction Walk into a modern AI training hall and the first thing you notice is not the processors. You cannot see them. They sit under cold plates and manifolds, buried inside racks that weigh more than a small car. What you can see is cable. Bundles of it run down the backs of the racks, up into overhead trays, across the room and into rows of switches. Some of it is copper, thick and stiff, bent into careful radii. Much of it is glass fibre, ending in pluggable transceivers, each one a small metal brick with a laser, a receiver, a signal-processing chip and a heat sink of its own. A large cluster contains hundreds of thousands of those bricks. Every one of them burns power around the clock, whether or not any useful data is moving. That cabling is not decoration. It is the nervous system of the machine, and in large-scale AI it has become the part of the machine that most often decides how fast the whole thing runs. A single modern accelerator can perform trillions of operations every second, but a frontier model is trained on tens or hundreds of thousands of them working in concert. They must exchange gradients, activations and parameters constantly. When the network cannot keep up, expensive silicon sits idle, waiting for bytes. The industry's own shorthand for this is blunt: compute has scaled faster than communication, and the gap is now where the money leaks out. This book is about one specific answer to that problem, the one that has moved in the past few years from laboratory curiosity to shipping product. The answer is to stop treating optics as an accessory plugged into the front of a box, and instead to build the optical transmitter and receiver into the same package as the switch chip or the processor itself. The industry calls this co-packaged optics. The underlying technology that makes it possible is silicon photonics: the craft of building waveguides, modulators and photodetectors in the same kind of factories, and often on the same kind of wafers, that make ordinary chips. The argument of this book The case for co-packaged optics is usually presented as a story about speed. It is better understood as a story about distance. Every bit that leaves a chip must first travel as an electrical signal across some length of copper before it can be turned into light. The longer that electrical path, the more energy is spent fighting loss, the more signal-processing circuitry is needed to clean up the damage, and the more heat must be removed. A conventional pluggable transceiver sits at the front panel of a switch, perhaps fifteen or twenty centimetres from the chip it serves. At the data rates now in use, that short trip is brutally expensive. Co-packaged optics shrinks it to a few millimetres. The controlling idea of this book follows from that observation. Co-packaged optics wins by shortening the electrical path between logic and light, and almost everything interesting about it, its gains, its engineering difficulties, its business consequences, flows from what happens when you move a fragile optical system into the hottest, most crowded, least serviceable place in the data centre. The power savings are real and large. So are the costs: lasers that hate heat, fibres that must be aligned to fractions of a micrometre, test procedures that did not exist a decade ago, and a change in who is responsible when a link fails. Understanding the technology means holding both sides of that ledger at once. That framing also explains the order in which the technology is arriving. It reached network switches first, because a switch is a box whose entire job is input and output, and where the front-panel bottleneck is most acute. Nvidia's Quantum-X Photonics InfiniBand switches and Broadcom's co-packaged Ethernet switches are the first wave. The second wave, now in development and early sampling, puts optical input and output directly beside accelerators and memory, where the prize is not only lower power but an escape from the few-metre reach of copper that currently caps how many processors can be wired together as a single tightly coupled machine. Why now Engineers have predicted the arrival of optics at the chip for a long time. In 2006 Pat Gelsinger, then a senior executive at Intel, said that optics was a niche technology today and would be the mainstream of every chip tomorrow. Tomorrow took nearly twenty years. The delay is instructive, because it reveals the conditions under which a radical change in packaging becomes worth its risk. Three things had to happen together. Electrical signalling had to become painful enough that its incremental improvements no longer kept pace with demand; that happened as lane rates passed 100 and then 200 gigabits per second. Silicon photonics had to mature into a manufacturing discipline with foundry processes, wafer-scale test and reliable packaging, which happened gradually through the 2010s and early 2020s as pluggable transceivers built in silicon shipped by the million. And a customer had to appear whose demand for bandwidth was so large, and whose power budget so constrained, that the savings of co-packaging outweighed the comfort of familiar pluggable modules. Generative AI supplied that customer. Training clusters grew from thousands of accelerators to hundreds of thousands within a few years, and the networks that connect them grew with them. When those three conditions met, the industry moved quickly. Within roughly two years, the largest supplier of AI accelerators announced co-packaged switches, the largest merchant switch-chip supplier shipped its third generation of them, the largest semiconductor foundry put a photonic engine on its roadmap, and a chip company agreed to pay billions of dollars for a photonic interconnect startup. A technology that had lived for two decades on conference slides became a line item in data centre budgets. What the reader will find here The chapters build in a deliberate sequence. Chapter 1 explains why copper, which has carried computing's signals for seventy years, is running out of room at the data rates AI demands, and why the problem is one of physics rather than engineering effort. Chapter 2 opens the toolbox of silicon photonics: how light is guided, switched, modulated and detected in silicon, and why the laser remains the awkward guest. Chapter 3 does the accounting, tracing where the energy goes in an optical link, bit by bit, and showing why the electrical portion dominates the bill. Chapter 4 surveys the spectrum of integration, from front-panel pluggables to optics stacked directly on processors, and explains what each step buys and costs. Chapter 5 confronts the hard engineering: lasers, fibre attachment, temperature control and testing, the problems that kept co-packaged optics in the laboratory for so long. Chapter 6 examines the first commercial deployments in switches, what they promise and what the early reliability evidence says. Chapter 7 turns to the frontier, optical input and output for accelerators and memory, and the startups and incumbents competing to supply it. Chapter 8 looks at economics, supply chains, standards and serviceability, the factors that will decide how quickly and how widely the technology spreads. The conclusion argues what all of this implies for those who build, buy and plan data centres. A note on numbers. This is a field where marketing figures travel faster than measured ones. Where a figure comes from a vendor, the text says so. Energy per bit is the common currency throughout, measured in picojoules (a picojoule is a trillionth of a joule). To give a sense of scale: a link that spends 15 picojoules on every bit, running at 800 gigabits per second, consumes 12 watts. Multiply by the hundreds of thousands of such links in a large cluster and you reach megawatts, enough to power a small town, spent purely on moving data from one box to another. The reader needs no background in optics or electrical engineering. Mechanisms are explained in words, step by step. What the reader does need is patience with one idea that runs through everything: in large computers, moving a bit has become harder, and costlier, than computing with it. Co-packaged optics is the most serious attempt yet to change that balance. Chapter 1: The Wall of Wires For most of the history of computing, the wire was the least interesting part of the machine. Engineers lavished attention on transistors, caches and instruction sets, while the copper traces and cables that connected them were treated as plumbing. That attitude made sense when signals were slow. A wire carrying a few hundred megabits per second behaves, for practical purposes, like an ideal conductor over the distances found inside a computer. Push the same wire to a hundred gigabits per second and it becomes a hostile environment, one that smears, attenuates and distorts every pulse sent down it. This chapter explains why, and why the problem now governs the design of AI systems. What happens to a fast signal in copper A digital signal on a wire is a sequence of voltage transitions. The faster the data rate, the sharper and more frequent those transitions, and the higher the frequencies they contain. A modern high-speed electrical lane, the kind that connects a switch chip to its ports or a GPU to its neighbours, runs at roughly 200 gigabits per second. It does so using a scheme called PAM4, which encodes two bits in each symbol by using four voltage levels rather than two. That halves the symbol rate needed for a given bit rate, but the symbol rate is still about 112 billion symbols per second, and the signal's essential energy extends to around 56 gigahertz, a frequency engineers call the Nyquist frequency of the link. Copper at 56 gigahertz misbehaves in two main ways. The first is the skin effect. Alternating current at high frequency crowds toward the outer surface of a conductor, so the effective cross-section carrying the current shrinks and resistance rises, roughly with the square root of frequency. The second is dielectric loss. The insulating material around the conductor, whether the fibreglass-and-resin laminate of a circuit board or the plastic of a cable, absorbs a small amount of energy on every cycle of the field. That loss rises roughly in proportion to frequency. At low data rates both effects are negligible. At 56 gigahertz they dominate. The result is attenuation that climbs steeply with both frequency and distance. Engineers measure it in decibels, a logarithmic unit where every 10 decibels represents a factor of ten in power. A typical long-reach electrical lane at today's speeds is designed to tolerate something on the order of 30 to 40 decibels of loss at the Nyquist frequency. That means the receiver must recover data from a signal that has lost all but a thousandth or a ten-thousandth of its original power at the frequencies that matter most. And the loss is not uniform: low-frequency components arrive relatively intact while high-frequency components are crushed. Each pulse spreads out in time and bleeds into its neighbours, a phenomenon called inter-symbol interference. To recover the data, chips use a circuit called a SerDes, short for serialiser/deserialiser. On the transmit side it takes parallel data from the chip's internal logic, serialises it into a fast stream and pre-distorts the waveform to compensate for what the channel will do to it. On the receive side it equalises the signal, amplifying the high frequencies the channel attenuated, and uses increasingly sophisticated digital signal processing to decide which of the four levels each symbol represents. Modern long-reach receivers digitise the incoming waveform with high-speed analogue-to-digital converters and run it through digital filters, much as a cellular modem does. That works, but it is expensive in silicon area and, above all, in power. Every new generation of electrical signalling has doubled the data rate per lane: 25, 50, 100, now 200 gigabits per second, with 400 under development. Each doubling roughly doubles the Nyquist frequency, and with it the loss per unit length. To keep a link working, engineers have three options. They can spend more power on the SerDes, use better and costlier materials for boards and cables, or shorten the distance. In practice they have done all three, and the room left on the first two is shrinking. Low-loss board laminates are already exotic and expensive. Cable makers have moved to thicker conductors and better dielectrics, at the price of stiff, heavy cables that are hard to route. That leaves distance. The reach of copper is collapsing One way to see the problem is to track how far a passive copper cable can carry a signal at each generation. A passive direct-attach copper cable is just wire and connectors, with no active electronics in the cable itself. At 25 gigabits per lane such cables comfortably spanned several metres, enough to connect servers across a rack. At 100 gigabits per lane the practical reach fell to around two or three metres. At 200 gigabits per lane it is a metre or two, depending on cable gauge and the quality of the SerDes on each end. The next doubling will cut it again. This matters because the distance a fast copper link can span defines the size of the most tightly coupled unit of computing that can be built without optics. Nvidia's GB200 NVL72 system illustrates the point. It connects 72 Blackwell GPUs in a single rack into one NVLink domain, in which every GPU can talk to every other at 1.8 terabytes per second. Nvidia chose copper for that domain, running a spine of more than five thousand cables down the back of the rack, which the company described as roughly two miles of copper. The reason was power and cost: copper needs no lasers, no optical modules, no conversion electronics. The constraint was reach. Copper at those speeds can only span the inside of a rack, so the rack became the boundary of the domain. To build that domain, Nvidia had to cram an extraordinary amount of compute into one cabinet, producing racks that draw well over a hundred kilowatts and require liquid cooling throughout. In other words, the physics of copper is now dictating the physical architecture of AI computers. Systems designers would like larger scale-up domains, because training and serving very large models benefits from many accelerators sharing memory and communicating with low latency. But each extra metre of copper is hard to buy. You can pack the rack more densely, which pushes the limits of power delivery and cooling. You can add active electronics inside the cable to re-amplify the signal, as active electrical cables do, which adds power and cost. Or you can switch to optics, which can carry the same signal hundreds of metres with almost no loss, if you can afford the power and cost of converting electricity to light and back. The front-panel problem The same squeeze appears in the network switches that connect racks together, but in a different form. A switch chip is a device whose entire purpose is moving data in and out. Broadcom's Tomahawk family illustrates the pace of growth. Successive generations have roughly doubled aggregate bandwidth every two years or so: 12.8 terabits per second around 2019, 25.6 in 2020, 51.2 in 2022, and 102.4 terabits per second with Tomahawk 6, announced in 2025. Nvidia's Spectrum-X Ethernet and Quantum InfiniBand switch chips have followed a similar curve. All that bandwidth has to escape the chip. It leaves through the SerDes lanes at the edge of the die, passes through the package to the circuit board, travels across the board to the front panel of the switch, and enters a pluggable optical module, where it is finally converted to light. A 102.4 terabit-per-second chip has 512 lanes of 200 gigabits each. Each of those lanes must make that trip across the board, through a connector and into a module. Three things go wrong as the bandwidth climbs. The first is the electrical loss of the board trace itself, which, as we have seen, worsens with every doubling of the lane rate. Fifteen or twenty centimetres of board trace at 56 gigahertz is a substantial impairment, and switch designers have resorted to ever more expensive laminates, to flyover cables that lift signals off the board onto small twinaxial cables, and to retimer chips that regenerate the signal partway. The second is the front panel itself. A standard one-unit switch faceplate has room for about 32 of the common OSFP or QSFP-DD module cages. At 800 gigabits per module, 32 cages carry 25.6 terabits per second. A 51.2-terabit chip therefore needs 64 modules, which means a two-unit chassis or a denser cage. A 102.4-terabit chip needs either 64 modules at 1.6 terabits each or 128 at 800 gigabits. Faceplate area does not grow with Moore's law. The third problem is heat. Every pluggable module contains a digital signal processor that recovers the electrical signal coming from the switch chip and regenerates it, plus laser drivers, lasers and receivers. A 1.6-terabit module can dissipate roughly 25 to 30 watts. Sixty-four of them dissipate more than one and a half kilowatts, often more than the switch chip they serve. It helps to put a number on what each centimetre of copper costs. Engineers designing a switch board at 200 gigabits per lane work with a loss budget, the total attenuation the SerDes can tolerate between transmitter and receiver at the Nyquist frequency. Suppose, for illustration, that the budget is about 35 decibels. Part of it is consumed before the signal even reaches the board: the package traces under the chip, the solder balls, the vias that carry the signal through the layers of the board. Part of it is reserved for the connector at the front panel and the traces inside the module. What remains for the board trace itself may be well under half the total. Even high-grade, low-loss laminates lose on the order of a decibel per inch, roughly 0.4 decibels per centimetre, at frequencies around 50 gigahertz, and ordinary laminates lose several times more. Twenty centimetres of good board can therefore consume a third or more of the whole budget on its own. Every centimetre saved is margin returned to the link, and every centimetre added must be paid for with a stronger SerDes, a retimer or a better material. Designers describe the problem in exactly these terms: they do not ask whether a trace works but how many decibels it eats. This is why so much engineering effort in the past decade has gone into getting signals off the board. Flyover cables, small twinaxial cables attached at the package and routed directly to the front panel, have lower loss per centimetre than board traces and have become common in high-end switches. Retimers, chips that sit partway along the path and regenerate the signal, split one long, lossy channel into two shorter ones, at the cost of a few more watts per port and another component to fail. Each of these techniques works. Each adds cost and power. And each is a symptom of the same underlying fact: the electrical path from chip to optics is now one of the most expensive parts of the system to build. Between passive copper and optics lies a middle ground. Active electrical cables embed small signal-conditioning chips, either simple redrivers or full retimers with their own DSP, in the cable ends. Companies such as Credo have built substantial businesses supplying them to hyperscalers, and they extend copper's reach at 200 gigabits per lane to several metres, enough to span a rack and its neighbour. For many links within and between adjacent racks, active electrical cables are cheaper than optics, consume less power than optical modules, and avoid lasers altogether. But active copper does not change the underlying trend. Each generation of active cable needs more capable, more power-hungry chips in its ends to compensate for higher loss, and its reach still shrinks as lane rates rise. Copper cables also take up space. A rack carrying thousands of thick cables has to accommodate their weight, their bend radii and the airflow or coolant paths around them. At some point the mechanics alone become a constraint: there is simply no room left to route another bundle. Optical fibre is a fraction of the diameter of a high-speed copper cable, and a single fibre ribbon can carry the traffic of a large copper bundle. Active copper is best understood as a way of extending copper's useful life by a generation or two, not as an alternative to the transition. Why the escape path itself is a bottleneck There is a deeper geometric constraint underneath these practical ones. A chip's computing capacity scales with its area. Its input-output capacity, at least for signals that leave through the package edge, scales with its perimeter. Engineers call the usable edge length the beachfront or shoreline of a chip. As chips grow, area outpaces perimeter, and the bandwidth needed per millimetre of edge rises. The same applies one level up, to the package. A large accelerator package today may carry several compute dies and stacks of high-bandwidth memory, and the signals that leave the package must pass through solder balls on its underside and then through the board. Each high-speed lane needs a differential pair of connections plus surrounding ground connections to shield it, and the number of balls that can be placed under a package is finite. High-bandwidth memory solved this problem for memory by placing memory stacks right beside the processor on a silicon interposer, with thousands of very short, slow, parallel connections instead of a few long, fast ones. That approach trades reach for density. It works superbly over a few millimetres and does not work at all beyond the package. For connections that must leave the package, whether to a neighbouring GPU, a switch or a remote memory pool, the only options are fast serial electrical links, with all their loss, or optics. Taken together, these constraints describe a wall. On one side is the relentless growth of compute, roughly doubling per accelerator every couple of years and multiplied by the growth in cluster size. On the other is the physics of copper, which demands more power and delivers less reach with every doubling of the lane rate, and the geometry of packages and front panels, which limits how many lanes can escape at all. Engineers have been climbing that wall for a decade with better SerDes, better boards and better cables. Each step still works, but each step costs more than the last. Light in glass fibre does not suffer the problems that afflict copper. Standard single-mode fibre loses only a fraction of a decibel per kilometre at the infrared wavelengths used in data centres, essentially nothing over the tens or hundreds of metres within a building, and its loss does not depend on the data rate. A fibre can carry many independent signals at different wavelengths simultaneously. It is light, thin and immune to electromagnetic interference. For any link longer than a couple of metres, optics has long been the only practical choice, which is why data centre networks above the rack level have been optical for years. The problem is the conversion. To use optics, an electrical signal must be turned into light by a modulated laser and turned back into electricity by a photodetector and amplifier. In today's architecture that conversion happens in the pluggable module at the front panel, which means the signal first travels the full lossy electrical path from chip to panel. The optical part of the link is efficient. The electrical part that feeds it is not, and the module must include a power-hungry digital signal processor precisely to repair the damage done by that path. Put that way, the solution almost suggests itself. If the expensive part of an optical link is the electrical stretch in front of it, move the optics to the chip. Make the electrical path a few millimetres long instead of twenty centimetres. Let the SerDes run in a gentle, low-power mode because it only needs to drive a signal across a package substrate. Eliminate the digital signal processor in the module, because the signal arriving at the optical engine has barely been degraded. Free the front panel, because what leaves the box is now only fibre, and fibre is thin. That idea has been discussed for more than twenty years. It was not adopted earlier because putting optics on the package was much harder than putting it in a module, and because copper kept getting better just fast enough to postpone the decision. The rest of this book explains what has changed. The first part of the answer lies in the technology that lets optical components be manufactured like chips, at chip-like scale and cost. That technology is silicon photonics, the subject of the next chapter. Chapter 2: Light on Silicon Optical communication is old. Fibre-optic links have carried telephone traffic since the late 1970s, and the transceivers that power data centre networks today are descendants of telecom equipment. For most of that history, optical components were built one at a time from exotic materials: lasers from indium phosphide or gallium arsenide, modulators from lithium niobate, detectors from indium gallium arsenide, all assembled with lenses and alignment fixtures by skilled technicians or specialised machines. That approach produced superb devices and terrible economics. Each transceiver was a miniature hand-built instrument. Silicon photonics changed the economics by borrowing the most successful manufacturing system ever devised, the semiconductor fab. Its premise is that many of the functions an optical transceiver needs, guiding light, splitting and combining it, modulating it and detecting it, can be built with the same lithography, etching and deposition tools used for transistors, on silicon wafers, in volumes of millions. This chapter explains how each of those functions works, and why one function, generating the light in the first place, remains stubbornly difficult. Guiding light in a sliver of silicon Silicon looks opaque to the eye, but that is because visible light carries enough energy per photon to be absorbed by it. At infrared wavelengths longer than about 1.1 micrometres, the photons carry too little energy to excite electrons across silicon's bandgap, and silicon becomes almost perfectly transparent. Data centre optics works in two infrared bands: the O-band around 1,310 nanometres and the C-band around 1,550 nanometres. Both sit comfortably in silicon's transparent window. The basic structure is a waveguide, a strip of silicon that confines light the way a fibre does. Silicon has a refractive index of about 3.5, while the silicon dioxide surrounding it has an index of about 1.45. That large contrast means light is held very tightly inside the silicon by total internal reflection. A typical silicon waveguide is around half a micrometre wide and a fifth of a micrometre thick, hundreds of times smaller in cross-section than the core of a single-mode fibre. Because the confinement is so strong, the waveguide can bend around a radius of a few micrometres without leaking much light. This is what allows dense optical circuits: hundreds of components on a chip a few millimetres across. The starting material is usually a silicon-on-insulator wafer: a thin layer of crystalline silicon on top of a layer of buried oxide on top of a thick silicon substrate. The thin top layer becomes the waveguide; the buried oxide below it keeps the light from leaking into the substrate. Silicon-on-insulator wafers were already used in some electronics, which helped silicon photonics get started in existing fabs. From waveguides, designers build passive components. A directional coupler brings two waveguides close enough that light tunnels between them. A multimode interference coupler splits or combines beams. A grating etched into the waveguide surface can scatter light up out of the chip at a defined angle, which is one way to couple light to a fibre held above it. An edge coupler tapers the waveguide down at the chip's edge so that the light spreads into a larger spot that better matches a fibre. Each of these is a lithographic pattern, reproducible across a wafer. Putting data onto light A laser produces a steady beam. To carry information, the beam must be modulated, switched or varied in step with the data. In silicon this is harder than it sounds, because pure silicon lacks the strong electro-optic effects that make lithium niobate and indium phosphide good modulator materials. The breakthrough insight came from Richard Soref and Brian Bennett, who showed in 1987 that changing the concentration of free electrons and holes in silicon changes its refractive index and absorption by a small but usable amount. This is the plasma dispersion effect, and nearly every high-speed silicon modulator relies on it. A practical modulator embeds a diode in the waveguide. Applying a voltage pushes charge carriers into or out of the region where the light travels, which shifts the refractive index and therefore the phase of the light passing through. A phase shift on its own is invisible to a detector, so the phase shifter is placed in an interferometer. The most common design is the Mach-Zehnder modulator. Light is split into two arms, one arm's phase is shifted, and the beams are recombined. When the two arms are in phase they add, giving a bright output; when they are half a cycle apart they cancel, giving a dark one. Switching the voltage switches the light. Intel's research group, led by Mario Paniccia, demonstrated a silicon modulator operating above one gigahertz in 2004, a result published in Nature that convinced many sceptics that silicon could be fast. Mach-Zehnder modulators have since reached well beyond 100 gigabits per second. Their weakness is size. Because the plasma dispersion effect is weak, the phase-shifting arms must be long, typically millimetres, and driving a millimetres-long electrode at tens of gigahertz demands a travelling-wave design and a driver that swings several volts. Mach-Zehnder modulators are robust and tolerant of temperature, but they are large and power-hungry by the standards of what co-packaging wants. The alternative is the micro-ring modulator. A ring-shaped waveguide a few micrometres to tens of micrometres in diameter sits beside a straight bus waveguide. Light of a particular wavelength, one for which a whole number of wavelengths fits around the ring's circumference, couples into the ring and circulates many times before leaking back out. At that resonant wavelength, light is strongly drawn into the ring and the transmission of the bus waveguide dips sharply. Put a diode in the ring, shift its refractive index slightly, and the resonance moves. A laser tuned to the edge of the resonance sees its transmission swing from high to low as the voltage changes. Because the light circulates many times, a tiny index change has a large effect, so the ring can be small and needs only a small drive voltage. Michal Lipson's group at Cornell demonstrated a compact silicon micro-ring modulator in 2005. Two decades later, ring modulators are central to co-packaged optics. Nvidia's photonics switches use micro-ring modulators driven at 200 gigabits per second per wavelength with PAM4 signalling, manufactured with TSMC. Ayar Labs, Intel's optical compute interconnect chiplet and several other designs rely on rings as well. The reason is density and energy: a ring occupies a tiny fraction of the area of a Mach-Zehnder and switches with a fraction of the charge. Rings have a price, and it is temperature. Silicon's refractive index changes with temperature by roughly 0.02 per cent per degree. For a ring resonator, that moves the resonance wavelength by something on the order of a tenth of a nanometre per degree Celsius, and a ring's resonance may be only a fraction of a nanometre wide. A few degrees of warming can push the ring completely off its laser's wavelength, and the link stops working. Manufacturing variation adds a further offset: two rings drawn identically will resonate at slightly different wavelengths because of nanometre-scale differences in waveguide width and thickness. Every practical ring-based system therefore includes heaters, small resistive elements beside each ring, and control loops that continuously measure and correct each ring's position. This tuning consumes power and complexity, and it becomes harder still when the ring sits on a package beside a processor whose temperature swings by tens of degrees as its workload changes. Chapter 5 returns to this problem. Turning light back into electricity At the receiving end, light must be converted back to current. Silicon cannot do this at data centre wavelengths for the same reason it is transparent: the photons lack the energy to be absorbed. The solution is germanium. Germanium absorbs strongly at 1,310 nanometres and reasonably at 1,550, and it can be grown epitaxially on silicon, crystal layer upon crystal layer, in standard fab equipment. A germanium photodiode at the end of a silicon waveguide absorbs the light and produces a current, which a transimpedance amplifier converts to a voltage the receiving circuit can read. Germanium-on-silicon detectors now operate comfortably beyond 100 gigabits per second and are a mature, standard part of every silicon photonics process. With waveguides, couplers, modulators and detectors, the silicon photonics toolkit can build nearly everything a transceiver needs, and can build many copies of it side by side. One more technique multiplies the capacity: wavelength-division multiplexing. Because each ring modulator responds only to light near its own resonance, a row of rings tuned to slightly different resonances can sit along a single bus waveguide, each one modulating a different laser wavelength while letting the others pass. At the receiver, another row of rings acts as filters, each one dropping its own wavelength to its own detector. Eight or sixteen independent channels can share one waveguide and one fibre. Intel's optical I/O chiplet, for example, places eight wavelengths on each fibre, and Ayar Labs' TeraPHY uses sixteen wavelengths per port. It is worth following a single channel through a complete ring-based link to see how these pieces fit together. Continuous light from a laser, at one precise wavelength, enters the transmitting chip through a fibre coupler. On the chip it may first pass through a splitter that divides it among several lanes, since a single laser often supplies more than one channel. In each lane, the light travels along a bus waveguide past a row of ring modulators. All but one of those rings are tuned to other wavelengths and ignore it. The one ring tuned near this wavelength is driven by the electrical data signal, and as the voltage swings between its four PAM4 levels, the ring's resonance shifts slightly, and the light continuing along the bus is dimmed by four different amounts. The bus waveguide, now carrying several wavelengths each modulated with its own data, leads to an output coupler and into a fibre. At the far end, the fibre's light enters the receiving chip. It passes along another bus waveguide lined with filter rings, each tuned to drop one wavelength into its own short waveguide that ends at a germanium photodiode. The photodiode turns the varying light into a varying current. A transimpedance amplifier converts that current to a voltage, and a decision circuit determines which of the four levels each symbol represents, recovering two bits per symbol. Meanwhile, monitoring photodiodes and control loops at both ends keep each ring aligned to its wavelength, nudging tiny heaters up and down thousands of times a second. The whole path, from electrical input to electrical output, may span tens of metres of fibre but involves only a few millimetres of electrical wiring at each end. The choice of wavelength band also has practical consequences. Data centre links mostly use the O-band, around 1,310 nanometres, rather than the C-band favoured by long-haul telecommunications. Standard single-mode fibre has almost zero chromatic dispersion near 1,310 nanometres, which means different wavelength components of a pulse travel at nearly the same speed and the pulse does not spread out over the short distances involved. That allows simple, low-power receivers with no need for the dispersion compensation that long-haul links require. The C-band has lower fibre loss and a mature ecosystem of amplifiers, which matters over hundreds of kilometres but hardly at all within a building. For co-packaged optics, where every milliwatt of receiver power counts, the O-band's simplicity usually wins. The laser: silicon's missing piece Silicon can guide, modulate and detect light. It cannot make it. The reason is fundamental. Silicon has an indirect bandgap, which means an electron dropping from the conduction band to the valence band must also change its momentum, and it usually does so by shedding the energy as heat in the crystal lattice rather than as a photon. Silicon light emission is extraordinarily inefficient, and no practical silicon laser exists for communications. Light sources for silicon photonics therefore come from compound semiconductors, usually indium phosphide, whose direct bandgap makes them efficient emitters at the right wavelengths. There are three broad ways to bring that light to the silicon chip. The first is to keep the laser entirely separate, in its own package, and deliver its light to the silicon chip through fibre. The second is to attach finished laser chips to the silicon by flip-chip bonding, aligning their output to silicon waveguides. The third is heterogeneous integration: bonding unprocessed indium phosphide material onto the silicon wafer and then fabricating the laser in place, so that its light forms directly inside a silicon waveguide. Intel and John Bowers's group at the University of California, Santa Barbara, demonstrated this hybrid silicon laser in 2006, and Intel has used heterogeneously integrated lasers in its silicon photonics transceivers. A fourth approach, growing quantum-dot laser material directly on silicon, is the subject of active research and early commercialisation. Which approach to use is one of the defining choices in co-packaged optics, because lasers are the component most sensitive to heat and most likely to wear out. A laser bonded to a chip next to a 1,000-watt processor operates in a hot, fluctuating environment, and its output power and lifetime both suffer. A laser outside the package, in a cool, replaceable module, avoids that problem but requires an extra optical connection and loses some light in the fibre between. Most first-generation co-packaged systems, including Nvidia's and Broadcom's switches and Ayar Labs' chiplets, use external laser sources for precisely this reason. Intel's optical compute interconnect chiplet is the most prominent exception, integrating its lasers on the chip, and Intel cites a long track record of integrated lasers in its shipped transceivers as evidence that the approach can be reliable. Silicon is not the only material contending for the modulator. As per-wavelength rates climb from 200 toward 400 gigabits per second, the plasma dispersion effect becomes a limitation: moving charge carriers in and out of a waveguide takes time, and the electrical capacitance of silicon modulators makes them hard to drive at ever higher frequencies without spending more energy. Thin-film lithium niobate, a modern revival of the classic telecom modulator material in a form thin enough to integrate with silicon chips, offers a strong, nearly instantaneous electro-optic effect and very high bandwidth at low drive voltage. Other groups are exploring barium titanate, electro-optic polymers and indium phosphide modulators bonded onto silicon. None of these has displaced silicon in co-packaged products, which value the maturity and manufacturability of an all-silicon process, but they illustrate that silicon photonics is better understood as a platform onto which the best material for each job can be added than as a single material solution. The laser, discussed next, is the clearest case of that principle. Silicon photonics took a long time to become an industry. Luxtera, founded in 2001 as a spin-out from Caltech, was the first company to ship silicon photonic transceivers in volume, initially in active optical cables. Cisco bought Luxtera in 2019 for $660 million. Intel began shipping silicon photonics transceivers for data centres in 2016 and shipped millions of them before selling the pluggable transceiver business to Jabil in 2023, retaining the underlying technology for its own interconnect ambitions. Foundries opened photonics processes to outside designers: GlobalFoundries with its Fotonix platform, Tower Semiconductor, TSMC, and public-private efforts such as AIM Photonics in the United States and imec in Belgium. By the early 2020s, silicon photonics had captured a meaningful share of the data centre transceiver market, mainly for single-mode links where it competed against discrete indium phosphide designs. What it had not done was change the architecture of the systems it served. It was still building pluggable modules, cheaper and more integrated than before, but still sitting at the front panel, still fed by a long, lossy electrical path. The opportunity that co-packaging represents is to use the manufacturability of silicon photonics not just to make cheaper modules, but to put optical engines where no hand-assembled transceiver could ever go: on the package, millimetres from the chip. To see why that is worth the trouble, we need to follow the energy. Chapter 3: The Energy Ledger The single most persuasive argument for co-packaged optics is an accounting argument. It does not depend on futuristic claims about light being faster than electricity (in practice, signals in fibre and in copper both travel at a substantial fraction of the speed of light, and neither is meaningfully faster over data centre distances). It depends on a careful look at where energy is spent when one bit moves from one chip to another. Once the budget is laid out line by line, the case for moving the optics closer to the chip becomes almost arithmetic. This chapter lays it out. Following one bit through a pluggable link Consider a bit leaving a switch chip in a conventional system, bound for a server in another rack. Its journey has five stages, and each one costs energy. First, the switch chip's SerDes serialises the bit and drives it onto the package substrate and then the circuit board. Because the trip to the front panel is long and lossy, this is a long-reach SerDes, with strong transmit equalisation and the ability to tolerate a heavily attenuated channel. Second, the signal crosses perhaps fifteen to twenty-five centimetres of board trace, a connector and the module's own circuit board. This passive stage consumes no power directly, but it is the reason the stages either side of it are so expensive. Third, the signal reaches the digital signal processor inside the pluggable module. The DSP digitises the badly degraded waveform, equalises it, recovers the clock, corrects errors where required and regenerates a clean signal. It does this for every lane, in both directions, at full rate. In a typical DSP-based 800-gigabit module the DSP accounts for a large share, often around half, of the module's total power. Fourth, the cleaned signal drives the optical modulator through a driver amplifier, and the laser, biased continuously, supplies the light. Fifth, at the far end, a photodetector and transimpedance amplifier convert the light back to an electrical signal, the far module's DSP cleans and regenerates it, and the far module drives the signal back across another board to another long-reach SerDes on another chip. Count the circuits doing heavy lifting: two long-reach SerDes on the host chips, two DSPs in the modules, and the optical components in between. The optical components, the laser, the modulator, the detector, are a minority of the cost. The majority of the energy is spent pushing electrical signals across short distances of copper and repairing the damage those distances cause. Industry figures bear this out. Surveys of the field put DSP-based pluggable optics at roughly 15 to 20 picojoules per bit for the module alone, before counting the host SerDes. A representative 800-gigabit module dissipates on the order of 14 to 16 watts; divide by 800 gigabits per second and you get roughly 18 to 20 picojoules per bit. The long-reach SerDes driving it from the switch chip adds several more picojoules per bit, depending on design. What co-packaging removes Now move the optical engine to the package, a few millimetres from the switch die. The electrical path from SerDes to optics becomes a short run across the package substrate, or in the most aggressive designs, a connection across a silicon interposer or through a direct die-to-die bond. Three things change at once. The host SerDes can be replaced with a much simpler, lower-power interface, because it no longer needs to tolerate heavy channel loss. Interfaces designed for extra-short reach within a package run at a fraction of the energy per bit of their long-reach cousins. The DSP in the module disappears entirely, because the signal arriving at the optical engine has barely been degraded; a simple linear driver suffices. And the board traces, retimers and flyover cables that carried signals to the front panel are gone, along with the front-panel cages. What remains is the irreducible core of the optical link: the laser, the modulator and its driver, the detector and its amplifier, the tuning circuits, and the short electrical hop at each end. Vendors and researchers report the result in energy per bit. Intel described its optical compute interconnect chiplet, unveiled in 2024, as consuming about 5 picojoules per bit, against roughly 15 for pluggable modules. Surveys of co-packaged designs cite a range of about 5 to 10 picojoules per bit. Nvidia, announcing its Quantum-X and Spectrum-X Photonics switches in March 2025, claimed 3.5 times better power efficiency than conventional pluggable-based networks. These figures are not directly comparable, because each vendor draws the boundary of the link differently, some including the laser's wall-plug power and the host interface and some not. But they converge on the same story: co-packaging cuts the energy of an optical link by roughly a factor of two to four, and the bulk of that saving comes from eliminating electrical work, not from better optics. Table 1 sets out the main approaches side by side, with indicative energy figures and their sources. The ranges are wide, and the reader should treat them as orders of magnitude rather than specifications. Table 1. Indicative energy per bit for chip-to-chip links beyond the package. Approach Where light is made Indicative energy per bit Basis of figure DSP-based pluggable module Front-panel module 15–20 pJ/bit (module only) 2024 industry survey Linear pluggable (no DSP) Front-panel module Roughly half of DSP module Vendor and MSA claims Co-packaged optics, switch Package, external laser About 3.5x better than pluggable Nvidia, March 2025 Optical I/O chiplet Package, on-chip laser About 5 pJ/bit Intel, 2024 Passive copper, in rack No optics Lowest, but reach of 1–2 m Physics of 200G lanes Abstract figures become concrete when multiplied out. Take a 51.2-terabit-per-second switch, the generation that dominated AI back-end networks in 2024 and 2025. Populated with 64 pluggable modules at 800 gigabits each, and assuming 15 watts per module, the optics alone consume 960 watts. That is about 18.75 picojoules per bit, and it does not include the long-reach SerDes on the switch chip. Replace those modules with co-packaged optical engines at, say, 5 picojoules per bit including lasers. At 51.2 terabits per second, the optics now consume about 256 watts. The saving is roughly 700 watts per switch, and the switch chip itself saves further because its SerDes are simpler. These are illustrative numbers built from the published ranges, not a measured system, but the conclusion survives any reasonable choice within those ranges: the optical subsystem of a large switch shrinks from the dominant power consumer in the box to a minor one. Scale up to a cluster. A large AI training cluster with on the order of 100,000 accelerators, connected through a multi-tier network in which each accelerator needs several high-speed optical ports and each tier of switches needs its own, can easily contain several hundred thousand transceivers. Suppose half a million, at 15 watts each. That is 7.5 megawatts drawn by transceivers alone, and once cooling overhead is added, perhaps 9 or 10 megawatts. Cutting that by a factor of three frees several megawatts. In a world where new data centres are limited less by money than by available grid power, megawatts saved on networking are megawatts that can be spent on accelerators. That is why hyperscalers and GPU vendors care. The same arithmetic applied to scale-up networking is even more revealing. In Nvidia's Blackwell generation, each GPU has 1.8 terabytes per second of NVLink bandwidth, which is 14.4 terabits per second. Imagine carrying all of that over pluggable optics at 15 picojoules per bit. Each GPU would need about 216 watts of optics, a significant fraction of the power of the GPU itself, and a 72-GPU rack would spend more than 15 kilowatts on its scale-up links alone. That is why Nvidia uses copper inside the rack: passive copper, while it lasts, costs almost nothing in conversion energy. At 5 picojoules per bit, the optical figure falls to about 72 watts per GPU, still substantial but within reach of a system design. At 1 to 2 picojoules per bit, the target that several optical I/O developers have discussed for future generations, it falls to between about 15 and 30 watts per GPU, comparable to the power of copper SerDes driving a cable across a rack. The threshold at which optics becomes competitive for scale-up, in other words, lies somewhere between today's co-packaged figures and a further two- to fivefold improvement. That gap defines much of the research agenda described in Chapter 7. The new ledger: where co-packaged energy goes Co-packaging does not make the link free. It moves the energy into new places, and some of them are less obvious than the old ones. The laser becomes the largest single item. A laser converts electrical power to light with a wall-plug efficiency that rarely exceeds a few tens of per cent, and much of that light is then lost before it does useful work: in the fibre connection from an external laser module to the package, in the coupler onto the silicon chip, in the modulator itself, in splitters that divide one laser's output among many channels, and in the coupling to the outgoing fibre and again at the far end. The sum of these losses is the link budget, and a typical co-packaged link may lose most of the light the laser produced before it reaches the detector. Every decibel of loss must be paid for with more laser power. Reducing coupling losses is therefore one of the most valuable things packaging engineers can do. Ring tuning is the second item. As Chapter 2 explained, every micro-ring needs a heater and a control loop to hold it on its wavelength. The heater power per ring is small, but there are many rings, and in a hot, fluctuating package environment the heaters may need to work hard. In some designs thermal tuning has accounted for a significant fraction of total link power, and reducing it, through better thermal isolation, smarter control, or designs that tune by shifting which laser wavelength each ring is assigned to, is an active area of engineering. The drivers and receivers are the third. The modulator driver and transimpedance amplifier still run at full data rate, and at 200 gigabits per lane they are demanding analogue circuits. Placing them on an advanced electronic process node, stacked directly on the photonic chip with very short connections, lowers their power. This is one reason TSMC's approach, discussed in Chapter 4, stacks an electronic die on a photonic die with hybrid bonding rather than placing them side by side. The physicist David Miller of Stanford, who has argued for optical interconnects to chips for decades, has pointed out that the real limit on optical link energy lies in the electronics and the efficiency of conversion, not in the photons themselves, and that in principle optical links could operate at energies of tens or hundreds of femtojoules per bit, a femtojoule being a thousandth of a picojoule. Today's co-packaged designs are far above that limit. That gap is a measure of how much improvement remains. Energy is not the only quantity in the ledger. Latency, the time a bit takes to cross a link, matters greatly for AI workloads in which thousands of processors wait on each other at every step. In a DSP-based pluggable link, the signal is digitised, filtered and regenerated in each module, and at high lane rates it is usually protected by forward error correction, which adds redundant bits so that the receiver can correct errors, and which requires collecting and processing blocks of data before they can be passed on. Each of these steps adds delay, typically in the range of tens to hundreds of nanoseconds per link depending on design. A co-packaged link without module DSPs eliminates part of that delay, and designs aimed at scale-up traffic try to minimise error correction for the same reason. Ayar Labs, for instance, quotes a latency of about 10 nanoseconds through its chiplet, excluding time of flight in the fibre. For a single message the savings are small; for collective operations that traverse many links many times per training step, they accumulate. Reliability, too, belongs in the ledger, though it is harder to put a number on. Every circuit removed from a link is a circuit that cannot fail. The DSPs, retimers and connectors that co-packaging eliminates are not just energy consumers but potential points of failure, and their removal is part of the argument for higher network resilience examined in Chapter 6. Energy, latency and reliability point in the same direction: fewer stages of electrical processing make for a better link. Why not just remove the DSP? A natural objection arises. If the DSP in the pluggable module is the problem, why not remove it and keep the pluggable? This is the idea behind linear pluggable optics, which drew intense interest from 2023 onward. A linear pluggable module contains no DSP; it relies on the host chip's SerDes to do all the equalisation across the board, the module and the optical link. Removing the DSP can cut module power substantially, roughly in half by the claims of its proponents, while preserving the field-replaceable module that operators like. Linear pluggables are real products and have found deployments, particularly at 100 gigabits per lane. But they inherit the electrical path that co-packaging eliminates. The host SerDes must now handle the full impairment of the board trace plus the optical link, with no regeneration in between, which makes interoperability between vendors' chips and modules delicate and leaves less margin as lane rates rise. At 200 gigabits per lane the margins become tight, and at 400 they may vanish. A compromise, linear receive optics, keeps a DSP on the transmit side only. These approaches are best understood as a bridge: they recover some of the energy that co-packaging promises while preserving pluggability, and they buy time for co-packaged technology to mature. They do not address the front-panel density limit or the shoreline limit at the chip. The accounting, then, points one way. The cheapest bit is the one that crosses the least copper. The question is how close to the chip the optics can be brought, and what has to be built to bring it there. That is the subject of the next chapter. Chapter 4: Bringing the Optics Home "Co-packaged optics" sounds like a single thing. It is better thought of as a point on a spectrum. At one end sits the familiar pluggable module at the front panel, about as far from the chip as it can be while still inside the box. At the other end sit designs in which photonic circuits are built into the same silicon that carries the compute, or form the interposer on which the compute dies sit. Between these extremes lie several intermediate architectures, each trading electrical distance against serviceability, manufacturing difficulty and cost. Understanding the spectrum explains why different companies have made such different bets. The spectrum of integration The first step away from front-panel pluggables was on-board optics: optical modules mounted on the circuit board partway between the chip and the panel, connected to the panel by fibre. The Consortium for On-Board Optics, formed in 2015 with Microsoft among its founders, wrote specifications for such modules. On-board optics shortened the electrical path somewhat and freed front-panel space, but it did not shorten the path enough to remove the DSP or change the economics decisively, and it never achieved broad deployment. It is mainly remembered as a stepping stone that taught the industry how to manage fibre inside a chassis. The next step is near-package optics. Here the optical engines sit on the same board as the switch chip, or on a shared high-performance substrate, within a few centimetres of the chip, typically connected through high-quality short traces or sockets. The electrical path is short enough to permit simpler SerDes, and because the engines are socketed rather than soldered, they can in principle be replaced. Near-package optics appeals to operators who value serviceability and to vendors who want to keep the optical engine supply chain separate from the switch chip. True co-packaged optics places the optical engines on the same package substrate as the chip, typically arranged around the perimeter of the switch die like the petals of a flower. The electrical path is now a matter of millimetres across the organic substrate. This is the architecture of the first commercial co-packaged switches from Broadcom and Nvidia. Optical engines are attached to the substrate during package assembly, and fibre leaves each engine through a connector or a permanently attached fibre array. The laser usually sits outside, in a separate module at the front panel, delivering continuous light to the engines through fibre. Further along, optical engines move onto a silicon interposer shared with the compute die, or are stacked in three dimensions. Interposers are slabs of silicon carrying very fine wiring, already used to connect GPUs to their high-bandwidth memory. Putting photonic engines on the interposer, or making the interposer itself photonic, shortens the electrical connections to micrometres and allows enormous connection density. At the extreme, the electronics and photonics share a single die: so-called monolithic integration, in which transistors and optical devices are fabricated side by side in the same process. Table 2 summarises the main points on this spectrum. Table 2. The main levels of optical integration, from front panel to chip. Level Electrical path to optics Usual laser location Field replaceable? Examples Pluggable module 15–25 cm of board Inside module Yes, hot-swap 800G and 1.6T OSFP On-board / near-package A few cm Engine or external Sometimes, via socket COBO modules, prototypes Co-packaged on substrate A few mm External module Laser yes; engine no Broadcom, Nvidia switches Interposer or 3D stack Micrometres External module No Lightmatter Passage Optical I/O chiplet Die-to-die link External or on-chip No Ayar Labs, Intel OCI Engines, chiplets and stacks Inside a co-packaged design, the basic unit is the optical engine: a photonic integrated circuit carrying modulators, detectors, waveguides and couplers, paired with an electronic integrated circuit carrying drivers, amplifiers, serialisers and control logic. How those two dies are joined is one of the most consequential choices in the design. The simplest approach places the electronic and photonic dies side by side on the substrate, connected by short wires or traces. That is easy to manufacture but leaves the connection between driver and modulator long enough to waste energy and bandwidth. Better is to flip the electronic die face down onto the photonic die with micro-bumps, shortening the connections to tens of micrometres. Best of all, by the standards of current technology, is hybrid bonding, in which the copper pads of two dies are fused directly, with no solder, at pitches of a few micrometres. TSMC's Compact Universal Photonic Engine, known as COUPE, uses its SoIC hybrid-bonding technology to stack an electronic die on a photonic die. Nvidia's switch designs use COUPE-based optical engines: in the Quantum-X Photonics switch, each optical subassembly carries three COUPE engines, each handling 1.6 terabits per second in each direction, for 4.8 terabits per second per subassembly. TSMC has presented a roadmap in which COUPE begins in pluggable modules and moves progressively into the package and then onto the interposer, with bandwidth per engine rising at each step. A different organising principle is the optical I/O chiplet. Here the optical engine is designed as a self-contained die that a chip designer can place in a package next to a processor, just as one would place a memory stack or another compute die. The chiplet speaks a standard die-to-die electrical interface on one side and emits light through fibres on the other. Universal Chiplet Interconnect Express, or UCIe, an industry standard for die-to-die links launched in 2022, gives such chiplets a common electrical language. Ayar Labs' TeraPHY chiplet is the leading example: the company's published specifications describe eight optical ports at 512 gigabits per second each, roughly 4 terabits per second per chiplet, each port carrying sixteen wavelengths, a UCIe electrical interface and latency of about 10 nanoseconds per chiplet, with light supplied by its separate SuperNova laser source. Intel's optical compute interconnect chiplet, shown in 2024, offered 4 terabits per second bidirectionally over eight fibre pairs, each carrying eight wavelengths, with lasers integrated on the chip. The chiplet model matters commercially as much as technically. A processor designer who wants optical I/O does not have to become a photonics company; it can buy a qualified optical chiplet, place it in the package and connect it through a standard interface, much as it already buys memory stacks from memory makers. The optical supplier, in turn, can sell the same chiplet to many customers, spreading its development costs. This is the same division of labour that has made the chiplet approach attractive throughout the semiconductor industry, and it lowers the barrier to adopting optics in accelerators whose makers lack photonic expertise. The cost is that the die-to-die interface becomes a boundary in the system, with its own power and latency, and that integration testing across two suppliers' dies becomes a shared responsibility. Lightmatter has gone further with a photonic interposer. Its Passage M1000, announced in March 2025, is a large active photonic platform, more than 4,000 square millimetres, on which compute dies can be mounted. Rather than confining optical input and output to the edge of the package, it distributes optical connections across the whole surface, a property the company's chief executive, Nick Harris, summed up by saying that shoreline was no longer a limitation for I/O. Lightmatter quoted a total optical bandwidth of 114 terabits per second carried over 256 fibres, with eight wavelengths per fibre. The interposer approach attacks the shoreline problem described in Chapter 1 directly: if light can leave from anywhere on the surface, the edge of the die stops being the bottleneck. Walking around a co-packaged switch package, as an engineer might on a design review, makes these choices concrete. In the centre sits the switch die, or several dies joined together, under a heat spreader or cold plate. Around it, arranged on the substrate, are the optical engines, perhaps eight or sixteen of them in a switch generation of the mid-2020s, each a small stacked assembly of electronic and photonic silicon. Short electrical traces in the substrate join each engine to a bank of SerDes on the facing edge of the switch die. From the outer edge of each engine emerges a fibre array: some fibres carry outgoing data, some carry incoming data, and some bring in light from the external laser modules. The fibres are gathered into connectors at the edge of the package or routed directly toward the front panel, where they meet the panel connectors and the laser module cages. Everything that once filled the front of the box, the rows of module cages, their heat sinks and the board traces leading to them, has been replaced by a few hundred grams of fibre and a row of laser modules. That layout highlights a metric that matters as much as energy: bandwidth density, usually expressed as terabits per second per millimetre of die or package edge. The shoreline argument of Chapter 1 says that only so much edge is available. An optical engine that can deliver more bandwidth per millimetre of the edge it occupies allows more total bandwidth around a given package. Wavelength-division multiplexing raises density by placing several channels on each waveguide and fibre. Higher per-wavelength rates, from 100 to 200 gigabits and beyond, raise it further. Stacking electronics on photonics, rather than placing them side by side, reduces the area each engine needs. And interposer approaches like Lightmatter's sidestep the edge altogether. Each generation of co-packaged design can be read as an attempt to raise bandwidth density while holding energy per bit down. Monolithic versus heterogeneous Two philosophies compete over how the electronics and photonics should be made. The monolithic school builds both in the same process on the same wafer. GlobalFoundries' Fotonix platform is the main commercial example, combining radio-frequency-capable transistors with photonic devices on one chip. Ayar Labs' early chiplets were built on a GlobalFoundries monolithic process. The advantages are tight integration, the shortest possible connections between driver and modulator, and a single die to handle. The disadvantage is that the transistors available in a photonics-friendly process lag far behind leading-edge logic, and the photonic devices must accept compromises dictated by the electronics. The heterogeneous school makes each die in the process best suited to it, electronics in an advanced logic node and photonics in a dedicated photonic process, and joins them by stacking. This is TSMC's approach with COUPE, and it is the direction most of the industry is moving, helped by the maturity of hybrid bonding developed for other stacked products. It allows the electronics to improve on the logic roadmap while the photonics improves on its own, at the cost of a more complex assembly flow. Neither approach settles the third ingredient, the laser. As noted in Chapter 2, almost every first-generation co-packaged system keeps the laser off the package in an external laser source. That decision has consequences for packaging: every optical engine needs not only fibres to carry its data out but also fibres to bring the laser light in. A co-packaged switch therefore has two families of fibre attached to its package, and both must be aligned with micrometre precision. The case for stopping short of full co-packaging deserves a fair hearing, because several large operators have made it. Near-package optics, with engines socketed rather than permanently attached, preserves the ability to replace an optical engine in the field and keeps the optical supply chain separate from the switch chip, so an operator can buy engines from multiple suppliers. The electrical path, a few centimetres rather than a few millimetres, is long enough to cost some energy but short enough to allow simpler SerDes and to remove the module DSP. Advocates argue that this captures most of the energy benefit at a fraction of the operational disruption. Critics answer that sockets introduce their own electrical discontinuities and reliability concerns, that the extra centimetres become harder to justify with each doubling of lane rate, and that the replaceability benefit is smaller than it looks once external lasers have already moved the most failure-prone component off the package. The debate is unresolved, and it is likely that different tiers of the network, and different operators, will settle at different points. What the chip designer gives up Moving optics onto the package is not free for the chip itself. Optical engines occupy substrate area that could otherwise carry power delivery, decoupling capacitors or additional memory. They must sit close enough to the die to keep electrical paths short, which places them in the warmest part of the package. They add fibres that must leave the package without being crushed by the heat sink or cold plate, and the cooling hardware must be redesigned to accommodate them. The package becomes a mixed electronic, optical and mechanical assembly, and its yield becomes the product of the yields of every component on it. These costs explain why the industry approached co-packaging cautiously and why the first products are switches rather than processors. A switch chip, unlike a GPU, does not need room on its package for memory stacks, and its value lies almost entirely in its input-output capacity. The penalty of giving substrate area to optical engines is smallest, and the reward largest. For accelerators the calculus is harder: package area is fought over by compute dies and high-bandwidth memory, and the risk of a costly processor being ruined by a faulty optical component weighs heavily. The integration spectrum also clarifies a debate that is sometimes presented as binary. The question is not "pluggable or co-packaged?" but "how much electrical distance is worth paying to keep which kind of serviceability?" At each step along the spectrum, power falls and density rises, and at each step something that used to be replaceable becomes fixed. Where a given operator lands depends on how much it values the power savings against the operational habits built around replaceable modules. Those habits rest on a long experience of lasers and fibre connections failing in the field. The next chapter turns to the engineering problems that make that experience relevant: lasers, fibres, heat and test. Chapter 5: Lasers, Fibres and Heat If co-packaged optics is such a clear win on energy, why did it take so long to arrive? The answer lies in four engineering problems that are trivial in a pluggable module and severe on a package: where to put the laser, how to attach fibre to a chip, how to keep temperature-sensitive optics working beside a hot processor, and how to test an assembly whose components cannot easily be replaced. Each of these problems has been substantially solved, in the sense that products are shipping. None has been solved in the sense that the solutions are cheap, mature and standard. This chapter takes them in turn. The laser question Lasers are the component most likely to fail in an optical link. Their output degrades over time through mechanisms that include the slow growth of crystal defects in the active region and damage at the facet where light exits, and almost all of these mechanisms accelerate sharply with temperature. A common rule of thumb is that laser wear-out rates roughly follow an exponential law in temperature, so a laser running twenty or thirty degrees hotter may age several times faster. In a pluggable module this is a manageable problem. The laser sits at the front panel, in the coolest air in the chassis, and if it fails, a technician replaces the module in a minute. On a package beside a switch chip or GPU, the laser would face temperatures much higher and much more variable than at the front panel, and a failure would have no simple remedy. For this reason most co-packaged designs place the lasers in an external laser source: a separate, replaceable module, usually at the front panel, that produces continuous unmodulated light and sends it through fibre to the optical engines on the package. The engines modulate that light with data; they never have to generate it. Nvidia's photonics switches are a clear example. According to Nvidia's technical description, each external laser source module contains eight lasers that together supply 32 of the 576 transmit lanes in a Quantum-X Photonics switch. One laser feeds several lanes by splitting its light, which is part of the reason Nvidia claims the design uses four times fewer lasers than an equivalent network built from pluggable modules. The laser modules are field-replaceable, so the most failure-prone part of the system remains serviceable even though the optical engines are permanently attached. The Optical Internetworking Forum has published an implementation agreement for a standard external laser module, the External Laser Small Form Factor Pluggable, or ELSFP, so that switch builders and laser suppliers can work to a common mechanical, electrical and optical specification. External lasers carry their own costs. The light must travel through a connector at the front panel, along a fibre to the package, and through another coupling onto the silicon chip, losing power at each step, so the laser must be more powerful than an integrated one would need to be. Higher power means greater stress on the laser and, at sufficiently high power, safety and connector-damage concerns. Many silicon photonic devices respond only to light of one polarisation, so the delivery fibre must often preserve polarisation, which requires special fibre and careful rotational alignment at each end. And because one laser may feed many lanes, a single laser failure takes down several lanes at once; designs mitigate this with redundancy or by sizing the blast radius of each laser to what the network can tolerate. The alternative, integrating lasers directly into the optical engine, is championed by Intel, which pairs its on-chip lasers with a strong reliability argument. Intel has said that it has shipped more than 32 million integrated lasers in its silicon photonics products with a failure rate below 0.1 FIT, where one FIT means one failure per billion device-hours. If that record carries over to the hotter package environment, on-chip lasers would remove the external module, its connector and its losses. Most of the industry has not yet been willing to bet on it for first-generation products, but the long-term direction may well favour integration as reliability data accumulates. A little arithmetic shows why laser reliability dominates these decisions. Suppose a large AI cluster contains, across its switches, a million lasers. If each laser has a failure rate of 0.1 FIT, the cluster as a whole will experience one laser failure every ten thousand hours, a little more than once a year. At 10 FIT, still a respectable figure for many optical components, it will see one failure every hundred hours, roughly every four days. At 100 FIT, it is one every ten hours. The difference between these regimes is the difference between a cluster in which laser failure is a rare event and one in which technicians are permanently occupied replacing lasers. If the lasers are on the package and cannot be replaced, a rate of one failure every few days means retiring a switch every few days. If they are in external modules, the same failure rate is an inconvenience. Sharing lasers among channels, as the external-source designs do, cuts the number of lasers and therefore the failure count, but it enlarges the consequences of each failure. Designers must balance these effects, and the right balance depends on failure rates that are only now being measured at scale in the co-packaged environment. Laser safety adds a further consideration. External lasers must deliver enough light to survive the losses of the delivery path and still power many modulators, which means their output, concentrated in a single fibre, can be substantial. Infrared light at these wavelengths is invisible, and a high-power beam escaping from an unmated connector could harm an eye. Designs therefore include interlocks that shut down the laser when a connector is disturbed, and standards bodies have had to define safe power levels and handling procedures for the external laser modules that technicians will remove and replace. Attaching the fibre Light leaves a silicon chip through a waveguide about half a micrometre across. It must enter a single-mode fibre whose core is about nine micrometres across. The optical modes, the shapes of the light fields in each, are mismatched by more than a factor of ten in size, and coupling between them without loss requires both a mode converter and extraordinarily precise alignment. An offset of a single micrometre between waveguide and fibre can cost a decibel or more of light, which is a fifth of the power gone. Two coupling strategies dominate. Edge coupling tapers the silicon waveguide at the edge of the chip so that the mode expands to approach the size of the fibre mode, and the fibre is butted against the polished edge. Edge couplers can achieve low loss across a broad range of wavelengths, which suits wavelength-division multiplexing, but they require precisely prepared chip edges and fibre positioned with sub-micrometre accuracy in all three dimensions. Grating couplers instead etch a periodic pattern into the waveguide surface that scatters light upward toward a fibre held above the chip at a slight angle. Grating couplers are easier to test at wafer scale, because light can be shone onto them from above before the wafer is diced, and their alignment tolerances are somewhat looser, but they are typically lossier and work well only over a limited wavelength range and polarisation. In a pluggable module, fibre attachment happens once, in a controlled factory, on a small module. In co-packaged optics it must happen on a large, expensive package, often for dozens or hundreds of fibres, and the fibre must survive the subsequent steps of assembly. That last point is subtle. Electronic packages are built using solder reflow, which heats the entire assembly to around 250 degrees Celsius. Many of the adhesives and polymers used to fix fibres cannot survive that temperature, and the thermal expansion of different materials can shift fibres out of alignment. Packaging flows must either attach fibres after reflow, use materials that tolerate it, or use detachable optical connectors that are mated after the package is complete. Detachable connectors have the added advantage that a damaged fibre can be replaced without scrapping the package, which is why vendors and connector makers such as SENKO and Corning, both named partners in Nvidia's co-packaged program, have invested heavily in them. Passive alignment features help scale the process. Silicon chips can have V-shaped grooves etched in them, into which fibres drop and are self-aligned by the geometry of the groove, eliminating the need for active alignment in which light is shone through the fibre and its position adjusted while the output is monitored. Active alignment yields the best coupling but is slow; passive alignment is fast but demands extremely tight fabrication tolerances. The transition from active to passive alignment at high yield is one of the quiet manufacturing advances that has made co-packaging practical. Contamination is the other enemy. Anyone who has worked in a data centre knows that dirty connectors are among the most common causes of optical link problems. A speck of dust a few micrometres across on the end face of a single-mode fibre can block a substantial part of its nine-micrometre core, and oils from a fingertip can do the same. With pluggable optics the remedy is routine: inspect the connector with a small microscope, clean it, reinsert it. Co-packaged systems multiply the number of optical interfaces inside the box, at the package, at internal connectors and at the panel, and some of those interfaces are hard to reach. Designers respond by minimising the number of mated connections, sealing internal interfaces at the factory, using expanded-beam connectors that spread the light over a larger area so that a given particle blocks a smaller fraction of it, and building inspection and cleaning steps into assembly and service procedures. The external laser connections deserve particular care, because high optical power focused on a contaminated surface can heat the contaminant and permanently damage the fibre end face. Once attached, the fibres must be managed. A co-packaged switch may have more than a thousand fibres leaving its package, both for data and for laser delivery. They must route from the package to the front panel without sharp bends, without being pinched by the heat sink or cold plate, and in an arrangement that maps correctly to the front-panel connectors. Fibre shuffles, small assemblies that reorganise fibres from the order in which they leave the package to the order needed at the panel, have become a distinct product category. Heat and the drifting ring Chapter 2 described the temperature sensitivity of micro-ring modulators. On a co-packaged switch or accelerator, that sensitivity meets an environment that is both hot and unsteady. A switch chip under heavy load and a GPU swinging between compute-bound and communication-bound phases can both change temperature by many degrees within seconds. Each ring's resonance drifts with the local temperature, and the control loop must chase it. Engineers attack this from several directions. Thermal design keeps the photonic engines as cool and stable as possible: engines are placed at the edge of the substrate, separated thermally from the hottest parts of the die, and cooled by the same liquid cold plates that now cool the chips themselves. Nvidia's photonics switches are liquid-cooled, reflecting both the switch chip's power and the need to stabilise the optics. Control circuits measure each ring's alignment continuously, often by monitoring a small fraction of light at a detector, and adjust its heater. Designers can build in a margin of tuning range and, in multi-wavelength systems, allow each ring to lock to whichever laser wavelength is nearest rather than a fixed one, which reduces how far any ring must be tuned. Some research groups and companies explore athermal designs that combine silicon with materials whose refractive index changes in the opposite direction with temperature, cancelling the drift, though these are not yet standard. A rough calculation shows why the tuning range matters. If a ring's resonance moves by around a tenth of a nanometre per degree, then a 20-degree swing in local temperature moves it by about two nanometres. Wavelength-division multiplexed channels in co-packaged designs are often spaced by around one to two nanometres. A ring that drifts two nanometres without correction has not merely lost its own channel; it may have wandered onto its neighbour's. The control system must therefore be able to tune each ring over a range comparable to the channel spacing, and it must know which wavelength each ring is meant to lock to, so that rings do not end up following the wrong laser. Heaters can only raise temperature, so rings are usually designed to resonate slightly below their target wavelength at the coldest expected condition and are heated into position, meaning some heater power is spent continuously. Getting this right at start-up, when the system must locate every ring and every laser line from a cold state, is a nontrivial piece of control engineering in its own right. Mach-Zehnder modulators, being far less temperature-sensitive, offer an alternative for designers willing to accept their larger size and drive power. Broadcom's co-packaged switches have used Mach-Zehnder-based optical engines, while Nvidia chose rings. The divergence reflects a real engineering trade-off between density and robustness rather than a settled answer. Testing what cannot be reworked The least glamorous problem may be the most important commercially. A conventional switch is built from a chip, a board and a set of modules, each tested separately and each replaceable. If a module is bad, swap it. A co-packaged switch combines a costly switch chip, many optical engines, fibres and a substrate into one package. If any optical engine is defective after assembly, the whole package may be lost, including the chip. The arithmetic of compounding yield is unforgiving. Suppose each optical engine has a 99 per cent chance of being good after assembly. A package with eight engines has a probability of about 92 per cent of having all eight good. With sixteen engines the figure falls to about 85 per cent, and with thirty-two to about 72 per cent. Every lost package wastes a switch chip worth thousands of dollars. The only answer is to make sure that each component is known to be good before it is committed to the package, and to make the assembly steps themselves very reliable. That requires optical testing at wafer scale: probing photonic chips on the wafer with light as well as electrical signals, characterising every modulator, detector and coupler, and binning dies before they are diced. It requires testing optical engines again after the electronic and photonic dies are joined, and before they are mounted. It requires burn-in procedures that weed out early failures without wearing out good parts. Electronic test is a mature industry with standardised equipment; optical wafer test has had to be largely invented, by equipment makers and by foundries, over the past decade. Its maturation is a large part of why co-packaged products became manufacturable in the mid-2020s rather than earlier. Design for redundancy helps too. Spare lanes on each engine allow a defective lane to be mapped out. Separating the laser into a replaceable module removes the riskiest component from the yield calculation. Detachable fibre connectors allow damaged fibres to be fixed. Each of these measures reduces the penalty of a single failure, at some cost in area or complexity. These four problems, laser placement, fibre coupling, thermal stability and test, are not the kind that yield to a single breakthrough. They yield to engineering discipline, manufacturing investment and accumulated experience, the same forces that turned the transistor from a laboratory curiosity into a commodity. By the mid-2020s that investment had gone far enough for the first commercial products to ship. The next chapter looks at them. Chapter 6: First Light in the Switch Technologies that change infrastructure rarely arrive in the place where they will eventually matter most. They arrive where the problem is sharpest and the risk is easiest to bear. For co-packaged optics, that place was the network switch. This chapter traces how co-packaged switches went from demonstration to product, what the first generation actually delivers, and what the early evidence says about the question every operator asks first: does it break? Why the switch went first A network switch is a pure input-output device. Its silicon exists to receive packets on one port and send them out on another, as fast as possible. Its value is measured in terabits per second, and nearly all the power in the box goes into moving bits on and off the chip and through the optics. That makes the energy argument of Chapter 3 strongest for switches: a large fraction of switch power sits in exactly the electrical paths and module DSPs that co-packaging removes. Switches also escape some of the risks that make co-packaging harder for processors. A switch chip, unlike a GPU, does not share its package with high-bandwidth memory, so there is room around the die for optical engines. Switches are deployed in large numbers of identical units, which lets operators accumulate reliability statistics quickly. And the network is designed with redundancy: a cluster's fabric has many parallel paths, so a failed port or even a failed switch reduces capacity rather than halting the system, provided the software handles it gracefully. Finally, AI changed the shape of the problem. A conventional cloud data centre network is lightly loaded much of the time and tolerates occasional congestion. The back-end network that connects GPUs for training is run near full capacity, its traffic is synchronised across thousands of accelerators, and it is enormous: a cluster of 100,000 GPUs with a non-blocking fabric needs tens of thousands of switch-to-switch and switch-to-GPU optical links per tier. Any per-link saving in power, cost or failure rate is multiplied accordingly. Broadcom: three generations of engines Broadcom, the largest supplier of merchant Ethernet switch silicon, has pursued co-packaging longer and more steadily than anyone else. Its first commercial co-packaged switch, built around its 25.6-terabit-per-second Tomahawk 4 chip and known by the code name Humboldt, was released in 2022, with the Chinese cloud operator Tencent among its early users. The second generation, Bailly, paired the 51.2-terabit Tomahawk 5 with co-packaged optical engines and was introduced in 2024. Broadcom described it as substantially reducing the power needed for optical interconnect compared with pluggable modules, with claimed savings on the order of 70 per cent. The third generation, built on the 102.4-terabit Tomahawk 6 under the code name Davisson, followed in 2025, doubling capacity again while moving the electrical lanes to 200 gigabits each. Broadcom's approach is notable for its continuity. Each generation uses a set of optical engines arranged around the switch die on the package substrate, each engine built in silicon photonics, with light supplied by replaceable external laser modules at the front panel. Broadcom's engines have been built around Mach-Zehnder modulators, which are larger than rings but more tolerant of temperature, a choice consistent with the company's emphasis on manufacturability and robustness. The steady cadence matters because the hardest problems of co-packaging, test, yield, fibre handling and field reliability, improve with each production cycle. By the time Davisson arrived, Broadcom had several years of shipping experience to draw on. Meta has been the most visible hyperscale partner in evaluating Broadcom's co-packaged switches in production-like conditions. In 2025, Meta engineers presented results from Broadcom co-packaged switches operating in its data centres, describing cumulative operation on the order of a million device-hours and reporting reliability that compared favourably with pluggable optics, notably on the frequency of link flaps. The exact figures depend on how one counts devices and events, and data from a single operator's environment should not be over-generalised. But it was the first substantial public evidence that co-packaged optics could survive real data centre conditions, and it helped change the industry's conversation from "if" to "when". Nvidia: photonics for the AI fabric Nvidia came to co-packaged optics from a different direction. Its networking business, built largely on the acquisition of Mellanox in 2020, supplies both InfiniBand and Ethernet fabrics optimised for AI clusters, and Nvidia sells not just switches but whole systems in which it controls the GPU, the network adapter, the switch and the software. That vertical position lets Nvidia make architectural changes faster than a component vendor can, and it gives it a direct interest in the total power of an AI factory, since every watt saved on networking is a watt available for its GPUs. At its GTC conference in March 2025, Nvidia announced two families of co-packaged switches. Quantum-X Photonics is an InfiniBand switch offering 144 ports of 800 gigabits per second, built from four switch chips with co-packaged optics, for a total of 115.2 terabits per second; Nvidia said it would be available later in 2025. Spectrum-X Photonics is an Ethernet family scheduled for 2026, in configurations of 128 ports of 800 gigabits per second, a total of 102.4 terabits, and a much larger one offering 512 ports of 800 gigabits for about 400 terabits per second. Nvidia's announcement claimed 3.5 times better power efficiency, 63 times better signal integrity, ten times better network resiliency at scale, four times fewer lasers and 1.3 times faster deployment than conventional approaches. These are vendor claims, framed against Nvidia's own choice of baseline, and should be read as such, but they indicate where Nvidia believes the value lies. The order of Nvidia's launches is telling. The InfiniBand product came first, the Ethernet family later. InfiniBand is a fabric Nvidia controls nearly end to end, used in clusters where Nvidia typically supplies the adapters, switches, cables and software, and where the operator has already accepted a single-vendor network. That made it the natural place to introduce a technology that changes how optics are bought and serviced. Ethernet, by contrast, is a multi-vendor world in which operators expect to mix switches and optics from many suppliers. Bringing co-packaging there requires more attention to interoperability, management standards and operator habits, and benefits from experience gained in the more controlled environment first. Broadcom, whose business is Ethernet, has had to solve those problems from the outset, which partly explains its slower and more incremental approach. The technical design reflects the choices explored in earlier chapters. Nvidia's optical engines use micro-ring modulators running at 200 gigabits per second per wavelength using PAM4, developed with TSMC and built on TSMC's COUPE stacking technology. In Quantum-X Photonics, each optical subassembly carries three COUPE engines, each handling 1.6 terabits per second in each direction. In Spectrum-X Photonics, a switch module carries 32 silicon photonic engines at 3.2 terabits per second each. Light comes from external laser source modules at the front panel, each with eight lasers. The supply chain Nvidia assembled reads like a census of the optical industry: TSMC for the engines, Coherent and Lumentum for lasers, Corning for fibre, SENKO and Browave for connectivity, Fabrinet, Foxconn and others for assembly. The largest Spectrum-X configuration points to a benefit of co-packaging that goes beyond power: radix, the number of ports a single switch can offer. A network built from switches with more ports needs fewer tiers to connect a given number of endpoints. With a radix of 64, a two-tier non-blocking network can connect about two thousand endpoints; a three-tier network is needed for more. With a radix of 512, a two-tier network can connect on the order of a hundred thousand. Each tier removed eliminates a layer of switches, the optical links between them, their power and their latency. Pluggable modules limit radix because each port needs faceplate space and a module; co-packaged switches, whose ports leave the box as fibre, can offer far more ports per unit of rack space. For very large AI clusters, flattening the network in this way may prove as valuable as the energy saved on each link. Table 3 compares the first generations of co-packaged switches from the two leading vendors, using the vendors' own published figures. Table 3. First-generation co-packaged switch products. Product Vendor Switch capacity Ports or lanes Introduced Laser source Humboldt (Tomahawk 4) Broadcom 25.6 Tb/s 100G-class lanes 2022 External Bailly (Tomahawk 5) Broadcom 51.2 Tb/s 100G-class lanes 2024 External Davisson (Tomahawk 6) Broadcom 102.4 Tb/s 200G lanes 2025 External Quantum-X Photonics Nvidia 115.2 Tb/s 144 x 800G Announced 2025 External, 8 lasers per module Spectrum-X Photonics Nvidia 102.4 to ~400 Tb/s 128 or 512 x 800G 2026 External The link flap and why resilience matters Of the claims attached to co-packaged switches, the one that surprises newcomers most is resilience. Intuition suggests that a technology which welds optics to the chip should be less resilient, not more, since a failure cannot be fixed by swapping a module. The argument for the opposite runs through a phenomenon called the link flap. A link flap occurs when a connection between two devices briefly goes down and comes back up. In a conventional pluggable network, flaps have many causes: a marginal module, a dirty or slightly disturbed connector, a DSP losing lock on a degraded signal, a firmware glitch. Each pluggable link contains two modules, two DSPs, two sets of electrical connectors and the electrical channel between chip and panel, and each is an opportunity for intermittent trouble. In ordinary cloud networking, a flap is a nuisance, absorbed by routing protocols. In large-scale AI training it can be costly. Training jobs synchronise thousands of GPUs at every step, and collective communication libraries are sensitive to a link disappearing in the middle of an operation. A flap can stall a job, cause a timeout, and in the worst case force a restart from the last checkpoint, wasting minutes or hours of an entire cluster's time. Co-packaging eliminates the module DSPs, the front-panel electrical connectors and the long board traces. It reduces the number of discrete components per link. It also, in Nvidia's and Broadcom's designs, reduces the number of lasers by sharing each laser across several channels, and moves those lasers into cooler, managed modules. Fewer components mean fewer failure points, and particularly fewer of the intermittent faults that cause flaps. This is the logic behind Nvidia's resilience claim and behind the interest in Meta's link-flap data. The relevant comparison is not "can a co-packaged port be swapped like a pluggable?" but "how many disruptions per million hours does each architecture deliver to a running AI job?" There is a counterweight. When a co-packaged optical engine does fail permanently, all its ports go with it, and repair means replacing the switch, or at least its main board. Operators must plan spare switches rather than spare modules, and they must have confidence that hard failures will be rare enough for this to be acceptable. Early experience suggests the rate is manageable, but a few years of data at modest scale are not the same as a decade of data at hyperscale, and many operators are watching before committing. How does an operator actually decide whether to deploy a co-packaged switch? The process resembles the qualification any new network platform goes through, with some additions. The operator first tests a handful of units in a laboratory, checking that the optical links meet their specifications for power, error rate and margin across the full range of temperatures and loads the switch will see. It verifies that co-packaged ports interoperate with the pluggable modules and network adapters already in its fleet, since no operator replaces an entire network at once; a co-packaged switch must talk to pluggable optics at the far end of most of its links during any transition. It integrates the switch's telemetry into its monitoring systems, which means learning to read new kinds of data, such as the health of external laser modules and the tuning state of ring modulators, and to set alarms that predict failures before they occur. Next comes a pilot deployment in production, typically in a single cluster or part of one, where the switches carry real traffic and their failure statistics can be compared against the pluggable-based switches beside them. This is where link flaps, hard failures, laser replacements and any unexpected interactions with software are counted. It is also where the operator rehearses its repair procedures: how long does it take to replace a whole switch, how are its many fibres disconnected and reconnected without damage, and how quickly can the network route around it? Only after this stage, which can last many months, will an operator commit co-packaged switches to the core of a large training cluster. The economic test runs in parallel. The operator compares the capital cost of a co-packaged switch, including its laser modules and fibre, with the cost of a conventional switch plus its pluggable modules, then adds the value of the power saved over the switch's working life. In a facility where power is the binding constraint, the operator also asks how many additional accelerators the saved power would support, and what those accelerators would earn. For a large cluster, that last calculation can dominate all the others, and it is the main reason AI operators have been willing to move faster on co-packaging than conventional cloud operators. The first generation establishes several points. Co-packaged optics can be manufactured at commercial yield. It can be integrated into switch systems that operators deploy in production. It delivers the power savings that the accounting predicts, at least within the range of vendor claims and early independent observations. And it does not appear to suffer the catastrophic reliability problems that sceptics feared. What it does not yet prove is that co-packaging is the right answer for every network tier, every operator and every lane rate. Pluggable optics, including linear pluggables, remain the default across most of the world's data centres, backed by a deep, competitive and multi-vendor supply chain. Market analysts such as TrendForce have expected broad adoption of co-packaged switches to unfold over 2027 and 2028 rather than immediately. The first generation also does not touch the place where many engineers believe co-packaging will ultimately matter most: the accelerator itself. The switch is a proving ground. The GPU is the destination. That is the subject of the next chapter. Chapter 7: Optics at the Accelerator The network switch is where co-packaged optics proved itself. The accelerator is where it may change the shape of computing. Putting optical input and output directly on the package of a GPU or custom AI chip does more than save energy on links that already exist. It removes the reach limit that copper imposes on the most tightly coupled tier of an AI system, and with it the reason that today's largest scale-up domains are confined to a single rack. This chapter explains why that matters, who is building the technology, and what stands in the way. Scale-up, scale-out and the tyranny of the rack Large AI systems have two kinds of network. The scale-out network connects servers and racks across a data centre, typically using Ethernet or InfiniBand through tiers of switches. It is where co-packaged switches are arriving first. The scale-up network connects accelerators within a tightly coupled group so closely that they can share memory and exchange data at very high bandwidth and low latency, behaving almost like one giant processor. Nvidia's NVLink is the best-known scale-up fabric; others include AMD's Infinity Fabric and the open UALink standard, developed by a consortium formed in 2024. The two networks differ by an order of magnitude in bandwidth per accelerator. In Nvidia's Blackwell generation, each GPU has 1.8 terabytes per second of NVLink bandwidth, many times its scale-out network bandwidth. Scale-up traffic carries the most communication-intensive parts of model parallelism: the frequent exchanges needed when a single layer of a model is split across many chips, and, increasingly, the traffic generated when serving very large mixture-of-experts models, where each token is routed to experts spread across many accelerators. The larger the scale-up domain, the larger the model that can be split across it efficiently, and the more flexibly work can be distributed. As Chapter 1 described, scale-up fabrics today run over copper because copper is cheap and needs no conversion, and they are confined to a rack because copper at 200 gigabits per lane cannot reach further. The result has been an arms race in rack density. Nvidia's GB200 NVL72 put 72 GPUs in one rack, drawing on the order of 120 kilowatts. Nvidia's announced roadmap pushes scale-up domains into the hundreds of GPUs in later generations, packed into still denser racks with still greater power. That path is feasible, but it concentrates enormous power and heat in a small footprint, strains the power delivery and cooling of data centres, and makes every rack a feat of mechanical engineering. Optical scale-up offers an exit. If each accelerator can emit its scale-up bandwidth as light directly from its package, the domain is no longer bounded by the reach of copper. A scale-up domain could span several racks, or a whole row, with the accelerators spread out to ease power density and cooling. Light in fibre adds roughly five nanoseconds of delay per metre, so a link of tens of metres adds latency measured in tenths of a microsecond, acceptable for most scale-up traffic. The architecture of the AI system could then be designed around the needs of the workload rather than the reach of a cable. The contenders Several companies are competing to supply optical input and output for accelerators. They differ in their technology choices and business models. Table 4 sets out the principal contenders and the figures they have published. Table 4. Leading developers of optical I/O for accelerators and memory, as of late 2026. Company Approach Published headline figure Notable status Ayar Labs TeraPHY chiplet with external SuperNova laser About 4 Tb/s per chiplet, 16 wavelengths per port $500M Series E, March 2026 Lightmatter Passage photonic interposer and chiplets M1000: 114 Tb/s over 256 fibres M1000 announced March 2025 Celestial AI Photonic Fabric chiplets and memory links 16 Tb/s per chiplet (company figure) Acquired by Marvell, announced Dec 2025 Intel Optical compute interconnect chiplet, on-chip lasers 4 Tb/s at about 5 pJ/bit Demonstrated 2024 Ayar Labs, founded in 2015 out of research at MIT, the University of California, Berkeley, and the University of Colorado, has the longest track record of the startup contenders. Its TeraPHY chiplet is designed to sit in the package beside an accelerator, speaking the UCIe die-to-die interface on one side and emitting light on the other. Its SuperNova laser source supplies many wavelengths from a separate module. The company's investor list is a signal of strategic interest: its $500 million Series E, closed in March 2026 and led by Neuberger Berman, included AMD, MediaTek and Nvidia as strategic investors, and the company expanded the round to $650 million in September 2026. In 2026 Ayar Labs also joined Nvidia's NVLink Fusion ecosystem, the program through which Nvidia opens its scale-up fabric to third-party chips, and announced a partnership with the server maker Wiwynn to build optically connected rack-scale systems. Earlier, in late 2025, it formed a partnership with the ASIC design house GUC to integrate co-packaged optics into custom chip designs. Lightmatter, founded in 2017 and originally focused on photonic computing, has repositioned around interconnect. Its Passage M1000 photonic interposer, described in Chapter 4, offers 114 terabits per second of optical bandwidth from a single large platform. Alongside it, Lightmatter announced a co-packaged optical chiplet family, the L200, for designers who want optical I/O without adopting the full interposer. The interposer bet is that the edge of the die should stop being the interface to the outside world; if it succeeds, accelerators could emit far more bandwidth than any edge-based approach allows. Celestial AI took a different route, emphasising not just accelerator-to-accelerator links but optical connections between compute and memory. Its Photonic Fabric technology aimed to let accelerators reach large pools of memory optically, decoupling memory capacity from what fits on a package. In December 2025 Marvell agreed to acquire the company for $3.25 billion upfront, in cash and shares, plus up to $2.25 billion more contingent on revenue milestones. Marvell described the technology as delivering 16 terabits per second in a single chiplet and more than twice the power efficiency of copper interconnects, and said it expected meaningful revenue to begin in the second half of its fiscal 2028, building to a billion-dollar annualised run rate by the end of fiscal 2029. A company paying several billion dollars for a technology whose revenue is years away is a strong statement about where the industry believes the next battleground lies. Intel, meanwhile, demonstrated its optical compute interconnect chiplet in 2024, co-packaged with a CPU, carrying 4 terabits per second bidirectionally at about 5 picojoules per bit over distances up to 100 metres. Its distinguishing feature is on-chip lasers. Other large chipmakers have also moved: AMD acquired the silicon photonics design startup Enosemi in 2025, signalling that it intends to have in-house photonics expertise for future accelerators. An important channel to market for these technologies runs through custom silicon. The largest cloud operators increasingly design their own AI accelerators: Google's TPUs, Amazon's Trainium, Microsoft's Maia and Meta's MTIA among them. These chips are usually designed in partnership with ASIC design-service companies such as Broadcom, Marvell, Alchip and GUC, which handle physical design, packaging and the supply of interface intellectual property. For an optical I/O supplier, winning a place in the design-service partner's catalogue can mean access to several hyperscalers' chips at once. That logic helps explain Ayar Labs' partnership with GUC, and it is part of the strategic rationale for Marvell's acquisition of Celestial AI: Marvell's custom-silicon business gains an optical interconnect it can offer to its hyperscale customers as part of a complete accelerator design. Nvidia's NVLink Fusion programme, which lets other companies' chips attach to Nvidia's scale-up fabric, adds a further route by which optical chiplets can reach systems built around Nvidia hardware. For the accelerator vendors themselves, the choice of whether to build or buy optical I/O is strategic. Building in-house offers control over a technology that may define future system architectures, at the cost of developing expertise in an unfamiliar discipline. Buying chiplets from specialists is faster but creates dependence on a small supplier for a critical component. AMD's acquisition of Enosemi and its strategic investment in Ayar Labs suggest a hedged approach, building internal capability while maintaining access to an external supplier. Most large chip companies appear to be pursuing some variant of that strategy. Memory at a distance Celestial AI's focus on memory points to a second opportunity beyond scale-up networking. Today's accelerators are often limited not by compute but by memory capacity and bandwidth. High-bandwidth memory provides enormous bandwidth to the processor, but the amount that fits on a package is limited by package area and by how many stacks can be placed along the edge of the compute die. Large models, and especially long inference contexts whose key-value caches grow with every token, push against that limit. Optical links could let accelerators reach memory that is not on their own package. A pool of memory, built from high-bandwidth memory or conventional DRAM, could sit in a separate chassis, shared among many accelerators and connected by optical links with bandwidths approaching those of local memory and latencies in the hundreds of nanoseconds rather than microseconds. The idea of disaggregated memory is not new; it has been pursued over electrical fabrics such as CXL. What optics adds is bandwidth and reach. Whether it can deliver enough bandwidth per watt and low enough latency to be useful for AI workloads, rather than just for cold capacity, is being tested now. A thought experiment makes the stakes of optical scale-up concrete. Suppose a designer wants a scale-up domain of 512 accelerators, each with scale-up bandwidth in the range of today's leading GPUs, about 14 terabits per second in each direction. With copper, the domain must fit within the reach of 200-gigabit lanes, meaning a few racks at most packed against each other, or a single enormous rack. The total power of 512 accelerators at around a kilowatt or more each, plus memory, networking and cooling, approaches a megawatt in a space of a few square metres. That is beyond what most facilities can deliver or cool in so small a footprint. With co-packaged optical I/O on each accelerator, the same 512 accelerators could be spread across a dozen or more racks of ordinary density, connected through optical switches or directly through fibre. The added latency of tens of metres of fibre is a few hundred nanoseconds, small compared with the microseconds that collective communication operations typically take. The optical I/O power, at 5 picojoules per bit and 14 terabits per second per direction, would be roughly 70 watts per accelerator, or about 36 kilowatts across the domain. That is not trivial, but it buys a domain that could not otherwise be built at all, and it falls in proportion as optical energy per bit falls. The arithmetic shows why optics at the accelerator is less about saving power on an existing link than about enabling a system that copper forbids. Software must change to exploit such domains. Today's parallelism strategies, which decide how a model is divided among accelerators, are tuned to a world in which a small group of GPUs is tightly connected and larger groups communicate through a slower scale-out network. A much larger, uniformly fast domain changes the optimum. Models could be split more finely, more experts could be placed across more chips in mixture-of-experts architectures, and inference could draw on memory distributed across many accelerators. These changes take time to develop and validate, and the hardware will need to be available in quantity before software teams invest in them. Optics at the accelerator is therefore a co-evolution of hardware and software, not a drop-in replacement. The obstacles at the accelerator The barriers to optics at the accelerator are those of Chapter 5, intensified. An accelerator package dissipates more than a thousand watts in its latest generations and changes temperature rapidly with workload, making it a far harsher thermal environment than a switch. Package area is contested by compute dies and memory stacks, so every square millimetre given to optical engines must earn its place. The value of the processor is high, so the yield penalty of a defective optical component is severe. And the accelerator market is concentrated in a few vendors, each of which must decide whether to build optics in-house, buy chiplets, or wait. There is also a question of timing relative to copper. Copper scale-up has not stopped improving. Nvidia's choice of copper for NVLink in the Blackwell and announced Rubin generations shows that, for the next few years at least, the leading vendor believes copper within a rack remains the better trade-off for its core scale-up fabric. Co-packaged optics at the accelerator must therefore beat a moving target. Most observers expect the crossover to arrive when lane rates reach 400 gigabits per second, when the reach of passive copper shrinks below what a rack requires, or when the desire for scale-up domains spanning several racks becomes overwhelming. The startups' funding, the strategic investments and Marvell's acquisition all suggest that the industry expects that crossover within this decade. Google offers a reminder that optics in AI fabrics can take other forms. Its TPU v4 supercomputers, described in a 2023 paper, used optical circuit switches built from tiny movable mirrors to reconfigure the connections among thousands of TPU chips. Those switches redirect light between fibres rather than converting it, and they connect conventional transceivers rather than co-packaged ones. But they demonstrate that a hyperscaler is willing to build its AI fabric around optics when the system-level gains justify it, and they point to a future in which co-packaged optical I/O on accelerators feeds directly into optical switching layers, avoiding electrical conversion for long stretches of the path. The accelerator is where the stakes are highest and the engineering hardest. Whether optics arrives there in force will depend not only on technology but on economics, supply chains and the practical realities of operating data centres. Those are the subject of the next chapter. Chapter 8: Who Pays, Who Builds, Who Fixes A technology can be physically superior and still lose. The history of computing is littered with better designs that failed because they did not fit the way the industry made, sold, bought or maintained things. Co-packaged optics changes all four. It moves value from one set of companies to another, concentrates manufacturing in new places, alters how data centres are operated, and demands standards that did not exist a few years ago. This chapter examines those forces, because they will determine the pace of adoption as much as any picojoule. A shift in where the money goes The pluggable optics market is large, competitive and fragmented. Switch vendors sell boxes with empty cages. Operators buy modules separately, often from several suppliers, qualify them, and play suppliers against each other on price. Module makers, including Chinese firms such as Innolight and Eoptolink, American firms such as Coherent and Lumentum, and contract manufacturers such as Fabrinet in Thailand, compete fiercely. The AI boom has made this market highly profitable, and its suppliers have expanded rapidly to meet demand for 800-gigabit and 1.6-terabit modules. Co-packaging folds the optics into the switch or accelerator package. The operator no longer buys modules; it buys a switch with optics already inside. Value that was captured by module makers moves toward the chip vendor and its packaging partners: Broadcom, Nvidia, TSMC, and the suppliers who make engines, lasers, fibre assemblies and connectors for them. Module makers do not vanish. Many are repositioning as suppliers of external laser modules, optical engines, fibre assemblies or assembly services within co-packaged supply chains, and Nvidia's partner list includes several. But their bargaining position changes. A module maker that once sold directly to hyperscalers in a multi-vendor market now sells components to one of a few chip vendors, in a design the chip vendor controls. Operators feel this shift too. Buying pluggables separately gave them leverage: a single switch platform could accept modules from many suppliers, keeping prices competitive and supply resilient. Co-packaging bundles optics with silicon, so the operator's choice of switch largely determines its choice of optics. Hyperscalers worry about this kind of lock-in, and their engagement with co-packaging has been accompanied by pressure for standards and for multiple sources of each component. External laser modules, which remain pluggable and are being standardised, preserve some of the old flexibility; the optical engines themselves, at least for now, do not. Co-packaged optics depends on a supply chain that is in some respects deep and in others thin. The photonic integrated circuits come from a handful of foundries with silicon photonics processes, of which TSMC, GlobalFoundries and Tower Semiconductor are the most prominent. TSMC's entry has been especially consequential, because it brings the same advanced packaging and stacking technologies that dominate AI accelerators. That concentration has benefits, since the photonic engines can be co-designed with the packaging of the chips they serve, and risks, since TSMC's advanced packaging capacity has been one of the tightest constraints on AI hardware supply. If co-packaged optics competes with accelerators for the same scarce packaging lines, it inherits their bottleneck. Lasers come from compound semiconductor fabs that grow indium phosphide, a smaller, more specialised industry than silicon. Demand for lasers has surged with the transceiver boom, and continuous-wave lasers for external laser sources must meet demanding power and reliability specifications. Fibre, connectors and fibre-management assemblies come from companies such as Corning, SENKO and many smaller specialists, and a single co-packaged switch can need hundreds or thousands of fibres terminated with precision. Testing requires optical probe stations and automated test equipment, a market that grew up alongside silicon photonics and is still scaling. Geography matters. A large share of today's transceiver assembly is in China and Southeast Asia. Co-packaging shifts the critical steps toward the chip and packaging supply chains of Taiwan and, increasingly, the United States and elsewhere, as governments subsidise domestic advanced packaging. That shift may suit policymakers concerned about supply-chain security, but it also means the industry is building new capacity in new places, which takes time and money. Serviceability and the habits of operators Data centre operators have spent two decades building processes around replaceable optics. A failed pluggable module is diagnosed remotely, a technician is dispatched with a spare, the module is swapped in minutes, often without powering down the switch, and the failed module is returned to the supplier. Inventory systems track modules by serial number, failure analyses feed back into supplier qualification, and service-level agreements are written around these timelines. Co-packaging disrupts this. A failed optical engine cannot be swapped by a technician on the data centre floor. The repair unit becomes the switch or its main board, which must be removed from the rack, replaced with a spare, and returned for repair or scrap. Operators must stock spare switches rather than spare modules, plan for longer repair procedures, and design their networks to tolerate a whole switch being offline. For accelerators, the equivalent would be replacing a server tray or compute board, a still more expensive operation. Three arguments soften this disruption. First, as Chapter 6 explained, co-packaging is expected to reduce the frequency of failures, particularly the intermittent ones that cause most operational pain, so the operator trades many small repairs for fewer larger ones. Second, the most failure-prone component, the laser, remains replaceable in external modules, as do detachable fibre connectors. Third, AI clusters already operate in a world of large replacement units: a liquid-cooled GPU tray is serviced as a unit, not chip by chip, and operators of such systems are accustomed to treating failures at the level of the board or the tray. Still, the habits run deep, and the burden of proof falls on the new technology. Operators will want years of reliability data before they abandon pluggables in their core networks. That is why the first deployments have concentrated on AI back-end networks, where the power savings are largest, the hardware is replaced on short cycles anyway, and the operators are a small number of sophisticated hyperscalers and specialised cloud providers. There is a human dimension as well. The technicians who maintain data centre networks are skilled at handling pluggable modules, cleaning connectors and tracing cables. Co-packaged systems ask them to handle switches with hundreds of internal fibres, external laser modules carrying invisible high-power light, and diagnostic data about ring tuning and laser health that did not exist before. Operators deploying co-packaged equipment must invest in training, tooling and procedures, and vendors must design their systems to be serviceable by people who are not photonics engineers. The vendors that make co-packaged switches as easy to operate as conventional ones, with clear telemetry, simple replacement procedures and tolerance of imperfect handling, will win adoption faster than those that ship the most impressive specifications. Standards and interoperability Pluggable optics thrived because of standards. Multi-source agreements defined the mechanical form, electrical interface and management protocols of each module type, so any compliant module could plug into any compliant switch. Co-packaged optics has no equivalent tradition, and building one is a work in progress. Several bodies are involved. The Optical Internetworking Forum has published work on co-packaged optics, including a framework for co-packaged modules and the implementation agreement for external laser source modules mentioned in Chapter 5. The UCIe consortium governs the die-to-die interface that optical chiplets like TeraPHY use to talk to processors. The Open Compute Project, through which hyperscalers share hardware designs, has hosted co-packaged optics work streams. The UALink and Ultra Ethernet consortia define the protocols that co-packaged links will carry for scale-up and scale-out traffic. What is still missing is a fully interchangeable ecosystem at the level of optical engines, in which an engine from one supplier could be dropped into a package designed around another's. Given how tightly engines are co-designed with switch chips and packaging, that may never be practical. More realistic goals are standard external laser modules, standard fibre connectors at the package and the panel, standard management interfaces so that operators can monitor and diagnose co-packaged links with familiar tools, and interoperability at the optical layer, so that a co-packaged port on one vendor's switch can talk to a pluggable or co-packaged port on another's. That last point is crucial for gradual adoption: a network must be able to mix co-packaged and pluggable equipment during the long transition. The economics of the transition Operators ultimately decide on total cost of ownership: the price of equipment plus the cost of power, cooling, space and operations over its life. Co-packaging has a clear power advantage, which, as Chapter 3 showed, can amount to megawatts per large cluster. It frees front-panel and rack space. It may reduce operational disruption from link flaps. Against these, early co-packaged products carry the costs of new manufacturing processes, lower volumes than pluggables, and the operational changes described above. The balance depends heavily on the price of power, or rather on its availability. Through the mid-2020s, the binding constraint on AI data centre growth has increasingly been access to electricity and grid connections rather than capital. When power is scarce, a watt saved in the network is a watt available for compute, and its value is not the price of electricity but the revenue that the extra compute can generate. That reframing favours co-packaging strongly: the savings are worth far more to an operator who cannot get more power than to one paying a utility bill. The power saving also compounds through the facility. Every watt drawn by networking equipment must be delivered through transformers, switchgear and power distribution units, backed up by batteries and generators, and removed again as heat by chillers, pumps and cooling towers. Facility designers size all of that infrastructure for peak load. A reduction of several megawatts in network power is therefore not only a reduction in the electricity bill but a reduction in the capital equipment needed to deliver and remove that power, or, more realistically in a supply-constrained market, an increase in the compute that the same building can host. When a large AI data centre costs billions of dollars to build, freeing a few per cent of its power envelope for revenue-generating accelerators can be worth more than the entire cost of the network's optics. Operators who frame the comparison this way, as a question of how much compute a fixed power allocation can support, tend to find the case for co-packaging much stronger than those who compare module prices. Cost per bit should fall as volumes grow. Silicon photonics benefits from the economies of semiconductor manufacturing, and co-packaging eliminates the separate housings, DSPs and connectors of pluggable modules. But the learning curve takes time to descend, and pluggable modules are not standing still: linear pluggables, cheaper DSPs and competitive pressure continue to reduce their cost. The most likely pattern is the one seen in many infrastructure transitions: co-packaging first in the applications where its advantages are largest, then spreading as costs fall and confidence grows, while pluggables persist for years in applications where flexibility matters more than power. It is useful to sketch three plausible paths for the rest of the decade, not as forecasts but as a way of seeing which factors matter. In the first, the steady path, co-packaged switches take a growing share of AI back-end networks from 2026 onward, reaching a large fraction of new AI cluster deployments by around 2028, while pluggables, increasingly linear, remain dominant in general-purpose cloud networks. Optical I/O at accelerators begins to appear in specialised systems late in the decade. This path depends on reliability data continuing to be favourable and on manufacturing costs falling with volume. In the second, the accelerated path, a reach crisis at 400-gigabit lanes forces optics into scale-up fabrics sooner than expected. Copper cannot span a rack at that rate without heavy active electronics, and the leading accelerator vendors adopt co-packaged optical I/O for their scale-up domains, perhaps alongside copper for the shortest links. In this world, demand for photonic engines, lasers and fibre assemblies surges, and the supply chain becomes the bottleneck. Companies with secured capacity at photonic foundries and laser fabs gain advantage. In the third, the delayed path, a reliability problem surfaces in large-scale deployments, or pluggable technology improves faster than expected, and operators slow their adoption of co-packaging. Linear pluggables and active copper absorb more of the demand, and co-packaging remains a niche for the largest AI clusters for longer. Even in this world, the physics of rising lane rates eventually pushes the industry toward optics near the chip, but the timing shifts by a product generation or two. Which path unfolds will depend on facts not yet known: field failure rates at hyperscale, the cost curves of photonic engines and lasers, the progress of copper at 400 gigabits per lane, and the decisions of a handful of accelerator vendors. What is notable is that in none of these paths does the long-term direction change. The disagreement is over timing and scope, not over destination. The shift creates winners and losers, although the boundary will be blurred by repositioning. Chip vendors that control both silicon and optics, Nvidia and Broadcom foremost, capture more of the value of each port. Foundries with photonic and advanced packaging capabilities, TSMC above all, gain a new product line closely tied to their most profitable business. Laser makers gain demand for high-power continuous-wave sources. Fibre and connector makers gain volume and complexity. Startups with credible optical I/O technology for accelerators have attracted large investments and, in Celestial AI's case, a multi-billion-dollar acquisition. Module makers face the hardest adjustment, though the transition will be gradual and their pluggable businesses remain large. Operators gain power savings and potentially reliability, at the price of some flexibility and supplier leverage. The balance of these forces will play out over the rest of the decade, product generation by product generation. The concluding chapter asks what, given everything examined here, those who plan and build data centres should take from it. Conclusion The story of co-packaged optics is usually told as a story about light replacing copper. That is not quite right. Copper is not going away. It will continue to carry signals across packages, boards and racks wherever distances are short enough, and for the most tightly coupled tier of AI systems it remains, for now, the leading vendor's choice. What is changing is the place where electricity becomes light. For decades that place was the front panel, a convenient location for humans and an inconvenient one for physics. The argument of this book has been that co-packaging wins by moving the conversion to where the physics wants it, a few millimetres from the chip, and that both its gains and its difficulties follow from that move. Several consequences follow from taking that argument seriously. Distance, not speed, is the design variable The first is that anyone designing or buying AI infrastructure should think about interconnect in terms of electrical distance. The questions that matter are how far each signal must travel as electricity before it becomes light, how much energy is spent on that journey, and how much circuitry is needed to repair the damage. Framed that way, the choice between pluggable modules, linear pluggables, near-package optics, co-packaged optics and optical chiplets becomes a choice about how much electrical distance one is willing to pay for in exchange for serviceability and supply flexibility. There is no universally correct answer, but there is a clear direction of travel: as lane rates rise, every centimetre of copper grows more expensive, and the optimum moves closer to the chip. That direction is unlikely to reverse. Each doubling of the electrical lane rate, from 200 to 400 gigabits per second and beyond, shortens the reach of copper and raises the energy needed to cross it. Improvements in SerDes design, board materials and cables will continue, but they are fighting physics that grows steeper with every generation. Optics, by contrast, carries signals across a building with losses that do not depend on the data rate. The economic logic of the transition is built into the physics of the materials. The hard problems are now industrial, not scientific The second consequence is that the obstacles that remain are mainly industrial. The science of silicon photonics, modulators, detectors, waveguides and wavelength multiplexing, is mature. The difficult work lies in manufacturing: coupling fibres at high yield, testing optical components at wafer scale, managing heat around temperature-sensitive rings, and supplying reliable lasers in volume. These problems yield to investment, volume and experience rather than to breakthroughs. That is good news for predictability. It suggests that co-packaged optics will follow the familiar pattern of semiconductor technologies, in which costs fall and reliability improves steadily with cumulative production, rather than waiting on discoveries that may never come. It also means that the companies best placed to succeed are those with access to high-volume manufacturing and packaging capacity. The participation of TSMC, the largest foundry, and the commitments of Nvidia, Broadcom and Marvell signal that co-packaged optics has moved from the research agenda of a few laboratories into the product roadmaps of the companies that set the pace of the semiconductor industry. The real prize is architectural freedom The third consequence concerns where the largest benefit lies. The energy savings of co-packaged switches are real and valuable, and they will matter more as power becomes the binding constraint on data centre growth. But the more profound effect may come when optics reaches the accelerator. Today's AI systems are shaped by the reach of copper: scale-up domains are confined to a rack, so racks are packed to extraordinary densities of power and heat. Optical input and output at the accelerator would break that constraint, allowing tightly coupled domains to span several racks, memory to be pooled and shared across many processors, and the physical layout of a data centre to follow the needs of the workload rather than the length of a cable. That freedom will not arrive all at once, and it will not be free. It requires optical engines that survive beside thousand-watt processors, packaging that accommodates both memory and optics, and software that exploits larger and more flexible domains. But the investments now being made, by startups, by chipmakers and by an acquirer willing to pay billions for revenue years away, indicate that the industry sees the accelerator as the destination and the switch as the proving ground. It is also worth naming what co-packaged optics will not do. It will not make communication free, and it will not abolish the hierarchy of distances that shapes every computer, from registers to caches to memory to network. Moving a bit across a data centre will remain far more expensive than moving it across a chip. What co-packaging changes is the slope of that hierarchy beyond the package: the penalty for going off-package shrinks, and the penalty for going a long distance once a signal is already optical becomes very small. Architects will respond, as they always have, by rearranging their systems to exploit the new costs. The most interesting AI systems of the next decade may be those designed from the beginning around cheap, abundant optical bandwidth at the package edge, rather than those that simply swap one kind of cable for another. What operators and planners should do For those who operate data centres, the practical implications are concrete. They should evaluate co-packaged switches first where the case is strongest: in AI back-end networks, where power is scarce, utilisation is high, link flaps are costly and hardware is refreshed quickly. They should demand reliability data, not vendor claims, and share their own data with the industry, as the earliest adopters have begun to do. They should adapt their sparing and repair processes to larger replacement units, and insist on standard external laser modules, standard connectors and standard management interfaces to preserve as much flexibility and supplier competition as possible. And they should plan facilities on the assumption that interconnect power per bit will fall substantially over the coming product generations, which changes the trade-off between power devoted to compute and power devoted to moving data. For chip and system designers, the implication is that optical input and output must become a first-class element of architecture, designed in from the start rather than attached at the end. The packages of the next decade will be mixed electronic and photonic assemblies, and the teams that design them will need to understand lasers, fibres and thermal tuning as well as transistors and memory. For policymakers and investors, the lesson is that the supply chain for AI infrastructure is extending into new territory: photonic foundry processes, indium phosphide laser production, precision fibre assembly and optical test. These are smaller industries than logic chipmaking, and capacity in them can become a bottleneck. Attention to them now will matter later. The deeper point is the one with which this book began. In large computers, moving a bit has become more expensive than computing with it. Every advance in computing capacity is wasted if the data cannot reach the processors fast enough and cheaply enough. Co-packaged optics is the most serious attempt yet to rebalance that equation, by placing light where it can do the most good. Whether it succeeds broadly, or remains confined to the largest AI clusters, will be decided over the next few product generations. But the direction is set by physics, and physics is patient. Notes The following sources underpin the specific figures and claims in the text. Vendor figures are reported as the vendors stated them. Nvidia, press release, "NVIDIA Announces Spectrum-X Photonics, Co-Packaged Optics Networking Switches to Scale AI Factories to Millions of GPUs," 18 March 2025. Source of port configurations, availability timing, and the claims of 3.5 times power efficiency, 63 times signal integrity, ten times resiliency, four times fewer lasers and 1.3 times faster deployment, and of the partner list. Nvidia Technical Blog, "How Industry Collaboration Fosters NVIDIA Co-Packaged Optics," 2025. Source of the micro-ring modulator design at 200 Gb/s PAM4 per wavelength, the three COUPE engines per Quantum-X optical subassembly at 1.6 Tb/s each, the 32 engines at 3.2 Tb/s in Spectrum-X Photonics, the 115.2 Tb/s Quantum-X total, and the external laser source with eight lasers serving 32 of 576 transmit lanes. Intel, announcement of its optical compute interconnect chiplet, 2024. Source of the 4 Tb/s bidirectional figure, 64 channels at 32 Gb/s per direction, eight wavelengths per fibre on eight fibre pairs, about 5 pJ/bit against roughly 15 pJ/bit for pluggables, reach up to 100 metres, and the statement of more than 32 million on-chip lasers shipped at below 0.1 FIT. Lightmatter, press release announcing the Passage M1000 and L200, 31 March 2025. Source of the 114 Tb/s total optical bandwidth, area above 4,000 square millimetres, 256 fibres, eight wavelengths per fibre, and the remark by Nick Harris on shoreline. Marvell Technology, press release on the agreement to acquire Celestial AI, 2 December 2025. Source of deal terms ($3.25 billion upfront, up to $2.25 billion in earn-out), the 16 Tb/s chiplet and power-efficiency claims, and revenue expectations. Ayar Labs, company news releases, 2025–2026, and TeraPHY product specifications. Source of the Series E and its expansion, strategic investors, NVLink Fusion and Wiwynn announcements, GUC partnership, and TeraPHY figures (eight ports at 512 Gb/s, sixteen wavelengths per port, UCIe interface, about 10 ns latency per chiplet). Overview and review literature on co-packaged optics, 2024, as summarised in standard reference sources. Source of the 15–20 pJ/bit range for pluggable modules and 5–10 pJ/bit for co-packaged designs, the Broadcom product timeline (25.6 Tb/s in 2022, 51.2 Tb/s Bailly in 2024), and TrendForce's expectation of widespread adoption in 2027–2028. R. A. Soref and B. R. Bennett, "Electrooptical effects in silicon," IEEE Journal of Quantum Electronics 23, no. 1 (1987): 123–129. The foundational account of the plasma dispersion effect. A. Liu et al., "A high-speed silicon optical modulator based on a metal–oxide–semiconductor capacitor," Nature 427 (2004): 615–618. Intel's gigahertz-class silicon modulator. Q. Xu, B. Schmidt, S. Pradhan and M. Lipson, "Micrometre-scale silicon electro-optic modulator," Nature 435 (2005): 325–327. The early compact micro-ring modulator. A. W. Fang et al., "Electrically pumped hybrid AlGaInAs-silicon evanescent laser," Optics Express 14, no. 20 (2006): 9203–9210. The Intel and UCSB hybrid silicon laser. D. A. B. Miller, "Attojoule Optoelectronics for Low-Energy Information Processing and Communications," Journal of Lightwave Technology 35, no. 3 (2017): 346–396. The argument about fundamental energy limits of optical interconnects. N. P. Jouppi et al., "TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings," Proceedings of the 50th Annual International Symposium on Computer Architecture (ISCA 2023). Google's use of optical circuit switches. Nvidia, GTC 2024 presentation of the GB200 NVL72 system, describing an NVLink spine of more than 5,000 copper cables, roughly two miles in total, and 1.8 TB/s of NVLink bandwidth per GPU. Glossary Co-packaged optics (CPO). Optical engines mounted on the same package substrate as a switch chip or processor, a few millimetres from the die, instead of in pluggable modules at the front panel. DSP (digital signal processor). In a pluggable module, the chip that digitises, equalises and regenerates the electrical signal arriving from the host. It accounts for a large share of module power. External laser source (ELS). A separate, usually replaceable module that supplies continuous laser light through fibre to optical engines on a package. ELSFP is the OIF-defined pluggable form of it. FIT. Failures in time: one failure per billion device-hours, a standard reliability measure. Link flap. A brief loss and recovery of a network connection, damaging to synchronised AI training jobs. Mach-Zehnder modulator. An interferometric modulator that splits light into two arms and recombines it; robust to temperature but millimetres long. Micro-ring modulator. A ring-shaped waveguide resonator whose resonance is shifted electrically to modulate light; very compact and efficient but temperature-sensitive. PAM4. Pulse-amplitude modulation with four levels, carrying two bits per symbol. Picojoule per bit (pJ/bit). Energy spent to move one bit; a link at 10 pJ/bit running at 1 Tb/s uses 10 watts. Scale-up and scale-out. Scale-up networks tightly couple accelerators so they act almost as one machine (for example NVLink); scale-out networks connect servers and racks across a data centre. SerDes. Serialiser/deserialiser: the circuit that converts a chip's parallel internal data into fast serial lanes and back. Wavelength-division multiplexing (WDM). Carrying several independent signals on different wavelengths of light in a single waveguide or fibre. Further Reading Lukas Chrostowski and Michael Hochberg, Silicon Photonics Design: From Devices to Systems (Cambridge University Press, 2015).** The standard practical textbook on designing silicon photonic devices, from waveguides and ring resonators to modulators and detectors. Technical but clear. Graham T. Reed and Andrew P. Knights, Silicon Photonics: An Introduction (Wiley, 2004).** An accessible introduction to the physics of light in silicon, written as the field was emerging. Good on fundamentals such as the plasma dispersion effect. D. A. B. Miller, "Attojoule Optoelectronics for Low-Energy Information Processing and Communications," Journal of Lightwave Technology (2017).** A long, lucid review of the energy limits of optical interconnects and of why the electronics, not the photons, set the bill. R. A. Soref and B. R. Bennett, "Electrooptical effects in silicon," IEEE Journal of Quantum Electronics (1987).** The paper behind nearly every silicon modulator in use today. N. P. Jouppi et al., "TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings," ISCA 2023. A rare, detailed public account of how a hyperscaler built optics into the fabric of an AI supercomputer. Nvidia Technical Blog, "How Industry Collaboration Fosters NVIDIA Co-Packaged Optics" (2025). A readable vendor account of the micro-ring engines, COUPE stacking and external laser design in Nvidia's photonics switches. Optical Internetworking Forum, co-packaged optics publications, including the External Laser Small Form Factor Pluggable implementation agreement. The emerging standards that define how co-packaged systems are supplied with light and integrated. Hashtags: #SiliconPhotonics #SiliconPhotonicsCoPackaging #CoPackagedOptics #CPO #OpticalInterconnects #OpticalComputing #PhotonicIntegratedCircuits #PICTechnology #IntegratedPhotonics #DataCenterNetworking #DataCenterInnovation #DataCenterInfrastructure #AIDataCenters #AIInfrastructure #AITrainingClusters #HighPerformanceComputing #HPCNetworking #OpticalNetworking #FiberOpticCommunications #OpticalDataTransmission #HighSpeedInterconnects #HighBandwidthNetworking #DataCenterBottlenecks #BandwidthOptimization #LowLatencyNetworking #EnergyEfficientComputing #EnergyPerBit #PowerEfficiency #OpticalTransceivers #PluggableOptics #LinearPluggableOptics #NearPackageOptics #OnBoardOptics #OpticalIO #OpticalIOChiplets #PhotonicChiplets #ChipletIntegration #AdvancedSemiconductorPackaging #HeterogeneousIntegration #ThreeDimensionalIntegration #HybridBonding #PhotonicInterposers #SiliconWaveguides #OpticalWaveguides #MachZehnderModulators #MicroringModulators #GermaniumPhotodetectors #LaserTechnology #ExternalLaserSources #IntegratedLasers #IndiumPhosphideLasers #WavelengthDivisionMultiplexing #WDM #OpticalMultiplexing #PAM4Signaling #SerDesTechnology #DigitalSignalProcessing #SignalIntegrity #OpticalLinkBudget #FiberCoupling #OpticalAlignment #ThermalManagement #LaserReliability #PhotonicTesting #WaferScaleTesting #OpticalPackaging #OpticalEngine #SiliconOnInsulator #CMOSPhotonics #TSMCCOUPE #IntelOpticalInterconnect #AyarLabs #TeraPHY #LightmatterPassage #BroadcomTomahawk #NvidiaQuantumX #NvidiaSpectrumX #InfiniBand #EthernetSwitching #ScaleUpNetworking #ScaleOutNetworking #GPUInterconnects #AIAcceleratorNetworking #HyperscaleDataCenters #OpticalCommunicationStandards #UCIe #PhotonicManufacturing #NextGenerationInterconnects
- Algorithmic Warfare (The Ethics and Engineering of Lethal Autonomous Weapons)
Download the Book (PDF): Introduction In March 2021 a panel of experts reporting to the United Nations Security Council on the arms embargo in Libya published a long, dry document of the kind that rarely reaches a general audience. Buried in its account of the fighting around Tripoli the previous year was a paragraph describing how retreating forces loyal to Khalifa Haftar had been "hunted down and remotely engaged" by armed drones and by "lethal autonomous weapons systems such as the STM Kargu-2" and other loitering munitions. The panel added that these systems "were programmed to attack targets without requiring data connectivity between the operator and the munition: in effect, a true 'fire, forget and find' capability." Within weeks the paragraph had been read around the world as the first documented killing of human beings by a robot acting on its own. That reading went further than the evidence. The panel did not say that anyone had died as a result of an autonomous engagement, did not explain how it knew which mode the munitions were operating in, and did not establish what the weapons were actually instructed to look for. The Turkish manufacturer has long presented the Kargu-2 as a system in which a human operator selects the target. What the episode really showed was something less cinematic and more important: that a weapon capable of finding and attacking a target without a live human link could be fielded in an ordinary civil war, by an ordinary customer, and that the outside world would have no reliable way of knowing what it had done. That gap between capability and knowledge, between what a weapon can do and what anyone can verify it did, runs through every chapter that follows. The question behind the headlines Public debate about "killer robots" tends to swing between two images. One is the humanoid machine of science fiction, deciding of its own volition to turn on its makers. The other is the reassuring official formula that a human will always remain "in the loop". Neither is much help. No military is building sentient machines, and no serious analyst thinks the risk lies there. And the presence of a human somewhere in a decision chain tells you very little about whether that human is exercising judgment or merely ratifying the output of a system they cannot interrogate, at a tempo they cannot control, about targets they will never see. The actual developments are more prosaic and, for that reason, harder to resist. Sensors have become cheap. Machine-learning classifiers that would have been research curiosities fifteen years ago now run on chips small enough to fit in a hobbyist drone. Electronic warfare in Ukraine has made the radio link between a drone and its pilot the weakest point of the system, giving both sides a strong incentive to let the weapon complete its final approach without one. In Gaza, according to investigative reporting that the Israeli military disputes, software systems were used to generate target recommendations at a volume that would have been unthinkable a decade earlier. In early 2026 American newspapers reported that a commercial large language model had been used to help process intelligence and prioritise targets during United States strikes on Iran, at the same moment that the company that built the model was locked in a public confrontation with the Pentagon over its refusal to allow the technology to be used in fully autonomous weapons. None of these cases involves a machine that "decides to kill" in the sense the headlines suggest. All of them involve a steady transfer of the work of judgment, of perceiving, sorting, prioritising and timing, from people to software. That transfer is the subject of this book. The argument The controlling claim of what follows can be stated simply. The central danger of military autonomy is not that machines will take decisions away from humans in a single dramatic step. It is that human judgment over the use of lethal force will be compressed, by speed, by scale and by the brittleness of machine perception, until "human control" survives in name but not in substance. The governance problem is therefore not how to define a category of forbidden machine, but how to protect the conditions under which a human being can still understand, foresee and answer for what a weapon will do. This argument has consequences that cut against some familiar positions on both sides of the debate. It means that a weapon with a human formally "in the loop" can be more troubling than one that operates autonomously within tight limits of time, place and target type. It means that decision-support software, which most governments treat as outside the autonomous weapons conversation altogether, belongs at its centre. It means that technical improvements in accuracy, which defenders of autonomy rightly point to, do not answer the core objection, because the objection is not only about error rates. And it means that the diplomatic question about to be put to states at the Seventh Review Conference of the Convention on Certain Conventional Weapons in Geneva, in November 2026, is less about whether to "ban killer robots" than about whether states will accept binding limits on the conditions of their own control. How the book proceeds The first chapter sets out what autonomy in weapons actually means, and why the concept is older than the debate about it. Naval mines, homing torpedoes, ship-defence guns and radar-hunting loitering munitions have performed parts of the targeting process without real-time human direction for decades. Understanding why those systems were tolerated, and what has changed, is the first step to understanding the current controversy. Chapters 2 and 3 turn to the engineering, at the level a citizen or legislator needs and no further. Chapter 2 explains how machines come to "recognise" targets, and why that recognition is statistical, brittle and blind to the very features that make a person lawful or unlawful to attack. Chapter 3 examines swarming: the idea of many cheap systems coordinating through simple local rules, and why behaviour that emerges from such coordination is hard to predict, test or certify. Neither chapter describes how to build anything. Both describe how such systems fail, because the failures are what law and policy have to govern. Chapter 4 turns from the machine to the person supervising it. It draws on half a century of research into automation, and on two tragedies, the shooting down of Iran Air Flight 655 by USS Vincennes in 1988 and the Patriot missile fratricides of 2003, to show why a human "in the loop" is not a safeguard by default. Chapter 5 assembles the battlefield record so far, from Libya and Nagorno-Karabakh to Ukraine, Gaza and Iran, with careful attention to who has claimed what and on what evidence. Chapters 6 and 7 take up law and ethics. International humanitarian law already governs every weapon; the question is whether its rules of distinction, proportionality and precaution can be applied by, or through, a machine, and who answers when they are broken. The ethical arguments are not reducible to the legal ones, and some of the strongest objections to autonomous killing survive even the assumption that machines could become more accurate than soldiers. Chapter 8 follows the diplomacy, from the first informal meetings in Geneva in 2014 through the United Nations General Assembly resolutions of 2023, 2024 and 2025 to the rolling text negotiated in the Group of Governmental Experts in 2026. The conclusion argues for what should follow. A note on sources and limits Much of what can be said about military artificial intelligence rests on sources of uneven quality. Governments rarely disclose how their systems work, and when they do, they have reasons to emphasise safeguards. Manufacturers market capabilities they may not have. Investigative journalists depend on anonymous sources whose accounts cannot be independently checked. Advocacy groups, on every side, select their examples. This book tries to say at each point who is making a claim, what kind of evidence supports it, and what is contested. Where something could not be verified, it is left out or flagged as uncertain. There is one further limit, and it is deliberate. The technical chapters explain mechanisms conceptually, using the kind of material found in public research and official reports, because citizens cannot judge a governance debate about machines whose basic workings are a mystery to them. They do not go beyond that. The purpose is to make the risks legible, not to make the weapons better. The stakes justify the effort. Decisions about who may be killed in war have always been terrible, and have always been made imperfectly. What is new is the possibility that they will be made at a speed and scale, and through a chain of software, that leaves no one in a position to have actually decided. Whether that possibility becomes the norm is still, as of late 2026, a matter of political choice. Chapter 1: The Kill Chain and the Long History of Autonomy Military planners describe the use of force as a chain. The American version that became standard in the 1990s and 2000s ran find, fix, track, target, engage, assess: detect something of interest, establish where it is, follow it, decide that it is a lawful and worthwhile target, apply force to it, and then judge whether the effect was achieved. Other militaries use other words, but the structure is the same. Each link involves gathering information, interpreting it and acting on the interpretation. And each link can, in principle, be performed by a person, by a machine, or by some combination of the two. The debate about lethal autonomous weapons is, at bottom, a debate about which links of that chain may be handed to machines, under what conditions, and with what consequences for the human beings who remain attached to it. Seen this way, autonomy is not a property a weapon either has or lacks. It is a distribution of tasks. And the history of that distribution is much longer than the history of artificial intelligence. Machines that already choose Consider the oldest example. A naval contact mine, a technology in widespread use since the nineteenth century, "selects" its target in the crudest possible sense: it detonates against whatever touches it. An influence mine, which responds to the magnetic signature or the acoustic noise of a passing ship, is more discriminating. It is designed to react to the particular physical traces that large steel hulls produce and to ignore other things. The human decision that matters takes place when the minefield is laid: someone chooses where and when, and accepts in advance whatever the mine will do within that space and time. After that, the weapon acts alone. The same structure appears in a range of systems that most armed forces have operated for decades. Ship-defence guns such as the American Phalanx close-in weapon system can be set to detect, track and fire at incoming missiles and aircraft without a human pressing a button for each engagement, because an anti-ship missile arriving at close to the speed of sound leaves no time for one. The Aegis combat system that equips American and allied warships has modes in which it engages air threats automatically. Land-based air and missile defences, including Patriot and Israel's Iron Dome, can operate with a high degree of automation for the same reason. Torpedoes have homed on the sound of their targets since the Second World War. The Israeli Harpy, developed by Israel Aerospace Industries and exported to several countries, is in some ways the clearest precedent for current concerns. It is a loitering munition: launched toward an area, it flies over it searching for the electromagnetic emissions of enemy air-defence radars, and when it detects one that matches what it is looking for, it dives onto the emitter. Once launched, no human chooses which radar it attacks. The human decision is to release the weapon over a particular area for a particular period, against a particular class of target. The Stockholm International Peace Research Institute (SIPRI), in a 2017 survey of autonomy in weapon systems by Vincent Boulanin and Maaike Verbruggen, catalogued dozens of such systems in service and found that the large majority of those able to select and engage targets without human intervention were defensive, operated against materiel rather than people, and worked in relatively uncluttered environments such as the sky above a ship or the sea surface. A selection of these long-standing types is set out in Table 1. Table 1. Long-standing weapon types that can engage without a human decision per target. Type Example Target class Human role Operating context Influence mine Naval sea mines Ships by signature Decides where and when to lay Fixed area, long duration Ship self-defence gun Phalanx CIWS Incoming missiles, aircraft Activates mode, supervises Airspace around ship Air and missile defence Patriot, Aegis, Iron Dome Missiles, rockets, aircraft Sets mode, can veto Defended zone, minutes Anti-radar loitering munition IAI Harpy Radar emitters Chooses launch area and time Search area, hours Homing torpedo Various navies Vessels by acoustics Fires at a detected contact Short range, minutes Source: compiled from SIPRI, Mapping the Development of Autonomy in Weapon Systems (2017), and manufacturers' public descriptions. Two features of this older family of weapons matter for everything that follows. First, their target classes are defined by physical signatures that are relatively distinctive: the radar emission of a fire-control radar, the trajectory of an incoming missile, the magnetic mass of a warship. Second, the environments in which they act are bounded in space and time, and largely empty of civilians. The human who activates them can foresee, with reasonable confidence, what they will attack. The legal responsibility of that person is recognisable and the predictability of the effects is high. Neither condition is guaranteed for the systems now being developed. That is the first thing that has changed. What has actually changed Three developments, interacting, explain why a problem that militaries lived with quietly for half a century became a matter for the UN General Assembly. The first is the change in perception. Older systems could only recognise targets that announced themselves through a strong, distinctive signal. Machine learning, and in particular the deep neural networks that transformed computer vision after about 2012, made it possible for software to classify objects in ordinary camera images: to label a shape as a tank, a truck, a person or a building with a statistical confidence score. Chapter 2 explains how this works and why it is far less reliable than it appears. The essential point here is that machine perception has extended the range of possible targets from emitters and missiles to vehicles, structures and, most controversially, human beings. The second is the change in cost and scale. A Patriot battery costs hundreds of millions of dollars and is operated by trained crews within a doctrine developed over decades. A first-person-view drone of the kind used in enormous numbers in Ukraine costs a few hundred dollars and is assembled from commercial components. Ukrainian officials have spoken of producing drones in the millions per year. When weapons are that cheap, the constraint on their use stops being the number of weapons and becomes the number of people available to operate them. Every army facing that constraint has an incentive to let one operator supervise many systems, which in practice means letting each system do more on its own. Chapter 3 discusses the logic of swarming that follows. The third is the change in the electromagnetic environment. In Ukraine, jamming has made the radio link between drone and pilot unreliable over the final kilometres of an attack. One response has been to connect drones by fibre-optic cable. Another, widely reported by both Ukrainian manufacturers and Western analysts, has been to give drones the ability to lock onto a target the operator has designated and complete the final approach without the link. That is a modest form of autonomy, but it is a step along a gradient. Each increment makes the next increment easier to justify. A fourth change is less visible and arguably more consequential: the automation of targeting upstream of the weapon. Software that fuses sensor feeds, intercepts and databases to propose targets for human approval does not itself fire anything, and so it falls outside most official definitions of an autonomous weapon. But if it determines which targets humans consider, how quickly they must decide, and what information they see, it shapes the use of force as profoundly as any guidance system. Chapter 4 and Chapter 5 return to this. Why militaries want it It would be a mistake to treat the push toward autonomy as a matter of technological fascination. Armed forces pursue it for reasons that, from their own point of view, are compelling, and any serious governance proposal has to reckon with them. The first is speed. Modern military doctrine in the United States and elsewhere is preoccupied with shortening the time between detecting a target and striking it, often described as compressing the "sensor-to-shooter" timeline. An adversary who can find and strike faster can, in theory, act before the other side can respond. Against incoming missiles, rockets and drones, speed is not an advantage but a necessity: the time available for a decision may be seconds, and a human-paced decision may simply be too late. The second is mass. For a generation after the Cold War, Western militaries relied on small numbers of exquisite, expensive platforms. The war in Ukraine, and planning for a possible conflict with China in the Western Pacific, convinced many defence establishments that they would need far larger numbers of cheaper systems, and that they could not recruit enough people to operate them one by one. The third is resilience under jamming and disruption. Remotely operated systems depend on communications links that an adversary will try to cut. A system that can continue its task when the link fails is more useful than one that falls out of the sky or wanders aimlessly. The fourth is the protection of one's own personnel. A machine sent into a defended area instead of a pilot or an infantry squad means fewer dead and captured soldiers. Democratic governments, answerable to the families of those soldiers, feel this pressure acutely. Each of these motives is legitimate on its own terms. Taken together, they produce a steady gradient: every increment of autonomy offers an advantage, and the cost of each increment, in lost human judgment, is diffuse, delayed and borne mostly by people on the other side. That asymmetry between concentrated benefit and diffuse cost is what makes the problem political rather than technical. In the loop, on the loop, out of the loop The vocabulary that grew up to describe these distinctions is useful but treacherous. A system is said to have a human "in the loop" when it can only engage a target after a human authorises that specific engagement. A human is "on the loop" when the system can select and engage on its own but a supervisor can monitor and intervene to stop it. A system with a human "out of the loop" acts without the possibility of real-time intervention. The terms became current in the early 2010s. In November 2012 Human Rights Watch and the International Human Rights Clinic at Harvard Law School published Losing Humanity, which used them to frame a call for a pre-emptive ban on "fully autonomous weapons". In the same month the United States Department of Defense issued its first policy on autonomy in weapon systems, Directive 3000.09. The directive defined an autonomous weapon system as one that, "once activated, can select and engage targets without further intervention by a human operator" (the 2023 revision shortened this to "an operator"), and a semi-autonomous system as one intended only to engage targets or groups of targets that a human operator has selected. It was revised in January 2023, retaining that core definition and the requirement that such systems be designed to allow commanders and operators "to exercise appropriate levels of human judgment over the use of force". The International Committee of the Red Cross (ICRC), which has a special role as guardian of international humanitarian law, adopted a functional definition that has become the most widely used reference point in diplomacy. In its position of May 2021, it described autonomous weapon systems as those that "select and apply force to targets without human intervention": after initial activation by a person, the system itself triggers a strike in response to information from its environment received through sensors and on the basis of a generalised "target profile". The ICRC's point is that the user does not choose, and often does not even know, the specific target, the precise timing or the location of the resulting application of force. This functional approach has two great virtues. It focuses attention on what the ICRC and many states call the "critical functions" of selecting and engaging targets, rather than on navigation, take-off or other tasks where autonomy raises no particular concern. And it does not depend on how sophisticated the software is. A crude system that fires at anything with a certain heat signature is, on this definition, as autonomous as one using advanced machine learning, and may be more dangerous. The loop vocabulary, by contrast, has two weaknesses. It suggests that the presence of a human is a binary fact, when the quality of human involvement varies enormously. An operator who approves a target after twenty seconds of looking at a screen is formally "in the loop" in exactly the same way as one who has spent days studying intelligence about it. And "on the loop" supervision assumes that the supervisor can actually perceive and stop a mistake in time, which, as Chapter 4 shows, is often false. A human on the loop of a system operating at machine speed may have no more real control than a human who is out of it. Why definitions became a battlefield It might seem that defining the object of regulation is a technical preliminary. In the autonomous weapons debate it has been a central political struggle, because each definition favours a different outcome. Some states have favoured narrow definitions that capture only systems which do not yet exist. China's position paper submitted to the Convention on Certain Conventional Weapons in 2018 characterised lethal autonomous weapons by five features: lethality, autonomy, "impossibility for termination" once started, indiscriminate effect, and "evolution" through learning in ways that exceed human expectations. A weapon must possess all five to count. Since no state is likely to admit fielding a weapon that cannot be stopped and kills indiscriminately, such a definition makes any prohibition largely symbolic. China at the same time expressed support for a ban on the use, but not the development, of weapons so defined. Other actors, including many of the civil society organisations in the Stop Killer Robots coalition, have argued for broad, functional definitions that capture existing systems as well, in order to bring them under regulation. Their concern is that a narrow definition leaves the real drift of military practice entirely untouched. A third position, influential in the United States, the United Kingdom and several other militarily advanced states, has been that definitions matter less than the application of existing law: any weapon must comply with international humanitarian law, and the task is to ensure that commanders using autonomous functions do so lawfully. On this view a new treaty is unnecessary and might stifle beneficial technology, including systems that reduce civilian harm. The diplomacy described in Chapter 8 has partly worked around these disagreements by adopting a "two-tier" approach: prohibit certain autonomous weapon systems outright and regulate the rest. That approach does not require a single definition that settles everything. It requires a working characterisation broad enough to capture the systems that need regulating, and specific criteria identifying those that must be prohibited. It is no accident that the elements retained in the Group of Governmental Experts' report of September 2026 included what campaigners described as a workable characterisation of autonomous weapon systems: without one, nothing else could follow. Bounded and unbounded autonomy The history surveyed in this chapter suggests a distinction more useful than "in or out of the loop". Long-standing autonomous weapons were tolerated, and in most cases regarded as lawful, because their autonomy was bounded. Their target classes were narrow and physically distinctive. Their area and duration of operation were limited. They operated mainly where civilians were absent. And the human who activated them could form a reliable expectation of what they would do. The developments now under way loosen every one of those bounds. Machine perception widens the class of possible targets to objects and people that look like other objects and people. Cheapness and scale encourage long loitering times over wide areas. Electronic warfare pushes autonomy into the final moments of engagement, precisely where civilians and surrendering soldiers may appear. Decision-support systems move the human further from the point of effect even when no weapon is technically autonomous. The ICRC's 2021 recommendations, discussed further in Chapters 6 and 8, are best read as an attempt to restore those bounds by law: limits on the types of target, on the duration, geographical scope and scale of use, on the situations of use, and requirements for effective human supervision and timely intervention. The question for the rest of this book is whether such bounds can hold when the underlying technology, and the military incentives behind it, push steadily in the other direction. Answering it requires looking first, and closely, at how machines come to see. Chapter 2: How Machines See Targets When a soldier looks through a sight and decides that the figure in the distance is an enemy fighter, a great deal happens at once. The soldier recognises the shape of a person, notices what the person is carrying, how they are moving, where they are relative to friendly and enemy positions, what happened in this place an hour ago, and what the briefing said about who is likely to be here. The judgment that follows is not a single act of perception but an act of interpretation, drawing on context, memory and expectation. It is also fallible: soldiers misidentify civilians as fighters and friends as enemies, and the history of war is full of their mistakes. When a machine "recognises" a target, something quite different happens. Understanding that difference, at a conceptual level, is the precondition for understanding nearly every argument in the autonomous weapons debate. This chapter explains it without going into how such systems are built, because the aim is to see how and why they fail. From signatures to statistics The older autonomous weapons described in Chapter 1 recognised targets by rule. An anti-radar munition compares the emissions it detects against a library of known radar characteristics. A ship-defence system classifies a radar track as a threat if it is moving toward the ship at a speed and on a trajectory that only a missile would have. These rules were written by engineers and can be inspected. If the system attacks the wrong thing, one can, at least in principle, find out which rule produced the error. Automatic target recognition based on rules of this kind has a long history in military research, and its limitations were well understood. It works when targets have distinctive physical signatures and the background is simple. It struggles with cluttered scenes, with partial views, and with anything the rule-writers did not anticipate. Machine learning changed the approach. Instead of writing rules, developers assemble a large collection of examples, such as images labelled "tank", "truck", "car" or "person", and use them to adjust the internal parameters of a model until it produces the right labels for the examples. The model is then expected to produce the right labels for new images it has never seen. Deep neural networks, which became dominant in computer vision after a network trained on the ImageNet dataset dramatically outperformed rival methods in a 2012 competition, are extremely good at this under the right conditions. On benchmark tests they now classify everyday objects at rates that rival or exceed human performance. What a trained model has learned, however, is not a concept in the human sense. It is a mathematical function that maps patterns of pixels to scores. Those scores are often presented as "confidence", but they are not a reliable measure of how likely the model is to be right. Research published in 2017 by Chuan Guo and colleagues showed that modern neural networks tend to be systematically overconfident: a model that reports ninety-five per cent confidence may be right far less often than that. And the model has no access to any of the context that a soldier uses: no memory of the previous hour, no knowledge of the briefing, no sense of what is normal in this village. Four ways recognition breaks The weaknesses of machine perception are not speculative. They are among the most studied phenomena in artificial intelligence research, documented in civilian settings where the stakes are lower and the data are public. Four matter most for weapons. They are summarised in Table 2 and discussed below. Table 2. Documented failure modes of machine perception and their relevance to targeting. Failure mode What happens Public research example Why it matters in war Distribution shift Accuracy drops on data unlike training data Widely documented across vision benchmarks Battlefields change; adversaries adapt Shortcut learning Model keys on background cues, not the object Ribeiro et al. 2016; Geirhos et al. 2020 Right answer for wrong reason fails unpredictably Adversarial deception Small crafted changes flip the label Szegedy et al. 2014; Eykholt et al. 2018 Enemies can disguise targets or decoy civilians Unequal error rates Errors cluster in under-represented groups Buolamwini and Gebru 2018 Harm concentrates on particular populations The first is distribution shift. A model performs well on new examples only if they resemble the examples it learned from. When conditions differ, in lighting, weather, camera angle, terrain, season or the appearance of the objects themselves, performance can fall sharply and without warning. In civilian applications this is a nuisance. In war it is structural, because the environment is constantly changing and because an intelligent adversary is actively trying to make it change. Vehicles are camouflaged, covered with netting, painted, damaged, or replaced with new models. Decoys are deployed. Fighting moves from open fields into towns. The training data, which were collected in the past and often in different theatres or on exercise ranges, describe a world that no longer exists. The second is shortcut learning. Models frequently achieve high accuracy by latching onto features that happen to correlate with the right label in the training data but are not what the human designers intended. A much-cited 2016 study by Marco Tulio Ribeiro, Sameer Singh and Carlos Guestrin showed a classifier that distinguished wolves from huskies largely by detecting snow in the background, because most of the wolf photographs had been taken in snow. A 2020 review in Nature Machine Intelligence by Robert Geirhos and colleagues collected many such cases and argued that shortcut learning is a general property of deep networks, not an occasional bug. The danger is that a shortcut works perfectly until the correlation breaks, and then the model fails in ways that nobody predicted, because nobody knew what it was really looking at. The third is adversarial deception. In 2014 Christian Szegedy and colleagues reported that tiny, carefully calculated changes to an image, invisible to a human, could cause a neural network to misclassify it with high confidence. Later work showed that such effects could be produced in the physical world: a 2018 study by Kevin Eykholt and colleagues found that small stickers placed on a road sign could cause a classifier to read a stop sign as a speed-limit sign. There is now a large research literature on defending against such attacks, and no general solution. For a weapon, the implication is two-sided. An adversary may be able to make a military vehicle look like something else to the machine. And, more troublingly, an adversary willing to violate the law could make civilian objects look like military ones, inducing an autonomous system to attack them. The fourth is unequal error. A model's errors are not spread evenly. They cluster where training data were sparse or unrepresentative. Joy Buolamwini and Timnit Gebru's 2018 "Gender Shades" study of commercial facial-analysis systems found error rates in gender classification of up to about 35 per cent for darker-skinned women, compared with under one per cent for lighter-skinned men. There is no reason to think military datasets are immune from such imbalances, and some reason to think they are worse, since they are smaller, harder to collect and rarely subjected to outside audit. Where errors cluster, harm clusters, and a weapon that is "accurate on average" may be systematically dangerous to particular groups of people. Where the examples come from Every learned perception system is only as good as the examples from which it learned, and in the military domain good examples are scarce in ways that civilian developers rarely face. Civilian computer vision was built on enormous public collections of images, gathered from the internet and labelled by large numbers of paid workers. There is no equivalent for most military targets. Images of a particular adversary's newest air-defence vehicle, photographed from the angles and at the resolutions a drone will see, in the weather and terrain where it will be encountered, are rare by definition. Developers fill the gap in various ways: with imagery from previous conflicts, from exercises using their own or captured equipment, from models and simulations, and from computer-generated images. Each source carries its own distortions. Exercises happen in familiar terrain, under good conditions, with cooperative "targets". Simulated imagery is cleaner than the real world. Data from one war describe the equipment, tactics and camouflage of that war. Labelling is a further problem. Someone has to decide what each training example shows, and in military imagery that is often a matter of interpretation. Is the vehicle under the netting a tank or a decoy? Is the person beside the truck a fighter or a driver? Labelling errors and ambiguities in training data become, in the trained model, confident outputs that are wrong in ways no one can easily trace. Above all, there is the problem of the open world. Benchmarks test a model on a closed set of categories: every test image belongs to one of the classes the model was trained on. The real world is not closed. A drone over a battlefield will see tractors, ambulances, wedding parties, children carrying sticks, burnt-out vehicles, livestock, and objects no one thought to include. A classifier forced to choose among the categories it knows will assign each of these to one of them, sometimes with high confidence. Researchers have worked for years on ways for models to recognise that an input is unlike anything they have seen and to decline to classify it, and progress has been real but partial. For a weapon, the ability to say "I do not know what this is" is not a refinement. It is the difference between a system that holds fire when uncertain and one that resolves every uncertainty into a category, some of which are targets. These data problems are not peculiar to any one country or manufacturer. They follow from the structure of the task: learning to recognise rare, adversarial, context-dependent objects from limited and imperfect examples, in order to act on them in an open and changing world. The problem of persons All of these problems are serious when the target is a tank. They become categorically more serious when the target is a human being, because the features that make a person a lawful target are, for the most part, not visible. International humanitarian law, discussed in Chapter 6, permits attacks on combatants and on civilians only for such time as they take a direct part in hostilities. It forbids attacks on civilians, on medical personnel, and on anyone who is hors de combat: a fighter who has surrendered, been captured, or is incapacitated by wounds or sickness and is no longer able to defend themselves. Combatant status is a legal category, not a physical attribute. A uniform may indicate it, but many fighters in contemporary wars do not wear uniforms, and civilians routinely carry weapons for protection, work or tradition. Whether a person is directly participating in hostilities depends on what they are doing and why. Whether a soldier is surrendering depends on gestures, posture and intention that even humans read with difficulty under fire. A machine perception system trained to detect "persons" can do so very well in good conditions. A system trained to detect "armed persons" faces a harder task, and will confuse rifles with tools, sticks, cameras and umbrellas at some rate. A system that claims to detect "combatants" is attempting something that has no stable visual definition. It must rely on proxies: what people carry, where they are, how they move, who they are with, what patterns of behaviour they exhibit over time. The risk that proxies substitute for legal status is not hypothetical. In 2012 The New York Times reported that the United States government, in counting casualties from its drone campaign, treated military-age males in a strike zone as combatants unless there was explicit intelligence posthumously proving them innocent. That was a counting rule, not a perception system, but it illustrates the logic: when the legally relevant fact cannot be observed, an observable proxy takes its place, and the proxy quietly widens the circle of people who may be killed. A machine classifier trained on such proxies would encode the same widening into code, and apply it at scale. This is why the ICRC's 2021 position, discussed in Chapter 8, singles out autonomous weapon systems designed or used to apply force against persons as a category that should be prohibited. The argument is not that machines are worse than humans at recognising shapes. It is that the judgment about whether a particular person may be killed is not a recognition task at all. What testing can and cannot show Defenders of military artificial intelligence rightly point out that these weaknesses are known and that militaries test their systems. The United States Directive 3000.09 requires that autonomous and semi-autonomous systems undergo "rigorous hardware and software verification and validation" and "realistic system developmental and operational test and evaluation". Article 36 of Additional Protocol I to the Geneva Conventions requires states party to determine whether a new weapon would be prohibited by international law in some or all circumstances. Testing machine-learning systems is, however, fundamentally different from testing conventional weapons. A conventional munition can be tested for blast radius, accuracy and reliability under a defined range of conditions, and its behaviour will be broadly the same in war. A learned perception system's behaviour depends on its inputs, and the space of possible inputs in a real war is effectively infinite. Tests can establish performance on the conditions tested. They cannot establish performance on conditions no one thought to test, which is exactly where shortcut learning and distribution shift do their damage. Systems that continue to learn after deployment add a further problem: the system that was reviewed is no longer the system in use. There is also an asymmetry of knowledge. Researchers can study the failures of civilian vision systems because the models and datasets are often public. Military systems are classified, their training data are secret, and the conditions under which they are used are hidden from outside scrutiny. The United Nations Institute for Disarmament Research, in a 2020 study by Arthur Holland Michel titled The Black Box, Unlocked, argued that predictability and understandability are not fixed properties of a system but depend on who needs to understand it and for what purpose. A commander needs to know what a system will do in this operation; an investigator needs to know afterward why it did what it did. Machine-learning systems frequently satisfy neither need. Accuracy is not the whole question A final point guards against a common misunderstanding. It is often said that if machine perception becomes more accurate than human perception, the objection to autonomous weapons disappears. Humans misidentify targets too, sometimes catastrophically; perhaps machines, properly built, would make fewer mistakes. There are three answers. First, aggregate accuracy hides the distribution of errors. A system that is right ninety-nine times in a hundred but fails unpredictably, in correlated ways and against particular populations, can be more dangerous than a less accurate human whose failures are idiosyncratic and recognisable. When the same model is deployed on thousands of weapons, its blind spots are replicated across all of them. Second, error is not the only issue. A system that perfectly identifies people carrying rifles has still not determined whether any of them is lawful to kill, whether attacking them would cause excessive civilian harm, or whether a better option exists. Those judgments, which the law requires, involve weighing values and anticipating consequences. They are not classification problems, and no improvement in classification solves them. Third, the claim that machines will be more accurate than people is itself rarely testable from outside. It is made by those who build and buy the systems, about systems whose performance data are secret, in conditions that cannot be independently observed. In other fields, from pharmaceuticals to aviation, such claims are accepted only after independent verification. The absence of any such regime in weapons development is part of what the governance debate is about. The next chapter moves from the individual machine to many machines acting together, where a different kind of unpredictability appears: not the failure of a single classifier, but behaviour that no single component was designed to produce. Chapter 3: Swarms and the Problem of Emergence A flock of starlings over a winter reedbed can contain tens of thousands of birds, wheeling and folding in shapes that look choreographed. No bird is in charge. Each responds only to the handful of neighbours nearest to it, and the vast, coordinated pattern arises from those local interactions. Biologists call this collective behaviour, and computer scientists who studied it in the 1980s and 1990s gave it a name that has since migrated into military planning: swarm intelligence. The military interest in swarms is easy to understand. A single expensive platform is a single point of failure. A hundred cheap ones that coordinate among themselves can saturate defences, survive the loss of many members, and cover ground that no single platform could. The same features that make swarms militarily attractive make them a distinctive problem for the law and ethics of autonomous weapons, and that problem is the subject of this chapter. Its argument is that swarms do not merely multiply the risks of individual autonomous weapons. They add a new risk of their own: behaviour that no one designed, that cannot be fully tested, and that may be impossible to explain afterward. Order without a conductor The idea that complex group behaviour can arise from simple individual rules was demonstrated vividly in 1987, when the computer graphics researcher Craig Reynolds published a simulation of flocking. His simulated "boids" each followed a few local tendencies, roughly to avoid crowding neighbours, to steer in the same general direction as them, and to stay close to the group. Nothing in the program described a flock. Yet on screen the boids flocked, split around obstacles and rejoined, in ways that looked convincingly alive. The lesson, absorbed by robotics researchers in the decades that followed, was that coordination need not be centrally commanded. It can emerge. Emergence is the key concept. An emergent behaviour is a property of the system as a whole that is not specified in, and often cannot be straightforwardly predicted from, the rules governing its parts. Traffic jams that appear on motorways without any accident are emergent. So are stock market crashes that arise from the interaction of many trading programs, each behaving sensibly by its own lights. Military swarms, as publicly described, combine this decentralised logic with varying degrees of human direction. In October 2016 the United States Department of Defense tested 103 small Perdix drones released from fighter aircraft over China Lake, California; William Roper, then director of the department's Strategic Capabilities Office, described them afterward as "a collective organism, sharing one distributed brain for decision-making and adapting to each other like swarms in nature". Chinese state-linked industry publicised its own large swarm demonstrations in 2017 and 2018. In August 2023 the then Deputy Secretary of Defense, Kathleen Hicks, announced the Replicator initiative, intended to field "multiple thousands" of attritable autonomous systems across several domains within eighteen to twenty-four months, explicitly as a way of countering China's numerical advantages. None of these public descriptions explains how the swarms make decisions, and nothing more detailed is needed here. The governance-relevant point is structural. In a swarm, a human operator typically assigns a task to the group: search this area, defend this position, attack targets of this type. The allocation of that task among the members, their movements, and in more ambitious concepts their choice of which particular objects to engage, are then worked out among the machines. Human control operates at the level of the mission, not the individual engagement. Why emergence defeats prediction Chapter 2 showed that a single machine-learning classifier can fail unpredictably when its inputs differ from its training data. A swarm compounds that uncertainty in three ways. First, the behaviour of the whole depends on the interactions among the parts, and the number of possible interactions grows far faster than the number of parts. A swarm of ten can be simulated and tested across many scenarios. A swarm of a thousand, interacting with terrain, weather, electronic jamming, civilian traffic and an enemy, occupies a space of possible states so vast that testing can sample only a minute fraction of it. Engineers can establish statistical confidence about typical behaviour. They cannot establish what the swarm will do in the rare combinations of circumstances where accidents happen. Second, the parts may be individually correct and the whole still dangerous. Each drone might follow its rules perfectly and the collective result might nonetheless be an attack on the wrong place, a concentration of force far beyond what a commander intended, or a pattern of behaviour that an observer cannot interpret. When this happens there may be no "error" to find in any component. The failure lives in the interaction. Third, swarms on a real battlefield will meet other swarms, and other autonomous systems, controlled by an adversary. The best available analogy comes from finance. On 6 May 2010 United States stock markets fell by several percent within minutes and recovered almost as quickly, an event later called the "flash crash". The joint report of the Securities and Exchange Commission and the Commodity Futures Trading Commission attributed it to the interaction of a large automated sell order with high-frequency trading algorithms reacting to each other at machine speed. No trader intended the crash, and no human could have intervened quickly enough to stop it. Financial regulators responded with circuit breakers that halt trading when prices move too far too fast. Paul Scharre, a former Pentagon official whose 2018 book Army of None remains the most thorough general account of the subject, has drawn the obvious parallel. Two opposing sets of autonomous systems, each designed to react to the other faster than any human, could produce a "flash war": an escalation that neither side's commanders chose and that unfolds before either can intervene. Unlike financial markets, war offers no exchange operator to pull the plug. Each side's systems are secret from the other, so neither can model how they will interact. Testing against the real adversary is, by definition, impossible before the fact. Scale and the question of mass harm The second distinctive feature of swarms is scale. A weapon that requires one human operator per munition is limited in the harm it can do by the number of operators. A swarm that one person can launch at a mission level removes that limit. The concern was dramatised in Slaughterbots, a short film released in November 2017 by the Future of Life Institute with the computer scientist Stuart Russell, in which swarms of small drones using facial recognition assassinate selected individuals. The film was criticised by some defence analysts as alarmist, and it depicted a fictional scenario. But the underlying argument, which Russell and others made in more measured form at the United Nations, is not easily dismissed. If the cost of autonomous lethal drones falls far enough, and their supervision requires few enough people, then a small group could cause harm on a scale previously available only to states, and a state could cause harm on a scale previously associated with weapons of mass destruction, while remaining able to select its victims by category. The analyst Zachary Kallenborn has argued that sufficiently large, fully autonomous armed swarms should be considered for classification as weapons of mass destruction, on the grounds that they combine the potential for mass casualties with a degree of unpredictability that makes them hard to direct only at military objectives. This remains a minority view, and the threshold at which a swarm would deserve that label is contested. But it captures something the narrower debates about individual weapons miss. The ethical and legal difficulty of autonomy grows with scale, and not just in proportion to it. A single autonomous munition that misidentifies a target produces a tragedy. A thousand munitions sharing the same perception model and the same blind spot produce a pattern. There is a further concern that the combination of scale and selectivity makes swarms well suited to targeting people by group characteristics. A system that can be instructed to attack all persons matching some profile in an area, at a scale that requires little human labour, is a tool with obvious attractions for atrocity. Human Rights Watch and others have stressed that fully autonomous weapons would lack the human capacity to refuse an unlawful order. Soldiers have, at moments, declined to fire on crowds. Machines will not. Defence as the engine of autonomy Much public discussion of swarms imagines them as offensive weapons. In practice the strongest pressure toward autonomy at scale may come from defence, and this matters for how regulation is framed. A single incoming missile can be engaged by a human-supervised air-defence system. A hundred incoming drones arriving at once cannot be engaged one at a time by human decision. Russia's long-range attacks on Ukrainian cities, using large numbers of Iranian-designed Shahed-type drones together with missiles, created exactly this problem. Ukraine's responses, as widely reported, included mobile gun teams, electronic warfare, and increasingly the use of interceptor drones launched to collide with or destroy incoming attackers. As the volume of attacks rises, the pressure grows to automate the detection, assignment and engagement of interceptors, because human operators cannot keep pace. Defensive autonomy against incoming munitions raises the fewest ethical objections of any kind, as later chapters argue: its targets are military objects by nature, usually in the air, and the human decision to defend a city against an attack in progress is easy to justify. But the technologies are not easily separated. The perception that detects an incoming drone, the coordination that assigns interceptors among many threats, and the decentralised logic that keeps a defensive swarm functioning under jamming are much the same capabilities that an offensive swarm requires. A regime that permitted defensive autonomy while restricting offensive autonomy would have to draw the line by use and target type, not by technology. This is one reason why most serious proposals focus on what a system may attack and where, rather than on what it is made of. A swarm that may engage only incoming aerial munitions over a defended area is a very different object, legally and morally, from the same hardware instructed to engage ground vehicles or people in an inhabited region. The supervisor's view Swarms also change what human supervision looks like. A pilot of a single remotely operated drone watches one video feed and makes one set of decisions. An operator responsible for dozens of systems cannot watch dozens of feeds. Research on human supervision of multiple unmanned vehicles, conducted over two decades by groups including Mary Cummings's laboratory at the Massachusetts Institute of Technology, has consistently found that operator performance degrades as the number of vehicles rises, and that the degradation depends heavily on how much each vehicle can do on its own. The more autonomous each vehicle, the more of them one operator can manage; but the operator's grasp of what any single vehicle is doing correspondingly weakens. The supervisor of a swarm therefore sees an aggregate: a map of positions, a summary of status, alerts when something exceeds a threshold. The detail of individual engagements, which is where questions of distinction and proportionality arise, is exactly what is lost. The human is in the loop for the mission and out of it for the killing. That is not necessarily unlawful, as Chapter 6 explains, but it makes the legality of every engagement depend on decisions made before launch, about bounds and constraints, rather than on judgments made in the moment. Accountability in the collective Swarms also sharpen the accountability problems discussed in Chapter 6. When a single autonomous weapon attacks the wrong target, an investigation can at least begin with the question of what that weapon perceived and why it acted. When a swarm produces an unlawful outcome through emergent interaction, the causal story may be distributed across hundreds of machines and thousands of exchanges among them. Logging every one of those exchanges is technically demanding, and interpreting them afterward more so. The commander who launched the swarm assigned a lawful mission. The engineers who wrote the coordination rules wrote reasonable rules. The perception model was tested to the required standard. And yet civilians are dead. International humanitarian law places obligations on those who plan and decide upon an attack to take feasible precautions and to refrain from attacks expected to cause excessive civilian harm. But it is hard to say that a commander "expected" an outcome that arose from interactions nobody could predict. The law may be unable to attach responsibility to anyone, not because anyone is hiding, but because no one, in the relevant sense, decided. What can be bounded None of this means swarms are ungovernable. It means that the ways of governing them have to respect what emergence makes impossible. Since no one can predict the precise behaviour of a large swarm, reliance on post-hoc review of individual engagements is not a sufficient safeguard. The constraints have to be imposed in advance, on the conditions within which emergent behaviour is allowed to unfold. Several such constraints recur in the proposals of the ICRC, of SIPRI's research on limits on autonomy, and in the elements discussed by states in Geneva. One is limiting the types of target a swarm may engage, for instance to military objects by nature, such as armoured vehicles or air-defence systems, rather than persons or objects whose status depends on use. Another is limiting the area and duration of operation, so that the space within which surprises can occur is small and well understood. A third is limiting use to environments where civilians are not present, which in practice rules out most urban warfare. A fourth is requiring a reliable means of deactivation, so that a human can stop the swarm when it behaves in ways no one expected; the difficulty is that the very electronic warfare conditions that motivate autonomy may also disable the stop command. A fifth is a limit on scale: a cap on how many lethal systems a single human may be responsible for at once. There is a useful precedent for constraints of this kind in an older treaty. Amended Protocol II to the Convention on Certain Conventional Weapons, adopted in 1996, regulates landmines, which are, as Chapter 1 noted, the oldest autonomous weapons. Rather than banning them, it imposed conditions designed to limit their indiscriminate effects over time: remotely delivered anti-personnel mines must be equipped to self-destruct or self-deactivate within set periods, minefields must be recorded, marked and monitored, and parties must clear them after hostilities. The logic is directly transferable. A weapon whose behaviour within its operating window cannot be fully predicted can at least be prevented from operating outside that window, and its use can be recorded so that its consequences can later be traced. Whether the protocol's conditions were sufficient for mines is debatable, and many states concluded they were not, joining the outright ban in the Ottawa Convention. But the model of governing autonomy through limits on time, recording and recovery has a long pedigree in the very forum where autonomous weapons are now debated. Each of these constraints is costly from a military point of view, and each will be resisted by planners for whom the whole point of swarming is to operate at scale, over wide areas, in contested environments, without a vulnerable communications link. That tension is not a technical problem that better engineering will dissolve. It is the political core of the debate. Autonomy, and swarming in particular, are valuable to militaries precisely because they relax the bounds that make weapons predictable. The law's task is to reimpose them. The machines in this chapter and the last are only half of the system. The other half is the human being who is meant to supervise them. The next chapter asks how much that supervision can really accomplish. Chapter 4: The Human in the Loop On the night of 26 September 1983, Lieutenant Colonel Stanislav Petrov was the duty officer at a Soviet early-warning command centre south of Moscow when the satellite system he supervised reported the launch of an American intercontinental ballistic missile, and then several more. The procedure required him to pass the warning up the chain of command, where it could have triggered a retaliatory strike. Petrov judged that the alarm was false. A real American first strike, he reasoned, would involve hundreds of missiles, not five; the new satellite system was untested; ground radars had not confirmed anything. He reported a system malfunction. He was right: the satellites had mistaken sunlight reflecting off high-altitude clouds for missile launches. The Petrov story is told often, and for good reason. It is the ideal image of the human in the loop: a person with context, scepticism and the authority to say no, standing between a machine's output and a catastrophic action. Almost every government policy on military artificial intelligence invokes some version of this image. The United States requires "appropriate levels of human judgment". The United Kingdom speaks of "context-appropriate human involvement". Civil society calls for "meaningful human control". This chapter argues that the image is misleading, not because humans cannot exercise such judgment, but because the conditions that allowed Petrov to exercise it are exactly the conditions that modern military automation tends to remove. He had time, measured in minutes. He had a reason for doubt, in the system's novelty and the implausible size of the attack. He had independent information, from radar, to cross-check. And he was willing to bear the personal risk of disobeying procedure. Take away those conditions and a human in the loop becomes something else: a point of formal authorisation through which machine outputs pass on their way to the weapon. The ironies of automation The study of how people interact with automated systems is older than artificial intelligence, and its central findings are remarkably stable. In 1983, the same year as Petrov's night shift, the cognitive psychologist Lisanne Bainbridge published a short paper titled "Ironies of Automation". Her argument was that automating a process does not remove the human from it; it changes the human's job, usually for the worse. Operators of automated plants are left with the tasks the designers could not automate, which tend to be the hardest. They are asked to monitor systems that rarely fail, which humans do badly, because attention decays when nothing happens. And when the system does fail, they must take over in exactly the abnormal situations for which their now-atrophied skills are least prepared. Later research gave names to the specific failures. Automation bias, a term developed in studies by Kathleen Mosier, Linda Skitka and colleagues in the 1990s, describes the tendency of people to accept an automated recommendation without seeking contrary evidence, and to fail to notice problems the automation does not flag. Raja Parasuraman and Victor Riley, in a 1997 paper on the "use, misuse, disuse and abuse" of automation, described how operators come to over-rely on systems that are usually right, and how designers who automate whatever is technically possible often leave operators with an incoherent remnant of tasks. Mary Cummings, a former United States Navy fighter pilot who became a leading researcher in human supervisory control, applied these findings directly to military decision support in a 2004 paper on automation bias in time-critical systems. She argued that in fast, high-pressure environments, operators presented with a computer-generated recommendation tend to accept it, particularly when the reasoning behind it is opaque, and that the risk is greatest where the consequences of error are most severe. These are not marginal effects. They have been reproduced across aviation, medicine, process control and the military, and they apply to trained professionals as much as to novices. Two tragedies Two incidents involving highly automated American air-defence systems, operated by trained crews and subject to human authorisation, show what these findings mean in practice. On 3 July 1988 the cruiser USS Vincennes, equipped with the Aegis combat system, was engaged with Iranian gunboats in the Strait of Hormuz when a civilian airliner, Iran Air Flight 655, took off from Bandar Abbas on a scheduled flight to Dubai. The ship's crew identified the aircraft as a possible Iranian F-14 fighter. Under pressure of time and in the middle of a surface fight, they came to believe it was descending toward the ship, when the ship's own systems showed it climbing. The captain authorised two missiles. All 290 people aboard were killed. Subsequent analysis, including the United States Navy's own investigation and later academic work, highlighted what is sometimes called "scenario fulfilment": the crew expected an attack, and interpreted ambiguous information, and even the data on their own screens, to fit that expectation. The system did not fire on its own. Human beings in the loop, with seconds to decide, did. The second case is closer to the issues of this book. During the invasion of Iraq in 2003, Patriot missile batteries were involved in two fratricides. On 23 March a Patriot shot down a Royal Air Force Tornado returning to base, killing its two crew members; the system had classified it as an anti-radiation missile. On 2 April a Patriot shot down a United States Navy F/A-18 Hornet, killing its pilot. In both cases the human operators had a brief window in which to veto the engagement. They did not. John Hawley, an engineering psychologist who worked on Patriot for the United States Army for many years, later wrote a detailed account of these events for the Center for a New American Security, published in 2017 as Patriot Wars. His conclusion was that the Army had come to treat Patriot as more autonomous than it really was: operators were trained to trust the system's classifications, given little practice at recognising when it might be wrong, and placed in a role where, in Hawley's account, their nominal control was not real control. A 2005 report by the Defense Science Board task force on Patriot performance also pointed to problems with identification and with the operating protocols that allowed the system to engage with limited human scrutiny. The lesson of the two cases is not that humans are useless. It is that the formal presence of a human with authority to fire or veto is compatible with an absence of real judgment, when the human lacks time, lacks independent information, is primed to expect a threat, and has been trained to trust the machine. From weapons to decision support These dynamics apply with special force to the class of systems that most official definitions exclude from the autonomous weapons debate: decision-support systems that propose targets for human approval. Such systems do not pull triggers. They fuse intercepted communications, surveillance imagery, location data, social connections and other information, and produce lists of recommended targets, sometimes with scores or rankings. A human analyst or commander then approves or rejects each recommendation, and a separate human process conducts the strike. On paper, a human is fully in the loop at every lethal step. The question this chapter raises is what that human is actually doing. If the system produces recommendations faster than they can be examined, the human's role becomes throughput: approving or rejecting at the pace the machine sets. If the data on which recommendations rest are not visible, the human cannot evaluate them and can only evaluate the recommendation itself. If rejection requires justification and approval does not, the incentive structure favours approval. If the organisation measures performance by targets processed, scepticism becomes a cost. The Israeli military's former chief of staff, Aviv Kochavi, was quoted in 2023 as saying that in earlier conflicts the military's target unit would produce around fifty targets in Gaza per year, whereas its AI-enabled system produced a hundred targets a day. That ratio, whatever its precise accuracy, captures the problem. The volume of recommendations an organisation can generate has grown by orders of magnitude. The time available for any human to examine each one has not. Chapter 5 examines the investigative reporting on how such systems were reportedly used in Gaza, including claims, which the Israeli military disputes, that some approvals took around twenty seconds. There is a deeper difficulty. Automation bias research shows that people supervising automated systems not only accept bad recommendations but fail to look for what the system has missed. A decision-support system determines the space of options a human considers. It does not merely accelerate decisions; it frames them. The target that was never proposed is never weighed. The civilian presence the system did not register is not on the screen. Kabul, 2021: hours of watching, no real doubt A more recent case shows that even ample time does not guarantee judgment when the interpretive frame has already been set. On 29 August 2021, during the chaotic final days of the American withdrawal from Afghanistan, a United States drone strike in Kabul killed ten civilians, seven of them children. The target was a white Toyota driven by Zemari Ahmadi, an aid worker employed by an American charity. Analysts had tracked the car for some eight hours, interpreting his ordinary movements, stopping at his office, loading water canisters, driving home, as the behaviour of an Islamic State operative preparing an attack, in a setting where a suicide bombing at the airport had killed many people days earlier and another attack was expected. The Pentagon initially described the strike as righteous. After investigative reporting by The New York Times and others, the head of United States Central Command acknowledged that it had been a tragic mistake. No autonomous system was involved. But the episode is a precise illustration of the dynamics this chapter describes. The analysts were primed by a threat warning. A pattern of observed behaviour was read as a signature of hostile intent. Contrary evidence was not sought, or not credited. The human beings in the loop had hours, not seconds, yet the frame in which they interpreted what they saw was never challenged. Decision-support systems that score individuals on behavioural patterns, as some of those described in Chapter 5 are reported to do, would formalise exactly that kind of frame and apply it to thousands of people at once. The human reviewer would inherit the frame from the machine rather than from a briefing, and would be even less able to see outside it. Designing for doubt If formal human presence is not enough, what would make human judgment real? Research on automation and on organisational safety in fields such as aviation and medicine offers some answers, and they cut against the grain of what militaries usually want from technology. The first is to set the tempo by human need rather than machine capacity. That means deliberately limiting the rate at which recommendations are presented, or requiring a minimum period of review for certain categories of target, accepting that this reduces throughput. In air defence against incoming missiles, such delays may be impossible. In deliberate targeting of persons and buildings, where the decision is not forced by an incoming threat, they are almost always possible, and the argument against them is about efficiency, not necessity. The second is to show the evidence, not just the conclusion. A recommendation that a person or building is a target should come with the underlying information on which it rests, its sources, their age, and the known weaknesses of the method, in a form a reviewer can examine. Systems that present a score or a label alone invite automation bias. The third is to build in friction for approval as well as rejection. Where approving a recommendation requires a single click and rejecting it requires a written justification, the system has been designed to produce approvals. Symmetrical requirements, and periodic audits in which some approved targets are re-examined independently, change the incentives. The fourth is to preserve and exercise human skill. Analysts and operators who spend all their time reviewing machine output lose the ability to do the underlying work themselves, and so lose the ability to tell when the machine is wrong. Training that requires people to reach their own conclusions independently, and comparison of those conclusions with the machine's, keeps the skill alive. The fifth is to record everything. Logs of what the system recommended, what information it presented, what the human decided and how long the decision took are the precondition for later investigation and learning. They are also the evidence on which accountability depends. None of these measures is technically difficult. All of them have costs in speed and volume, and that is precisely why they are not adopted by default. They mark the boundary between human control as a design commitment and human control as a slogan. What meaningful human control requires The phrase "meaningful human control" was introduced into the debate around 2013 by Article 36, a British non-governmental organisation named after the article of Additional Protocol I that requires legal reviews of new weapons. Its strength was that it shifted the question from whether a human was present to whether the human's involvement had substance. Its weakness was that "meaningful" could be read in many ways. Subsequent work has given the concept more definite content. Michael Horowitz and Paul Scharre, in a 2015 primer for the Center for a New American Security, identified three essential components: that human operators make informed, conscious decisions about the use of weapons; that they have sufficient information to ensure the lawfulness of the action, given what they know about the target, the weapon and the context; and that the weapon is designed and tested, and human operators are trained, to ensure effective control over its use. A 2020 study by SIPRI and the ICRC, Limits on Autonomy in Weapon Systems, organised the practical elements into three types of control: controls on the weapon's parameters of use, such as target types, duration and geographical scope; controls on the environment of use, for example by excluding civilians from the area; and controls through human-machine interaction, meaning the ability of a human to supervise, intervene and deactivate. Drawing these together with the research on automation, one can identify the conditions that make human involvement meaningful rather than nominal: 1. Understanding: the person must know enough about how the system works, and how it fails, to judge when its output may be wrong. 2. Information: the person must have access to the underlying evidence and to independent sources, not only to the system's conclusion. 3. Time: the person must have enough time to deliberate, which means the tempo of the system must be set by the needs of human judgment, not the other way round. 4. Foreseeability: the person must be able to anticipate, with reasonable confidence, what the system will do within the bounds in which it operates. 5. Authority and incentive: the person must be able, and expected, to say no, without that refusal being treated as obstruction. 6. Answerability: the person must be someone who can be held responsible, legally and morally, for the outcome. These conditions apply equally to the supervision of an autonomous weapon and to the approval of a machine-generated target recommendation. That is why this book treats decision-support systems as part of the autonomy problem. A weapon that operates autonomously within tightly bounded parameters, against a narrow class of military objects, in an area without civilians, may satisfy these conditions better than a targeting process in which a human formally approves every strike but in practice ratifies a machine's output at machine speed. Petrov had all six conditions. The crews of Vincennes and the Patriot batteries had, in the decisive moments, perhaps two. The question for the governance of military artificial intelligence is not whether a human is present. It is which of these conditions the system, the organisation and the law actually guarantee. The next chapter examines what the record of recent wars suggests about the answer. Chapter 5: The Battlefield Record For most of the 2010s the debate about lethal autonomous weapons was conducted in the future tense. Diplomats, ethicists and military lawyers argued about systems that might one day exist. By the mid-2020s that was no longer true. Autonomous functions in weapons, and artificial intelligence in targeting, had moved from laboratories and trade shows into active wars. The record is partial, often contested and frequently distorted by the interests of those who report it. But it is substantial enough to test the arguments of the previous chapters against events. This chapter reviews four bodies of evidence: the reports from Libya and the South Caucasus that first suggested autonomous engagements; the war in Ukraine, which has become the largest laboratory of drone warfare in history; the use of AI-enabled targeting support in Gaza; and the reports of AI in American targeting during the 2026 strikes on Iran. In each case the emphasis is on what is known, who claims it, and what remains disputed. The main cases are summarised in Table 3. Table 3. Publicly reported uses of autonomy or AI in targeting, with source and status. Case Period Reported system or function Principal source Status of claim Libya 2020 Kargu-2 loitering munition, autonomous mode UN Panel of Experts, S/2021/229 Autonomous mode asserted, not demonstrated Nagorno-Karabakh 2020 Loitering munitions incl. Harop Press and analyst reporting Use confirmed; degree of autonomy unclear Ukraine 2022 onward Terminal-guidance autonomy on drones Manufacturers, officials, analysts Widespread; full autonomy disputed Gaza 2023 onward Gospel, Lavender target recommendation +972 Magazine and Local Call Systems acknowledged; usage claims denied Iran 2026 Commercial AI model in targeting support US press reporting Reported; extent of human review unclear Libya and the South Caucasus The Libya case, described in the Introduction, is the one most often cited as the first use of a lethal autonomous weapon against people. The text of the UN Panel of Experts report of March 2021 is suggestive: it described the Kargu-2 and other loitering munitions as "programmed to attack targets without requiring data connectivity between the operator and the munition". But the panel did not say it had evidence that any particular engagement was conducted autonomously, did not report casualties attributable to autonomous engagement, and did not explain the basis for its characterisation. The manufacturer, STM, has described the Kargu-2 in its public material as a system in which an operator selects the target before the munition dives. What the Libya episode does show is the opacity of the field. A UN body charged with monitoring an arms embargo, with access to a wide range of sources, could not establish with confidence how a weapon had been used. If that body could not, it is hard to imagine anyone else doing so routinely. The 2020 war between Azerbaijan and Armenia over Nagorno-Karabakh was the first conflict in which loitering munitions and armed drones played a decisive and highly visible role. Azerbaijan used Turkish Bayraktar TB2 drones and Israeli-made loitering munitions, including the Harop, a descendant of the Harpy described in Chapter 1, against Armenian air defences, armour and positions. Video released by Azerbaijan's defence ministry showed many such strikes. Whether any were conducted autonomously cannot be determined from the public record; the Harop can be operated by a human controller who selects the target. The war's significance for this book lies elsewhere. It demonstrated to militaries around the world that cheap loitering munitions could defeat expensive conventional forces, and so accelerated the investment in such systems that brought the autonomy question to the fore. Ukraine: autonomy by attrition The war that began with Russia's full-scale invasion of Ukraine in February 2022 transformed drone warfare in scale and character. Both sides came to depend on enormous numbers of small, cheap drones: first-person-view attack drones flown by operators wearing video goggles, larger long-range strike drones, loitering munitions such as Russia's Lancet, and maritime drones that Ukraine used with striking effect against Russia's Black Sea Fleet. Ukrainian officials spoke of national drone production in the millions per year. Electronic warfare drove the move toward autonomy. As each side deployed increasingly effective jamming against the radio links connecting operators to drones, a large share of attacks failed in their final stage. The responses have been described by Ukrainian manufacturers and officials, by Western journalists and by defence analysts at institutions such as the Royal United Services Institute and the Center for Strategic and International Studies. One was the fibre-optic drone, which unspools a physical cable and cannot be jammed. Another was the addition of terminal guidance: the operator designates a target, and if the link is lost the drone continues to the designated target on the basis of its own onboard vision. That second development is a modest step, but its significance is easy to understate. A drone that holds a target the operator selected is semi-autonomous on almost any definition, comparable in principle to a fire-and-forget missile. Yet the conditions differ. The target may be a vehicle or a person; the drone may lose the link several hundred metres out; and in the seconds before impact, the scene can change. A soldier may raise their hands. A civilian vehicle may pass in front. The operator, if still connected, might abort; the drone, if not connected, will not. Reports of more fully autonomous systems have appeared repeatedly. In 2023 some outlets reported claims by a Ukrainian company that its drone system could identify and attack targets autonomously in some modes; Russian manufacturers have made analogous claims for their own products. Such statements are hard to evaluate. Manufacturers have commercial reasons to advertise capabilities, governments have reasons both to exaggerate their own sophistication and to deny that they cross normative lines, and battlefield verification is almost impossible. Ukrainian officials, for their part, have generally emphasised that a human makes the decision to strike. The most careful assessment is that autonomy in Ukraine has advanced incrementally, mainly in navigation and terminal guidance, with humans still selecting most targets, but that the pressure toward further autonomy is intense and the technical barriers are falling. Ukraine also illustrates the point made in Chapter 3 about scale. The decisive constraint for both armies is not weapons but trained operators. A technology that allows one operator to manage many drones, or allows drones to operate when operators cannot reach them, is worth a great deal. The incentives that drive this are not peculiar to Ukraine. They will operate in any future war between technologically capable adversaries. And when the war ends, the systems, the expertise and the precedents will spread. Gaza: the targeting factory The use of artificial intelligence in Israel's military operations in Gaza after the Hamas-led attack of 7 October 2023 has been the subject of the most detailed, and most contested, public reporting on AI in targeting anywhere. The Israeli military itself had publicised its investment in AI-enabled targeting before the war. In November 2023 it described on its website a system called the Gospel (Habsora in Hebrew) that it said allowed the rapid production of targets. On 30 November 2023 the Israeli-Palestinian publication +972 Magazine, together with the Hebrew-language site Local Call, published an investigation describing the Gospel as generating recommendations of buildings and structures to attack, including private homes said to be associated with militants, and quoting sources who described the operation as a "mass assassination factory". On 3 April 2024 the same outlets published a second investigation by the journalist Yuval Abraham, based on the accounts of six Israeli intelligence officers, about a system called Lavender. According to these sources, Lavender assigned residents of Gaza a rating reflecting the likelihood that they were members of Hamas or Palestinian Islamic Jihad, and at one stage had marked some 37,000 Palestinians as suspected militants. The sources said that the system was known to be wrong in roughly ten per cent of cases, and that during the early weeks of the war human review of its outputs was in some cases cursory: one source described spending "20 seconds" per target, mainly to confirm that the target was male. The investigation also described a separate system, referred to as "Where's Daddy?", used to track marked individuals and indicate when they had entered their family homes, where, the sources said, many strikes were carried out. The Israeli military rejected central elements of this account. In a response published alongside the investigation, it said that it does not use an artificial intelligence system that identifies terrorist operatives or tries to predict whether a person is a terrorist, that Lavender is a database used to cross-reference intelligence sources, and that information systems are tools for analysts. It stated that an independent examination by an intelligence analyst is required to verify that identified targets are legitimate targets for attack, in accordance with international law and military directives. There is therefore a direct factual dispute, which this book cannot resolve. What can be said is that the existence of the systems is not contested; that the military's own public statements emphasised the speed and volume of target generation, as Kochavi's remark about a hundred targets a day indicates; and that the reported pattern, if accurate, matches closely the dynamics described in Chapter 4: a recommendation system producing outputs at a volume that compresses human review, data proxies standing in for legal status, and a formal human in the loop whose scrutiny is reported to have been minimal. Further reporting by other outlets, including The Washington Post in late 2024, described internal debate within the Israeli military about the reliability of AI-generated intelligence. Reporting by the Associated Press in early 2025 described the use of commercial cloud and AI services supplied by United States technology companies in Israeli military operations, and in September 2025 Microsoft said it had ceased providing certain services to a unit of the Israeli Ministry of Defence after reviewing reports that its cloud platform had been used to store intercepted Palestinian communications. The Gaza reporting matters for the autonomous weapons debate for two reasons. First, it involves systems that are not autonomous weapons under any official definition: no machine fired. If the reported practices are accurate, they nonetheless raise exactly the concerns about compressed judgment that the debate is meant to address. Second, the Gaza case shows the role of decision support in the specific context of attacks on persons, the category the ICRC argues should be prohibited for autonomous weapons. A regime that prohibited autonomous anti-personnel weapons but left decision-support systems entirely unregulated would not reach the practices described. Iran, 2026: commercial AI in the kill chain The newest chapter in this record opened in early 2026. On 28 February 2026 the United States and Israel began a campaign of strikes against Iran. In the days that followed, American newspapers including The Wall Street Journal and The Washington Post reported that the United States military had used a commercial large language model, Anthropic's Claude, in its targeting processes, reportedly to help analyse intelligence and to assist with target recommendations and prioritisation. NBC News reported in March that the military was using AI to help plan air attacks on Iran. Stop Killer Robots cited reports that more than a thousand targets were struck in the first twenty-four hours. The extent of human review of AI-assisted recommendations has not been publicly established, and the details of how the model was used remain unclear. The episode had an unusual backdrop. In the weeks before the strikes, Anthropic had been in a public dispute with the Department of Defense. According to widespread reporting, the department had demanded the ability to use the company's models for "all lawful purposes", and Anthropic had refused to remove two restrictions from its terms: on use for mass domestic surveillance and on use in fully autonomous weapons. Anthropic's chief executive, Dario Amodei, said the company supported lawful military and intelligence uses, including partially autonomous weapons of the kind used in Ukraine, but not fully autonomous ones. President Trump ordered federal agencies to stop using the company's products, and in early March the Pentagon designated Anthropic a supply-chain risk. On 26 March a federal district judge in California granted a preliminary injunction against the designation, describing it as retaliation for the company's public stance; litigation continued thereafter. Several aspects of this episode deserve attention, whatever its eventual resolution. It showed that general-purpose AI models built for civilian markets can be inserted into military targeting processes quickly, with little public visibility, and that the distinction between "decision support" and "autonomy" may be decided by commercial contracts as much as by law. It showed that companies building such systems may attempt to draw lines that states have not drawn, and that states may resist. And it brought the question home to the countries that have led opposition to a binding international instrument. When the UN Secretary-General and the President of the ICRC renewed their joint appeal in August 2026, they observed that "when the architects of this technology are calling for limits, States cannot afford to remain passive". How we know what we know Every account in this chapter depends on a chain of evidence, and it is worth being explicit about the kinds of links in those chains, because the governance debate will turn on whether violations can ever be detected. The first kind is official disclosure: statements, videos and documents released by militaries and governments. These are indispensable but selective. Militaries release footage of successful strikes, describe their systems in terms that emphasise safeguards, and rarely disclose failures. Israel's own publicity about the Gospel, and the American Defense Department's descriptions of Replicator and Perdix, tell us what those institutions wished to say. The second is commercial material: product brochures, trade-show demonstrations and interviews with company executives. Manufacturers have reasons to overstate capabilities to customers and investors, and occasionally to understate them when public scrutiny rises. The third is investigative journalism based on confidential sources. The +972 and Local Call reporting on Gaza, and the American reporting on targeting in Iran, belong here. Such reporting can reveal what official sources conceal, but it depends on the honesty and knowledge of sources who cannot be named, and the affected governments can dispute it without producing contrary evidence. The fourth is open-source investigation: the analysis of publicly available video, satellite imagery, wreckage photographs and social media by researchers and journalists. Groups using these methods have been able to identify weapon types, establish the location and timing of strikes, and sometimes reconstruct how an attack unfolded. They are much less able to determine what happened inside a weapon or a targeting process: whether a drone was being steered by a person at the moment of impact, or how long an analyst looked at a recommendation. Autonomy is, for the most part, invisible from the outside. The fifth is the work of international bodies, such as UN panels of experts and commissions of inquiry, and of humanitarian organisations working in conflict zones. These have greater legitimacy but often, as in Libya, little more access to the technical facts than anyone else. The implication is sobering. Under present conditions, the use of autonomous functions in a particular attack can almost never be established with confidence by anyone outside the military that conducted it. Any international rule on autonomy will therefore depend heavily on what states record, retain and are willing to disclose, and on whether any mechanism exists to require them to do so. What the record shows Taken together, these cases support three conclusions. First, the most important developments have not been the dramatic ones. There is no confirmed case, in the public record, of a fully autonomous weapon selecting and killing people without any human having designated a target or a narrowly defined target class. What there is, abundantly, is incremental autonomy in the final stages of engagement, and AI-enabled acceleration of targeting upstream of the weapon. Second, the opacity problem is pervasive. In every case, crucial facts depend on sources that cannot be independently verified: manufacturers, militaries, anonymous officers, leaked documents. The people best placed to know how systems were used are those with the strongest reasons not to say. Any governance regime that depends on outside actors detecting violations after the fact will struggle. Third, the record confirms the controlling argument of this book. The erosion of human judgment has come not through machines taking decisions, but through speed, volume and the reliance of humans on outputs they cannot examine. The question for law, to which the next chapter turns, is whether its existing rules can capture that erosion. Chapter 6: The Law of Armed Conflict Meets the Algorithm A common misconception holds that autonomous weapons exist in a legal vacuum. They do not. Every weapon used in armed conflict, whether a rifle, a cruise missile or a self-guiding drone, is subject to international humanitarian law: the body of rules, codified principally in the Geneva Conventions of 1949 and their Additional Protocols of 1977, and in customary international law, that regulates the conduct of hostilities. Outside armed conflict, international human rights law governs any use of force by states, including by police. The question is not whether law applies, but whether its rules can be meaningfully applied when part of the decision-making is delegated to machines, and who is responsible when they are broken. This chapter argues that existing law already implies significant limits on autonomy, but that it was written for human decision-makers and leaves dangerous gaps when the decisions are distributed between people and software. Those gaps explain why most states now accept that something more is needed, even if they disagree about what. Three rules that require judgment Three foundational rules of the law of targeting do most of the work. Each is binding on all states as customary law, whether or not they have ratified Additional Protocol I. The first is distinction. Parties to a conflict must at all times distinguish between civilians and combatants, and between civilian objects and military objectives, and may direct attacks only against the latter. Additional Protocol I, in Article 48, states the basic rule; Article 51 prohibits attacks on civilians and indiscriminate attacks, including those employing a method or means of combat that cannot be directed at a specific military objective. Chapter 2 showed why distinction is hard for machine perception: the legally relevant status of persons is not a visual property, and objects can change status depending on how they are used. A school used as a firing position may become a military objective; when the fighters leave, it ceases to be one. The second is proportionality. Even an attack on a lawful military objective is prohibited if it may be expected to cause incidental loss of civilian life, injury to civilians or damage to civilian objects that would be excessive in relation to the concrete and direct military advantage anticipated. This rule, codified in Articles 51 and 57 of Additional Protocol I, requires a comparison between two quantities measured in different currencies: human lives and military advantage. Military lawyers have long acknowledged that it calls for judgment in context, informed by the commander's understanding of the operation's aims. No formula captures it. Software can estimate likely civilian casualties from a strike, and militaries have used collateral damage estimation tools for decades, but deciding whether that harm is excessive in relation to the anticipated advantage is an evaluative judgment. The third is precaution. Article 57 requires those who plan or decide upon an attack to do everything feasible to verify that targets are military objectives, to choose means and methods that minimise incidental civilian harm, and to cancel or suspend an attack if it becomes apparent that the target is not a military objective or that the attack would be disproportionate. The duty to cancel or suspend is especially important for autonomy. It assumes that someone is watching and able to act when circumstances change. A weapon that, once launched, cannot be recalled, and a supervisor who cannot perceive the change in time, both sit uneasily with it. Table 4 sets out how each rule interacts with autonomous functions. Table 4. Core targeting rules of international humanitarian law and the challenges autonomy poses. Rule Source What it requires Challenge from autonomy Distinction AP I arts. 48, 51, 52; custom Attack only combatants and military objectives Legal status not visually observable Proportionality AP I arts. 51(5)(b), 57; custom No excessive incidental civilian harm Value judgment, not a calculation Precaution AP I art. 57; custom Verify targets; cancel if circumstances change Requires timely awareness and ability to abort Weapons review AP I art. 36 Assess new weapons for legality Learned behaviour hard to review in advance Martens Clause AP I art. 1(2); Hague preambles Principles of humanity, public conscience Basis for arguments beyond specific rules Who is the decision-maker? Notice the grammatical subject of these rules. They are addressed to "those who plan or decide upon an attack". The law assumes that a human being forms a judgment about a specific attack, based on the information reasonably available at the time. That is how responsibility attaches: a commander is judged by what they knew, or should have known, when they decided. An autonomous weapon complicates this in a precise way. The human decision occurs at the moment of activation, when the commander chooses to deploy the system within certain parameters. The actual application of force occurs later, at a time, place and against a specific target that the commander did not choose and may not know. The law's judgments must therefore be made at activation, on the basis of what the commander could foresee about how the system would behave in the circumstances. This is why predictability, discussed in Chapters 2 and 3, is a legal requirement and not merely an engineering virtue. If a commander cannot reasonably foresee what a weapon will attack, they cannot make the judgments of distinction, proportionality and precaution that the law requires of them. A weapon whose effects cannot be anticipated may fall foul of the prohibition on indiscriminate attacks. The ICRC's 2021 recommendation that "unpredictable autonomous weapon systems should be expressly ruled out, notably because of their indiscriminate effects" is, in this sense, a restatement of existing law, not a new rule. Some lawyers, including many in the United States and allied governments, argue that existing law is therefore sufficient: commanders may use autonomous functions only where they can foresee the effects well enough to comply, and must refrain otherwise. That is correct as far as it goes. The difficulty is that the law sets no concrete standard for how much foreseeability is enough, how long a system may operate, how large an area it may cover, or what types of target it may engage. Each state applies its own judgment, behind a veil of secrecy, to systems whose behaviour it may not fully understand. The rules are clear in principle and indeterminate in practice. Weapons review and its limits Article 36 of Additional Protocol I requires each state party, in the study, development, acquisition or adoption of a new weapon, means or method of warfare, to determine whether its employment would be prohibited in some or all circumstances. Some states not party to the Protocol, including the United States, conduct such reviews as a matter of policy. Weapons review is a crucial safeguard in principle, and many states in the autonomy debate point to it as the main mechanism for ensuring compliance. In practice it has three limitations. Only a minority of states are known to have formal review procedures, and very few publish their results. Reviews are conducted by the state developing or acquiring the weapon, with no international oversight. And, as Chapter 2 explained, machine-learning systems are hard to review in advance, because their behaviour depends on conditions that cannot be fully anticipated, and because systems updated after deployment may differ from the version reviewed. Several states have proposed in the Group of Governmental Experts that reviews of autonomous systems should be strengthened and that good practice should be shared, and these proposals appear in many of the draft elements. Whether that is enough is a central point of disagreement. The accountability gap The law also asks who is responsible after the fact. International criminal law punishes individuals for war crimes, which generally require intent or recklessness: deliberately attacking civilians, or launching an attack knowing it will cause excessive civilian harm. State responsibility attaches to states for internationally wrongful acts. Human Rights Watch and the Harvard International Human Rights Clinic set out the problem in a 2015 report, Mind the Gap. If an autonomous weapon attacks civilians because of an unforeseeable failure of its perception system, who has committed a crime? Not the machine, which cannot be prosecuted or punished. Not the commander, who neither intended nor foresaw the outcome. Not the programmer, who wrote the code years earlier for a different purpose. Not the manufacturer, which is typically shielded from liability for military use of its products. The result may be serious violations of the law for which no one bears criminal responsibility. Defenders of autonomy respond that the same gap exists for other technologies. A commander who uses an artillery shell that malfunctions is not criminally liable either; the state remains responsible; and the doctrine of command responsibility holds superiors liable for failing to prevent crimes they knew or should have known about. These responses have force, but they understate what is different. A malfunctioning shell is a rare event with a recognisable cause. An autonomous system that misidentifies a class of targets because of a training-data imbalance may fail systematically, across many engagements, in ways no one detected. The law's model of individual culpability for individual decisions fits poorly with harm produced by a pattern embedded in software. The philosopher Robert Sparrow, in an influential 2007 paper titled "Killer Robots", argued that this gap is not merely practical but moral: that it is a condition of fighting justly that someone be responsible for each death, and that weapons which make that impossible should not be used. That argument is taken up in the next chapter. For the law, the minimal implication is that states using autonomous systems must take responsibility for the conditions of their use, through reviews, rules of engagement, logging and investigation, since individual criminal law will often have nothing to grip. State responsibility, investigation and the role of companies If individual criminal responsibility often has nothing to grip, other forms of responsibility become more important. Three deserve attention. The first is the responsibility of states. Under the law of state responsibility, a state is responsible for violations of international humanitarian law committed by its armed forces, whether or not any individual can be convicted. The Geneva Conventions also oblige states to respect and to ensure respect for the Conventions in all circumstances. Applied to autonomy, this means that a state which fields a weapon that attacks civilians, or adopts a targeting process that produces unlawful strikes at scale, has committed a wrongful act and owes reparation, regardless of whether a particular officer foresaw the outcome. State responsibility is enforced weakly in practice, but it supplies a framework within which the question "who is responsible?" always has at least one answer. The second is the duty to investigate. States must investigate alleged war crimes by their nationals or on their territory, and good practice, reflected in military manuals, extends to investigating incidents of civilian harm more generally. An investigation is only as good as the evidence available to it. For autonomous functions and AI-enabled targeting, that evidence lies in system logs, model versions, training records and records of human approvals. Unless such records are created and preserved by design, the duty to investigate becomes impossible to fulfil. This is why several proposals in the GGE, and much of the academic literature, stress traceability: the ability to reconstruct after the fact what a system did and why. It is telling that, according to campaigners' accounts, this was among the concepts deleted from the GGE's September 2026 report. The third concerns the companies that build these systems. International humanitarian law binds states and individuals rather than corporations as such, and defence contractors have generally been shielded from liability for the use of their products by governments. The entry of general-purpose AI companies into military markets, described in Chapter 5, complicates this picture. Their products are not designed as weapons, their terms of use may restrict certain applications, and they may have more insight into a model's failure modes than the military using it. The United Nations Guiding Principles on Business and Human Rights, endorsed by the Human Rights Council in 2011, set out an expectation that companies respect human rights and conduct due diligence on the impacts of their products, including in conflict-affected areas. They are not binding. But they provide a vocabulary for asking what a technology supplier should know, and refuse, when its products are used in targeting. Beyond the battlefield and beyond the rules Two further strands of law widen the debate. International human rights law applies to the use of force by states outside armed conflict, in law enforcement, border control and counter-terrorism. Its standards are stricter than humanitarian law: lethal force may be used only where strictly necessary to protect life. Christof Heyns, the UN Special Rapporteur on extrajudicial, summary or arbitrary executions, warned in a 2013 report to the Human Rights Council that autonomous weapons could not meet those standards and called for national moratoria. In April 2025 Human Rights Watch and the Harvard clinic published A Hazard to Human Rights, arguing that autonomous weapons would threaten rights to life, to peaceful assembly, to privacy and to remedy, and that the treaty debate should not be confined to armed conflict. Since many systems developed for war migrate to policing and border control, this is not a theoretical concern. The second strand is the Martens Clause, which first appeared in the Hague Conventions of 1899 and is restated in Article 1(2) of Additional Protocol I. It provides that in cases not covered by specific agreements, civilians and combatants remain under the protection of "the principles of humanity and the dictates of public conscience". Lawyers disagree about how much independent legal force the clause carries; the International Court of Justice, in its 1996 advisory opinion on nuclear weapons, affirmed its continuing relevance without resolving that debate. In the autonomy context the clause is invoked to argue that delegating life-and-death decisions to machines offends humanity even where no specific rule is broken. The Martens Clause matters because it has been used before to justify new prohibitions. In 1995 states adopted Protocol IV to the Convention on Certain Conventional Weapons, banning laser weapons specifically designed to cause permanent blindness, before such weapons had been widely deployed. The protocol was a pre-emptive ban, grounded largely in a shared sense that the effect was abhorrent. Campaigners for a treaty on autonomous weapons cite it as proof that the CCW can act before harm occurs. Opponents note that blinding lasers were a narrow category with few military advocates, whereas autonomy is a broad capability in which major powers are investing heavily. What follows for the law The analysis in this chapter points to a specific conclusion. Existing international humanitarian law already requires that commanders be able to foresee and limit the effects of the weapons they use, that they verify targets and cancel attacks when circumstances change, and that states review new weapons. Those requirements already exclude some uses of autonomy: systems whose effects cannot be anticipated, systems unable to distinguish lawful from unlawful targets in their environment of use, and systems that cannot be stopped when an attack becomes unlawful. What existing law does not do is translate those requirements into concrete limits that states must observe and can verify: how long, how far, against what, with what supervision. Nor does it address the practices described in Chapter 5, where humans formally decide but their decisions are hollowed out by machine-generated volume. The case for a new instrument rests less on the claim that existing law is silent than on the claim that it is too indeterminate, applied in too much secrecy, to hold the line against the incentives described in earlier chapters. The ethical arguments, examined next, push further. Some hold that even a perfectly lawful autonomous killing would be wrong. Chapter 7: The Ethics of Delegated Killing Imagine, for the sake of argument, an autonomous weapon that works exactly as advertised. It attacks only lawful military targets, never misidentifies a civilian, calculates incidental harm better than any human staff officer, and aborts whenever circumstances change. Would there be anything wrong with using it? Many people's intuitions say yes, and many say no, and the disagreement is not a matter of misunderstanding the technology. It reflects different views about what makes killing in war permissible at all. This chapter examines the main ethical arguments, for and against, and tries to show which survive scrutiny. Its conclusion is that the strongest objections are not about error rates, which might in principle be reduced, but about the relationship between the person who dies and the person who decides, and about the effects of autonomy on the institutions and practices of war. Those objections connect directly to the controlling argument of this book: what matters is not whether a machine is involved, but whether a human being still exercises, and answers for, the judgment that a particular death is justified. The case for autonomy The most serious ethical case for autonomous weapons was made by the roboticist Ronald Arkin in his 2009 book Governing Lethal Behavior in Autonomous Robots. Arkin did not argue that autonomous weapons were desirable in themselves. He argued that human soldiers commit atrocities, that they do so for reasons rooted in human psychology, and that machines without those weaknesses might behave better. He drew on evidence such as the United States Army's mental health advisory team survey of troops in Iraq, published in 2006, which found that fewer than half of the soldiers surveyed agreed that non-combatants should be treated with dignity and respect, and that around one in ten reported having mistreated non-combatants. Fear, anger, exhaustion, the desire for revenge after the death of comrades, and the tendency to see every local person as a potential threat all contribute to war crimes. A machine feels none of them. It need not shoot first to protect itself. It can be programmed to accept risk to itself in order to protect civilians, as human soldiers often will not. There are other arguments in the same vein. Autonomous systems may allow more precise strikes, reducing civilian harm. They may protect a state's own soldiers, which is itself a moral good. They may be necessary for defence against adversaries who are deploying them, so that a state which refuses them leaves its population exposed. Ukraine's position illustrates the last point: a country defending itself against a larger invader has strong reasons to use every technology that reduces its dependence on scarce human soldiers. These arguments deserve to be taken seriously. They rest on real facts about human failure in war. But each has a limit. The comparison between idealised machines and actual humans is unfair. The machines that exist are not idealised. They have the failure modes described in Chapter 2, and they fail in ways that are systematic rather than individual. Arkin himself emphasised that his proposals were a research agenda and that machines should be used only where they could be shown to outperform humans; that demonstration has not been made for any system operating against persons in complex environments. The precision argument conflates accuracy in hitting a target with correctness in choosing it. A weapon can strike precisely the wrong thing. The force-protection argument is morally real but cuts in both directions, as discussed below: reducing the cost of war to the side that wages it may make war more frequent. The defensive necessity argument is strongest, and it is one reason why most proposals for regulation do not seek to ban all autonomy. A carefully bounded system defending against incoming missiles or drones raises few of the concerns in this chapter. The argument is much weaker as a justification for anti-personnel autonomy or for unbounded systems, which are not obviously necessary for defence and carry the greatest risks. Dignity and the question of who decides The most distinctive objection to autonomous weapons is that being killed by a machine, on the basis of a machine's assessment, violates human dignity in a way that being killed by a human does not. This argument was made most forcefully by Christof Heyns, the UN Special Rapporteur, and by the philosopher Peter Asaro, whose 2012 article in the International Review of the Red Cross argued that the decision to take a human life must be made by a human being, because only a human can recognise the person as a person and weigh the significance of ending their life. On this view, the moral requirement is not only that killings be accurate and lawful but that they result from a human judgment directed at the particular person. A machine processing sensor data against a target profile does not see a person. It sees a pattern that matches a category. To be killed in that way is to be treated as an object. Critics reply that this objection is sentimental. The dead are equally dead whoever pulls the trigger. A soldier firing artillery at grid coordinates does not "recognise the humanity" of the people at the other end, nor does a bomber crew at thirty thousand feet. Modern war has long been conducted at a distance, against targets defined by category. Why should the final step of automation make a moral difference? The answer, I think, is that it is not the distance that matters but the presence or absence of a judgment for which someone is answerable. The artillery officer decided to fire on that location, knowing that people were there, and must answer for that decision. The bomber crew was ordered to attack that target by commanders who had decided it was lawful. In each case a human being made a judgment about this attack, and that judgment can be examined, defended or condemned. The dignity objection is strongest when understood not as a claim about the inner experience of the victim, who cannot know how the decision was made, but as a claim about the moral structure of killing: that taking a life is permissible only as the conclusion of a human judgment that someone owns. Sparrow's argument about the responsibility gap, mentioned in Chapter 6, is the other face of this objection. If no one can fairly be held responsible for a death, then, he argued, the killing fails a basic condition of justice in war. One need not accept every step of Sparrow's argument to accept its core: a practice of killing in which responsibility becomes systematically diffuse is morally degraded, even if the individual outcomes are not worse. These two objections share a feature that matters for the governance debate. They apply with special force to autonomous systems that select and attack persons, and they apply also to decision-support systems in which human approval has become nominal. They apply much less to bounded systems that attack military objects by nature, such as incoming missiles, where the targeting judgment is in effect made in advance by the human who defined the narrow class of targets. Lowering the threshold A second family of objections looks not at individual killings but at the effects of autonomous weapons on the frequency and character of war. The first concern is that autonomy lowers the political cost of using force. Democracies in particular are constrained by the reluctance of their citizens to see their own soldiers killed. Weapons that remove soldiers from danger make it easier to start and sustain military operations, and harder for publics to hold leaders accountable. The same concern was raised about armed drones in the 2000s and 2010s, and the expansion of drone strikes outside declared war zones in that period suggests it was not misplaced. Pope Leo XIV, in an encyclical on artificial intelligence published in 2026, warned that such technologies risk lowering "the threshold to war" and "reducing victims to data". The second concern is strategic instability. Chapter 3 described the possibility of "flash wars" arising from interactions between opposing autonomous systems. More broadly, if states believe that autonomy gives a decisive advantage in speed, they will race to field it, cutting corners on testing and pushing human decision-making further out of the loop for fear of being too slow. The result could be crises that escalate faster than political leaders can manage, a concern especially acute if autonomy reaches systems connected to nuclear command and control. The third concern is proliferation. Cheap autonomous weapons will not remain in the hands of states that comply with international law. The technology for small drones, onboard vision and basic autonomy is widely available and largely civilian. Once military forces normalise and refine these systems, they will spread to armed groups, criminal organisations and repressive governments, who will use them without the safeguards their originators claimed. Over three thousand artificial intelligence and robotics researchers signed an open letter in July 2015, organised by the Future of Life Institute, warning that autonomous weapons would become "the Kalashnikovs of tomorrow", cheap and ubiquitous. The developments in Ukraine and elsewhere since then have not made that warning look exaggerated. Riskless war and the distribution of harm A related argument concerns the moral structure of combat itself. The legal scholar Paul Kahn, in a 2002 essay titled "The Paradox of Riskless Warfare", argued that the traditional moral justification for killing enemy combatants rests in part on reciprocity: soldiers may kill one another because each poses a threat to the other. When one side can kill without any risk to its own forces, he suggested, the activity begins to look less like combat and more like policing or punishment, and needs a different justification, with stricter standards. Kahn wrote about precision air power and drones operated from far away, but the argument applies with greater force to autonomy. A remotely piloted drone still involves a human who watches, decides and carries the psychological weight of the decision; studies of drone crews have documented the moral injury some experience. An autonomous system removes even that. Killing becomes an output of a process in which no one on the attacking side is exposed to danger, witnesses the death, or necessarily knows that it happened. Whatever one thinks of reciprocity as a basis for the ethics of war, a practice of killing that imposes no cost of any kind on those who conduct it lacks one of the brakes that has historically restrained violence. The distribution of harm also matters. Autonomous weapons will be developed by a small number of technologically advanced states and companies. The history of armed drones suggests where they will be used: in the territories of poorer states, against populations with little political voice in the countries that deploy them. The datasets on which targeting models are trained will reflect the appearance, behaviour and patterns of life of those populations, as filtered through the assumptions of those who build the systems. If errors cluster, as Chapter 2 showed they tend to, they will cluster among people already marginal in global politics. This is one reason why so many states of the Global South have been among the strongest supporters of binding rules, and why civil society campaigners speak of "digital dehumanisation": the reduction of people to data points processed by systems built elsewhere. The erosion of moral skill A third, quieter objection concerns what autonomy does to the humans who remain. Military ethics depends on the cultivation of judgment: the capacity of soldiers and commanders to perceive morally relevant features of a situation, to restrain themselves, to refuse unlawful orders. That capacity is developed through practice and responsibility. The philosopher Shannon Vallor has written of "moral deskilling", the atrophy of ethical capacities when tasks requiring them are delegated to machines. Chapter 4 described the same phenomenon in the language of human factors: operators who supervise automated systems lose the skills they need to intervene. In ethics the loss is more serious, because the skill in question is not only technical but moral. An army whose targeting is generated by software, and whose officers' role is to approve recommendations at volume, may gradually lose the institutional habit of asking whether a particular death is justified. The scholar Elke Schwarz, in her 2018 book Death Machines, argues that the framing of killing as a technical optimisation problem tends to crowd out precisely those questions. There is also the question of refusal. Human soldiers can, and occasionally do, refuse orders to fire on civilians or on surrendering enemies. That capacity has mattered at decisive moments in history, including in the collapse of authoritarian regimes whose soldiers would not fire on crowds. A system that executes its instructions without that capacity removes a last check against atrocity. Is restraint a form of disarmament? The most practically powerful objection to limits on autonomy is not philosophical. It is the claim that restraint is unilateral disarmament in disguise. If one state forgoes autonomous weapons and its adversary does not, the restrained state will lose the speed and mass that autonomy provides, and its soldiers and citizens will pay the price. On this view, ethical objections are a luxury available only to those who are not under attack, and a treaty would bind the law-abiding while leaving the unscrupulous free. The objection deserves a direct answer, because it is the one most often heard in defence ministries. There are three parts to the answer. First, the objection proves too much. The same argument could be made against every restriction in the law of war: against prohibitions on poison, on attacking hospitals, on killing prisoners. Each of those restraints, in some circumstances, gives an advantage to an adversary who ignores it. States accept them because they judge that a world in which such restraints are generally observed is better for everyone, including themselves, than a world without them, and because violations carry costs of their own. Second, the restrictions most commonly proposed do not touch the uses where the military case for autonomy is strongest. Defence against incoming missiles and drones, autonomy in navigation and in resistance to jamming, and the engagement of clearly military objects within bounded areas and times would all remain permitted under a two-tier approach. What would be prohibited, autonomous selection of human targets and systems whose effects cannot be foreseen, are not the capabilities on which any state's defence depends. Third, the absence of rules does not produce a stable advantage for anyone. It produces a race in which each side fields systems faster and with less testing than the last, in which the technology spreads to actors who will use it against the states that pioneered it, and in which the risk of unintended escalation rises for all. The arms-control agreements of the Cold War were negotiated between adversaries who trusted each other very little, precisely because each recognised that unconstrained competition threatened both. None of this makes the objection disappear. A state facing invasion will weigh the moral costs of autonomy differently from one that is not, and any international rule must be robust enough to survive that pressure. But the argument from necessity is an argument for drawing the lines carefully, not for refusing to draw them. What survives Some objections to autonomous weapons depend on assumptions about technical failure that may weaken over time. If they were the only objections, the case for regulation would be a case for better testing. But the strongest objections survive the assumption of technical improvement. The dignity and responsibility arguments are about the relationship between the decision and the decider, not about accuracy. The threshold and proliferation arguments are about the effects of autonomy on political and strategic behaviour, which improve with neither accuracy nor reliability, and may worsen as systems become cheaper and more capable. The deskilling argument identifies a cost that is paid even when the machines work as intended. These arguments also explain why the line most proposals draw is around anti-personnel autonomy and around the quality of human judgment, rather than around autonomy as such. A bounded system that destroys incoming missiles threatens none of these values seriously. A system that selects people for death on the basis of a profile, or a process in which human approval of such selections has become a formality, threatens all of them. The task for diplomacy, to which the next chapter turns, is to translate those distinctions into rules that states will accept. Chapter 8: Thirteen Years in Geneva International efforts to regulate autonomous weapons have now lasted longer than most of the wars that gave them urgency. They began in 2013, when the systems at issue were largely hypothetical, and they have reached a decisive moment in 2026, when the systems are in use. The story of those years is often told as one of failure: a consensus-based forum in Geneva where a handful of powerful states blocked progress while technology raced ahead. That account is partly true. But it misses how much the debate has moved, how far the substance of a possible agreement has been worked out, and why the remaining disagreement is so hard to bridge. From informal meetings to guiding principles The diplomatic process started from two directions at once. In April 2013 a coalition of non-governmental organisations launched the Campaign to Stop Killer Robots, now known as Stop Killer Robots, which by the mid-2020s counted more than three hundred member organisations in over seventy countries. In the same year Christof Heyns presented his report to the UN Human Rights Council calling for national moratoria. In November 2013 the states party to the Convention on Certain Conventional Weapons (CCW) agreed to hold an informal meeting of experts on lethal autonomous weapons systems the following year. The CCW was a natural forum. Concluded in 1980, it is a framework treaty to which protocols regulating specific weapons can be added, and it had produced the pre-emptive ban on blinding lasers in 1995. Its main drawback was equally clear: it operates by consensus, so any one of its more than one hundred and twenty states parties can block a decision. After three years of informal meetings, the CCW's Fifth Review Conference in 2016 established a formal Group of Governmental Experts (GGE), which first met in 2017. Its first significant product was a set of eleven guiding principles, agreed in 2019. They affirmed that international humanitarian law applies fully to all weapons systems, including autonomous ones; that "human responsibility for decisions on the use of weapons systems must be retained since accountability cannot be transferred to machines"; that human-machine interaction should ensure compliance with international law; and that legal reviews, risk assessments and mitigation measures are required. The principles were non-binding and general, but they established a vocabulary that has structured every later discussion. The next step proved much harder. At the Sixth Review Conference in December 2021, a majority of states wanted a mandate to negotiate a legally binding instrument. A small number of states, with Russia most prominent, prevented consensus. The GGE was given only a mandate to continue its work. The two-tier approach and the rolling text Frustration with the CCW pushed activity into other venues. Between 2023 and 2024, regional conferences hosted by Costa Rica, Trinidad and Tobago, the Philippines, Sierra Leone and others produced declarations supporting a new legally binding instrument. Austria hosted a large international conference in Vienna in April 2024 under the title "Humanity at the Crossroads". In October 2023, the UN Secretary-General, António Guterres, and the President of the ICRC, Mirjana Spoljaric, issued a joint appeal for states to conclude negotiations on a legally binding instrument by 2026, setting a deadline that has framed everything since. Guterres had repeatedly described machines with the power and discretion to take lives without human involvement as "politically unacceptable and morally repugnant". Within the CCW, meanwhile, the substance converged on what became known as the two-tier approach. The idea, championed in various forms by the ICRC and by groups of states, is that certain autonomous weapon systems should be prohibited outright, and all others regulated. The ICRC's May 2021 position, which has served as a reference point, proposed three elements: 1. Unpredictable autonomous weapon systems should be expressly ruled out, notably because of their indiscriminate effects. 2. The use of autonomous weapon systems to target human beings should be ruled out. 3. The design and use of autonomous weapon systems that would not be prohibited should be regulated, including through limits on the types of target, on the duration, geographical scope and scale of use, on situations of use, and through requirements for human supervision, timely intervention and deactivation. In November 2023 the CCW states parties gave the GGE a mandate for 2024 to 2026 to "further consider and formulate, by consensus, a set of elements of an instrument, without prejudging its nature". The phrase "without prejudging its nature" was the compromise that made agreement possible: states favouring a treaty could read it as preparation for one, and states opposed could read it as compatible with a non-binding outcome. Under the chairmanship of Ambassador Robert in den Bosch of the Netherlands, the GGE worked on a "rolling text" of elements, revised between sessions, covering a characterisation of autonomous weapon systems, the application of humanitarian law, prohibitions, regulations, human control and responsibility. The General Assembly steps in Because the CCW operates by consensus, states seeking faster progress turned to the UN General Assembly, where decisions are taken by majority vote. Austria, with a cross-regional group of co-sponsors, tabled the first resolution specifically on lethal autonomous weapons systems in 2023. The votes on that and subsequent resolutions, set out in Table 5, have shown consistent and overwhelming majorities. Table 5. UN General Assembly votes on resolutions on autonomous weapon systems, 2023-2025. Year Resolution and stage In favour Against Abstentions 2023 Res. 78/241, plenary (22 December) 152 4 11 2024 Draft L.77, First Committee (5 November) 161 3 13 2025 Draft L.41, First Committee (November) 156 5 8 Source: UN General Assembly records as reported by Stop Killer Robots; votes against in 2023 were Belarus, India, Mali and Russia. The 2023 resolution asked the Secretary-General to seek the views of states and others and to report back; his 2024 report collected an unusually large number of submissions. The 2024 resolution mandated two days of open informal consultations in New York, held in May 2025, which for the first time allowed all UN member states, and not only CCW parties, to discuss the issue without any state having a veto; according to Stop Killer Robots, ninety-six states took part. The 2025 resolution, adopted in the First Committee for the third successive year, called on the CCW to complete elements of an instrument, while stopping short of mandating negotiations elsewhere. The General Assembly has also opened a parallel track on artificial intelligence in the military domain more broadly, beyond weapons. A resolution adopted in late 2025 mandated discussions among states in Geneva, which took place in June 2026. That track matters for the argument of this book, since decision-support systems of the kind described in Chapters 4 and 5 fall more naturally under it than under the autonomous weapons heading. Stop Killer Robots and others have criticised the military AI discussions for producing only principles of "responsible use" with a permissive approach. National positions Behind the votes lie sharply different national positions. Table 6 summarises those of several key states. Table 6. Selected national positions on autonomous weapon systems. State On a new binding instrument Key national policy Notable point United States Opposed DoD Directive 3000.09 (2023) Led 2023 Political Declaration on military AI Russia Opposed Existing law sufficient Blocked negotiating mandate in 2021 China Narrow ban on use, not development 2018 CCW position paper Five-part definition limits scope India Sceptical Existing law sufficient Voted against 2023 UNGA resolution United Kingdom Opposed at present "Context-appropriate human involvement" Says it has no fully autonomous systems Austria Strongly supportive Hosts Vienna conference, leads UNGA text Main sponsor of UNGA resolutions Source: national statements to the CCW and UN General Assembly; government policy documents. The United States position combines domestic regulation with opposition to a new treaty. Directive 3000.09, revised in January 2023, requires that autonomous weapon systems be designed to allow "appropriate levels of human judgment over the use of force", and that systems not otherwise exempted undergo senior-level review, by the Under Secretary of Defense for Policy, the Under Secretary of Defense for Research and Engineering and the Vice Chairman of the Joint Chiefs of Staff, before formal development and again before fielding. Internationally, the United States launched a Political Declaration on Responsible Military Use of Artificial Intelligence and Autonomy in 2023, which more than fifty states endorsed. The declaration sets out non-binding principles: legal reviews, appropriate training, auditable methodologies, and the ability to disengage systems that demonstrate unintended behaviour. Critics note that it says nothing about prohibitions and leaves "appropriate" undefined. The 2026 dispute between the Department of Defense and Anthropic, described in Chapter 5, suggested that the current administration regards restrictions on fully autonomous weapons as an unacceptable constraint even when imposed by a supplier. The United Kingdom has stated that it does not possess fully autonomous weapon systems and has no intention of developing them, while opposing a new treaty. A House of Lords select committee report of December 2023, Proceed with Caution, urged the government to ensure human control at every stage and to lead on an effective international instrument. France's Defence Ethics Committee concluded in 2021 that France should renounce fully autonomous lethal weapons while permitting partially autonomous systems under human command. The Netherlands and Germany have supported the two-tier approach within the CCW. Many states of the Global South, including the African Group of states, Brazil and much of Latin America and the Caribbean, have been among the strongest supporters of a legally binding instrument. Their argument is partly humanitarian and partly about power: autonomous weapons will be developed by a few states and used, as drones have been, largely in the territories of others. Outside the UN, the Netherlands and the Republic of Korea co-hosted summits on Responsible AI in the Military Domain (REAIM) in The Hague in February 2023 and in Seoul in September 2024, producing a "Call to Action" and a "Blueprint for Action" endorsed by several dozen states. These processes have the virtue of including industry and of addressing military AI broadly, and the limitation of producing only voluntary commitments. Engineers, companies and the private lines States are not the only actors drawing lines. The people and companies who build artificial intelligence have, at intervals, tried to set their own limits, with mixed results. In 2018 several thousand Google employees signed a letter protesting the company's work on Project Maven, a Pentagon programme using machine learning to analyse drone surveillance footage. Google announced that it would not renew the contract and published a set of AI principles that included a commitment not to design or deploy AI for weapons. In February 2025 the company revised those principles and removed that commitment, citing a changed geopolitical landscape. The trajectory was typical. Through the late 2010s many large technology firms kept a public distance from weapons work; by the mid-2020s, with defence budgets rising and a growing sense of strategic competition with China, most had moved toward it, and a new generation of defence-technology companies built their business models around military AI. The 2026 dispute between Anthropic and the Pentagon, described in Chapter 5, was the most visible exception. A company insisted on a contractual prohibition on use of its models in fully autonomous weapons and mass domestic surveillance, and the government treated that insistence as a threat. Whatever one thinks of the merits, the episode illustrated two things. Private restrictions can matter, because states increasingly depend on commercial technology. And private restrictions are fragile, because they can be overridden by government pressure, abandoned by competitors willing to accept fewer conditions, or simply dropped when commercial conditions change. Stop Killer Robots, in a statement in June 2026, also raised concerns about the growing presence of defence-AI firms in international governance forums, where their commercial interests in permissive rules might shape the outcome. The lesson for the diplomacy is that voluntary corporate commitments, like voluntary state commitments, are no substitute for binding public rules. They can show what is possible and signal where informed people see danger. They cannot hold the line alone. The 2026 endgame The GGE held its final sessions of the mandate in Geneva from 2 to 6 March and from 31 August to 4 September 2026. The March session took place days after the start of the American and Israeli strikes on Iran, and the reports of AI in targeting described in Chapter 5 shaped its atmosphere. According to Stop Killer Robots, more than seventy states expressed support for moving to negotiations on the basis of the chair's rolling text. The final session was, in the campaign's account, a hard-fought one. A majority of states supported moving toward negotiations; Algeria, on behalf of the African Group, and Brazil, on behalf of forty-seven states, delivered statements confirming their willingness to negotiate. A minority of militarily powerful states opposed legally binding rules and sought to weaken the elements, in formal sessions and in informal consultations that ran through lunches and evenings. The report was adopted in the early hours of Saturday, 5 September. According to Stop Killer Robots' account, it retained a workable characterisation of autonomous weapon systems, recognition of the need for human control, core measures to give effect to that control, and the affirmation that responsibility cannot be transferred to machines. Language on ethical considerations and on the concepts of reliability, predictability, traceability and explainability was deleted. On 25 August 2026, shortly before that session, Guterres and the ICRC President renewed their joint appeal. "We are dangerously close to crossing a moral red line," they said, describing it as "the autonomous targeting of humans by machines". They called on states to adopt specific prohibitions and restrictions, and identified the CCW's Seventh Review Conference as "the clearest path available for States to decide to move to negotiate a legally binding instrument". That conference meets in Geneva from 16 to 20 November 2026. Its choice is stark. It can adopt a mandate to negotiate an instrument, presumably a new CCW protocol, on the basis of the elements the GGE has agreed. It can extend discussions again. Or, if consensus fails, it can produce nothing, in which case the majority of states will face a decision about whether to pursue negotiations outside the CCW, as states did for anti-personnel landmines in the Ottawa process of 1996 and 1997 and for cluster munitions in the Oslo process of 2007 and 2008. Those precedents cut both ways. The Ottawa and Oslo treaties were negotiated without the major military powers, and the United States, Russia and China have never joined either. Yet they changed behaviour well beyond their parties, by stigmatising the weapons and shaping military practice and procurement. They have also proved vulnerable: in 2025 several states bordering Russia, including Poland, Finland and the Baltic states, announced their withdrawal from the Ottawa Convention, citing the threat from Russia. A treaty on autonomous weapons negotiated without the states that build them would face the same strengths and the same fragility. Why agreement is hard The obstacle in Geneva is not ignorance. The technical and legal issues have been studied for thirteen years, and the elements under discussion reflect a sophisticated understanding of them. The obstacle is that the states most invested in military autonomy see it as central to their future security, and do not wish to bind themselves to limits their rivals might evade. Each sees the others racing. The United States justifies Replicator by reference to China's mass; China watches American investment and draws its own conclusions; Russia and Ukraine are demonstrating daily what cheap autonomy can do. Verification compounds the problem. Unlike nuclear weapons or chemical agents, autonomy is mostly software. The same drone can be operated with or without autonomous targeting, and inspecting hardware reveals little. A treaty would have to rely heavily on transparency measures, national reviews, and the reputational cost of violation, rather than on intrusive inspection. The conclusion that follows draws these threads together and argues for what an instrument worth having would actually need to contain. Conclusion The argument of this book has been that the central danger of military autonomy lies not in machines seizing decisions from humans in a single step, but in the gradual compression of human judgment, by speed, by scale and by the brittleness of machine perception, until "human control" persists only as a formality. If that diagnosis is right, it has consequences for what an international response should contain, and for how it should be judged. Five follow. Regulate conditions, not labels The first consequence is that an instrument organised around the question "is this an autonomous weapon?" will miss much of what matters. The cases examined in Chapter 5 show the erosion of judgment happening most visibly in systems that are not autonomous weapons under any official definition: target recommendation software in Gaza, a commercial language model reportedly used in planning strikes on Iran. Meanwhile, some systems that clearly are autonomous, such as ship-defence guns engaging incoming missiles, raise few of the concerns that drive the debate. An effective regime therefore needs to regulate the conditions under which force is applied, whatever the label on the system: the time available for human judgment, the information available to the human, the bounds within which a machine may act, and the traceability of what was done. That points to an instrument with a scope wide enough to reach decision-support systems used in lethal targeting, or to a parallel set of rules for them developed in the General Assembly's track on AI in the military domain. Leaving them outside both would leave the most consequential current practice unregulated. Draw the hard line where the arguments converge The second consequence concerns prohibitions. The legal, technical and ethical analyses of Chapters 2, 6 and 7 converge on two categories. Autonomous systems whose effects cannot be reasonably foreseen and limited are already incompatible with the prohibition of indiscriminate attacks; an explicit ban would make that implication concrete. And autonomous systems designed or used to select and attack human beings combine every weakness identified in this book: the invisibility of legal status, the clustering of error, the dignity and responsibility objections, and the potential for selective mass harm. The ICRC proposed prohibiting both in 2021. Nothing in the five years since has weakened the case, and the battlefield record has strengthened it. A prohibition on anti-personnel autonomy would cost the major military powers relatively little in their most important missions, which concern ships, aircraft, missiles, vehicles and air defences. Its value is disproportionate to its cost, because it addresses the scenario most likely to produce atrocity and proliferation. It would also have to be drafted carefully to reach systems that attack vehicles or buildings because of the persons believed to be inside them; otherwise the line would be evaded by redefining the target. Make "human control" measurable The third consequence is that regulations for permitted systems must give "human control" content that can be checked. Chapter 4 set out six conditions: understanding, information, time, foreseeability, authority to refuse, and answerability. Each can be translated into obligations. Limits on target types, area, duration and scale make behaviour foreseeable. Requirements on what information is presented to an approving human, and minimum review periods for deliberate targeting, protect time and understanding. Training requirements and independent verification of system performance address the gap between the confidence of a model and its reliability. Mandatory logging and retention of records make investigation possible. These obligations are unglamorous. They resemble aviation safety rules more than arms-control prohibitions. That is their strength. They target the mechanisms through which judgment is eroded, and they do not depend on resolving philosophical questions about what autonomy is. It is a measure of how much such obligations would bind that the concepts underlying several of them, predictability, reliability, traceability and explainability, were among the elements reportedly deleted from the Group of Governmental Experts' report in September 2026 under pressure from a minority of states. Accept imperfect verification The fourth consequence is that the absence of intrusive verification should not be treated as a reason for inaction. Chapter 5 showed how hard it is for outsiders to know when autonomy has been used. Chapter 8 noted that autonomy is mostly software and cannot be inspected the way warheads can. Opponents of a treaty sometimes argue that unverifiable rules would bind only the conscientious. But many of the most successful humanitarian norms have operated without strong verification. The prohibitions on anti-personnel mines and cluster munitions rely mostly on transparency reports, national implementation and stigma. Their force comes from making certain practices shameful and costly to acknowledge, and from shaping what militaries buy, train for and build into doctrine. A rule against autonomous anti-personnel targeting would work in the same way. It would not stop every violation. It would change what states are prepared to develop openly, what companies are prepared to sell, and what militaries are prepared to describe as normal. Verification can also be strengthened from within. Requirements to keep logs, to conduct and share the outlines of weapons reviews, and to report on national measures create an evidentiary trail that investigators, courts and other states can later use. The deletion of traceability from the 2026 text matters for this reason. Choose the forum, but choose The fifth consequence concerns the decision before states in November 2026. The CCW's Seventh Review Conference, meeting in Geneva from 16 to 20 November, can launch negotiations on a binding instrument on the basis of the elements the GGE agreed. A handful of states can prevent it from doing so. If that happens, the majority that has voted year after year in the General Assembly for action will face a choice it has deferred for thirteen years: whether to negotiate elsewhere, without the states that most need to be bound. The case for staying in the CCW is that an instrument the major powers accept, even a weaker one, may shape their behaviour more than a strong one they reject. The case for moving is that the CCW's consensus rule has allowed a few states to delay indefinitely while the technology spread, and that the landmine and cluster munition treaties shaped the conduct of states that never joined them. The withdrawal of several European states from the landmine treaty in 2025 shows that such norms are not permanent. It does not show that they were worthless: for a quarter of a century they saved many lives. Either path is defensible. What is not defensible is to treat the absence of a perfect forum as a reason to keep talking while the practices described in this book become the normal way of waging war. The incentives toward autonomy, set out in Chapter 1, operate continuously. Every year without rules is a year in which doctrine, procurement and expectation harden around the systems that exist. What remains in human hands There is a temptation, in writing about technology and war, to treat the future as already determined: the machines are coming, the arms race is under way, nothing can be done. The record does not support that fatalism. States have, at various points, decided not to build or use weapons that were technically available to them, and have stuck to those decisions for long periods. They have done so when they came to believe that a weapon threatened something worth more than the advantage it offered. What autonomous weapons threaten is the principle that when a human being is killed in war, another human being has judged that death necessary and lawful and will answer for it. That principle has always been imperfectly honoured. It has nevertheless been the foundation on which the law of war, military professionalism and the moral standing of soldiers rest. The engineering described in this book does not make it obsolete. It makes it optional. Whether it remains the rule is, for a little while longer, a decision people can still make. Notes The following sources are cited by name in the text. They are listed in approximately the order in which they first appear. 1. United Nations Security Council, Final report of the Panel of Experts on Libya established pursuant to resolution 1973 (2011), S/2021/229, 8 March 2021. 2. Vincent Boulanin and Maaike Verbruggen, Mapping the Development of Autonomy in Weapon Systems, Stockholm International Peace Research Institute, November 2017. 3. Human Rights Watch and International Human Rights Clinic, Harvard Law School, Losing Humanity: The Case against Killer Robots, November 2012. 4. United States Department of Defense, Directive 3000.09, "Autonomy in Weapon Systems", 21 November 2012; revised 25 January 2023. 5. International Committee of the Red Cross, "ICRC Position on Autonomous Weapon Systems", 12 May 2021. 6. People's Republic of China, Position Paper submitted to the CCW Group of Governmental Experts on Lethal Autonomous Weapons Systems, CCW/GGE.1/2018/WP.7, April 2018. 7. Chuan Guo, Geoff Pleiss, Yu Sun and Kilian Q. Weinberger, "On Calibration of Modern Neural Networks", Proceedings of the 34th International Conference on Machine Learning, 2017. 8. Marco Tulio Ribeiro, Sameer Singh and Carlos Guestrin, "'Why Should I Trust You?': Explaining the Predictions of Any Classifier", Proceedings of the 22nd ACM SIGKDD Conference, 2016. 9. Robert Geirhos et al., "Shortcut Learning in Deep Neural Networks", Nature Machine Intelligence 2 (2020). 10. Christian Szegedy et al., "Intriguing Properties of Neural Networks", International Conference on Learning Representations, 2014. 11. Kevin Eykholt et al., "Robust Physical-World Attacks on Deep Learning Visual Classification", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018. 12. Joy Buolamwini and Timnit Gebru, "Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification", Proceedings of Machine Learning Research 81 (2018). 13. Jo Becker and Scott Shane, "Secret 'Kill List' Proves a Test of Obama's Principles and Will", The New York Times, 29 May 2012. 14. Arthur Holland Michel, The Black Box, Unlocked: Predictability and Understandability in Military AI, United Nations Institute for Disarmament Research, 2020. 15. Craig W. Reynolds, "Flocks, Herds, and Schools: A Distributed Behavioral Model", Computer Graphics 21, no. 4 (1987). 16. United States Department of Defense, "Department of Defense Announces Successful Micro-Drone Demonstration", press release, 9 January 2017. 17. Kathleen Hicks, remarks at the National Defense Industrial Association Emerging Technologies conference, 28 August 2023, announcing the Replicator initiative. 18. U.S. Commodity Futures Trading Commission and U.S. Securities and Exchange Commission, Findings Regarding the Market Events of May 6, 2010, 30 September 2010. 19. Paul Scharre, Army of None: Autonomous Weapons and the Future of War (New York: W. W. Norton, 2018). 20. Zachary Kallenborn, "Are Drone Swarms Weapons of Mass Destruction?", U.S. Air Force Center for Strategic Deterrence Studies, Future Warfare Series no. 60, 2020. 21. Lisanne Bainbridge, "Ironies of Automation", Automatica 19, no. 6 (1983). 22. Raja Parasuraman and Victor Riley, "Humans and Automation: Use, Misuse, Disuse, Abuse", Human Factors 39, no. 2 (1997). 23. Kathleen L. Mosier, Linda J. Skitka and colleagues, studies of automation bias in aviation decision-making, International Journal of Aviation Psychology, 1990s. 24. M. L. Cummings, "Automation Bias in Intelligent Time Critical Decision Support Systems", AIAA 1st Intelligent Systems Technical Conference, 2004. 25. John K. Hawley, Patriot Wars: Automation and the Patriot Air and Missile Defense System, Center for a New American Security, January 2017. 26. Defense Science Board, Report of the Defense Science Board Task Force on Patriot System Performance, January 2005. 27. Michael C. Horowitz and Paul Scharre, Meaningful Human Control in Weapon Systems: A Primer, Center for a New American Security, March 2015. 28. Vincent Boulanin, Neil Davison, Netta Goussac and Moa Peldán Carlsson, Limits on Autonomy in Weapon Systems: Identifying Practical Elements of Human Control, SIPRI and ICRC, June 2020. 29. Yuval Abraham, "'A Mass Assassination Factory': Inside Israel's Calculated Bombing of Gaza", +972 Magazine and Local Call, 30 November 2023. 30. Yuval Abraham, "'Lavender': The AI Machine Directing Israel's Bombing Spree in Gaza", +972 Magazine and Local Call, 3 April 2024, including the response of the Israel Defense Forces. 31. Stop Killer Robots, "Reports of AI-enabled Targeting in Iran Bring Real-World Impacts to the UN", 6 March 2026, summarising reporting in The Wall Street Journal, The Washington Post and other outlets. 32. Stop Killer Robots, "Nothing Less than Negotiations", 8 September 2026, on the final session of the CCW Group of Governmental Experts. 33. United Nations Secretary-General and President of the ICRC, joint appeal on autonomous weapon systems, 5 October 2023; renewed joint call, 25 August 2026. 34. Human Rights Watch and International Human Rights Clinic, Harvard Law School, Mind the Gap: The Lack of Accountability for Killer Robots, April 2015. 35. Robert Sparrow, "Killer Robots", Journal of Applied Philosophy 24, no. 1 (2007). 36. Christof Heyns, Report of the Special Rapporteur on extrajudicial, summary or arbitrary executions, UN Doc. A/HRC/23/47, 9 April 2013. 37. Human Rights Watch and International Human Rights Clinic, Harvard Law School, A Hazard to Human Rights: Autonomous Weapons Systems and Digital Decision-Making, April 2025. 38. International Court of Justice, Legality of the Threat or Use of Nuclear Weapons, Advisory Opinion, 8 July 1996. 39. Ronald C. Arkin, Governing Lethal Behavior in Autonomous Robots (Boca Raton: CRC Press, 2009). 40. Peter Asaro, "On Banning Autonomous Weapon Systems: Human Rights, Automation, and the Dehumanization of Lethal Decision-Making", International Review of the Red Cross 94, no. 886 (2012). 41. Paul W. Kahn, "The Paradox of Riskless Warfare", Philosophy and Public Policy Quarterly 22, no. 3 (2002). 42. Future of Life Institute, "Autonomous Weapons: An Open Letter from AI and Robotics Researchers", 28 July 2015. 43. Elke Schwarz, Death Machines: The Ethics of Violent Technologies (Manchester: Manchester University Press, 2018). 44. Shannon Vallor, "Moral Deskilling and Upskilling in a New Machine Age", Philosophy and Technology 28 (2015). 45. CCW Group of Governmental Experts, Guiding Principles affirmed by the Group, annexed to CCW/MSP/2019/9, 2019. 46. UN General Assembly resolution 78/241, "Lethal Autonomous Weapons Systems", 22 December 2023. 47. Stop Killer Robots, "161 States Vote Against the Machine at the UN General Assembly", November 2024; and "156 States Support UNGA Resolution on Autonomous Weapons", November 2025. 48. House of Lords AI in Weapon Systems Committee, Proceed with Caution: Artificial Intelligence in Weapon Systems, HL Paper 16, December 2023. 49. United States Department of State, Political Declaration on Responsible Military Use of Artificial Intelligence and Autonomy, November 2023. 50. Pope Leo XIV, encyclical letter Magnifica Humanitas, 2026, as quoted by Stop Killer Robots, 27 May 2026. 51. Reaching Critical Will (Women's International League for Peace and Freedom), meeting pages for the 2026 CCW Group of Governmental Experts and the Seventh CCW Review Conference. Further Reading Paul Scharre, Army of None: Autonomous Weapons and the Future of War (2018).** The most complete general account, by a former Pentagon official who helped draft the original Directive 3000.09. Strong on history and military logic, fair to all sides. Paul Scharre, Four Battlegrounds: Power in the Age of Artificial Intelligence (2023).** Places military AI in the wider contest between the United States and China over data, computing power, talent and institutions. International Committee of the Red Cross, "ICRC Position on Autonomous Weapon Systems" (2021). A short, authoritative statement of the two-tier approach that has shaped the diplomatic debate. Freely available from the ICRC website. Vincent Boulanin, Neil Davison, Netta Goussac and Moa Peldán Carlsson, Limits on Autonomy in Weapon Systems (SIPRI and ICRC, 2020).** The clearest practical analysis of what human control requires, organised around controls on parameters, environment and human-machine interaction. Arthur Holland Michel, The Black Box, Unlocked (UNIDIR, 2020).** A lucid explanation of why predictability and understandability in military AI are harder than they sound, written for policymakers. Human Rights Watch and Harvard International Human Rights Clinic, Losing Humanity (2012), Mind the Gap (2015) and A Hazard to Human Rights (2025).** The core advocacy case for prohibition, on humanitarian, accountability and human rights grounds respectively. Ronald C. Arkin, Governing Lethal Behavior in Autonomous Robots (2009).** The most serious argument that autonomous systems might behave more ethically than human soldiers. Essential for understanding the strongest version of the opposing view. Robert Sparrow, "Killer Robots", Journal of Applied Philosophy (2007).** The classic statement of the responsibility-gap argument. Elke Schwarz, Death Machines: The Ethics of Violent Technologies (2018).** A philosophical critique of how technological framing reshapes the ethics of killing. John K. Hawley, Patriot Wars (CNAS, 2017).** An insider's account of automation, trust and fratricide in the Patriot system; the best single case study of the human-in-the-loop problem. Lisanne Bainbridge, "Ironies of Automation" (1983). Four pages that anticipate most of what has since been learned about humans supervising machines. Stop Killer Robots and Reaching Critical Will. The two most useful sources for following the diplomacy as it happens, including statements, documents and session reports from Geneva and New York. Hashtags: #AlgorithmicWarfare #LethalAutonomousWeapons #AutonomousWeaponSystems #LAWS #MilitaryArtificialIntelligence #AIInWarfare #MilitaryAI #AutonomousWarfare #EthicsOfWar #MilitaryEthics #AIEthics #ResponsibleMilitaryAI #EthicalAI #AutonomousWeaponsEthics #HumanControl #MeaningfulHumanControl #HumanInTheLoop #HumanOnTheLoop #HumanOutOfTheLoop #HumanMachineInteraction #HumanMachineTeaming #HumanJudgment #MilitaryDecisionMaking #AutomatedDecisionSupport #AITargetingSystems #AlgorithmicTargeting #TargetRecognition #MachinePerception #ComputerVision #MachineLearning #AIModelReliability #DistributionShift #ShortcutLearning #AdversarialDeception #AlgorithmicBias #AIUncertainty #AIExplainability #AITransparency #AutomationBias #HumanOversight #AutomationAccountability #MilitaryAccountability #CommandResponsibility #InternationalHumanitarianLaw #LawsOfArmedConflict #IHLCompliance #PrincipleOfDistinction #PrincipleOfProportionality #PrecautionInAttack #CivilianProtection #CivilianHarmPrevention #HumanRights #WeaponsLaw #Article36Reviews #WeaponsReview #AutonomousSystemsGovernance #AIGovernance #MilitaryTechnologyGovernance #ArmsControl #ArmsRegulation #DisarmamentDiplomacy #UnitedNations #ConventionOnCertainConventionalWeapons #CCW #GroupOfGovernmentalExperts #ICRC #SIPRI #UNIDIR #StopKillerRobots #AutonomousDrones #DroneWarfare #LoiteringMunitions #SwarmIntelligence #DroneSwarms #EmergentBehavior #SwarmAccountability #AutonomousSystemsSafety #AIWeaponsTesting #VerificationAndValidation #OperationalPredictability #HumanitarianDisarmament #EthicsOfAutonomy #CivilMilitaryTechnology #MilitaryInnovation #ResponsibleInnovation #FutureOfWarfare
- AI in Retail and E-Commerce (Personalization, Pricing, and Logistics)
Download the Book (PDF): Introduction A weather forecast does not change the weather. That simple fact is the quiet assumption behind most of what people learn about machine learning: you collect historical data, fit a model that predicts some outcome, measure how well it predicts on data it has not seen, and deploy it. If the model is good at predicting, it is a good model. Retail breaks that assumption at almost every turn. When a model predicts that a shopper will like a pair of running shoes and the website shows those shoes at the top of the page, the shopper's next click is partly caused by the prediction. When a pricing model concludes that demand for a blender is insensitive to price and raises it by eight percent, the sales that follow are generated by the model's own decision, and the data collected next month is data the model helped create. When a routing algorithm sends pickers down the same aisle at the same time, the congestion it causes becomes part of the travel times it will learn from tomorrow. In retail, the model is never merely an observer of the market. It is a participant. This booklet is built around that idea. Its argument is that artificial intelligence in retail and e-commerce succeeds or fails less on the accuracy of its predictions than on how well it handles the fact that its predictions are also interventions. The organizations that get durable value from machine learning in consumer markets are those that treat their models as decision-makers acting inside a feedback loop: they estimate causal effects rather than correlations, they run experiments rather than trusting backtests, they explore deliberately rather than exploiting blindly, and they build simulators and guardrails for the decisions too expensive to learn by trial and error. The ones that struggle usually have excellent models answering the wrong question. Three arenas, one problem The booklet examines three areas where retailers have invested most heavily and where the stakes are clearest. The first is personalization: deciding what each shopper sees, from the products in a recommendation carousel to the order of search results to the content of a promotional email. This is where machine learning entered retail at scale, through the recommendation engines that Amazon described publicly in 2003 and that the Netflix Prize turned into a research industry later in that decade. It is also where the gap between predictive accuracy and business value has been documented most thoroughly, because a recommender can be very good at predicting purchases that would have happened anyway. The second is pricing. Prices are the most direct lever a retailer has, and estimating how demand responds to them, the price elasticity, is one of the oldest problems in applied economics. Machine learning has changed what can be estimated and how often prices can be changed, but it has not changed the core difficulty: observed prices are not set at random, so the relationship between price and sales in historical data is contaminated by the reasons prices were set the way they were. Pricing is also where AI in retail has drawn the sharpest public and regulatory attention, from algorithmic collusion to what the United States Federal Trade Commission has called surveillance pricing. The third is logistics, and specifically the inside of the warehouse. Order picking, the work of walking or driving to storage locations to collect the items in customer orders, has long been estimated to account for roughly half of warehouse operating cost, and most of that is travel time. Choosing routes, batching orders and coordinating fleets of mobile robots are classical problems in operations research. Reinforcement learning promises to solve versions of them that classical methods handle poorly, particularly when many agents interact and conditions change by the minute. Here the feedback loop is physical: a route chosen for one picker changes the congestion every other picker experiences. A fourth topic runs through the middle of the book because it connects shoppers to products in a way older methods could not: visual search, the ability to photograph a lamp in a friend's living room and find something like it in a catalogue of millions. Visual search is a showcase for modern representation learning, and it is also a useful corrective, because it is one of the few retail applications where the core problem really is closer to perception than to decision-making. Seeing where the feedback-loop argument applies and where it does not is part of understanding it. What this booklet covers and what it leaves out The chapters move from the general to the particular. Chapter 1 sets out why retail is a closed-loop environment and what that does to data, beginning with demand forecasting, the foundation everything else rests on. Chapters 2 and 3 cover personalization: first the models that power recommendation and ranking, then the methods for measuring what those models actually change. Chapters 4 and 5 cover pricing: first the econometrics of elasticity estimation, then the systems that turn elasticity estimates into prices, and the limits, legal and ethical, on what those systems should do. Chapter 6 covers visual search. Chapters 7 and 8 cover the warehouse: first the order-picking problem and the classical methods that remain hard to beat, then reinforcement learning and the conditions under which it earns its considerable cost. Some important subjects are left out or treated only in passing. Generative AI shopping assistants and agentic commerce, in which software agents browse and buy on a consumer's behalf, are developing quickly in 2026, and they appear where they bear on the argument, but this is not a guide to them. Fraud detection, supply-chain network design, store layout, workforce scheduling and last-mile delivery each deserve their own treatment. The booklet also does not attempt a market survey of vendors. Its aim is to give a reader the concepts needed to understand any of these systems, judge claims made about them, and ask the questions that separate a model that helps from one that merely looks good on a dashboard. Who this is for The intended reader is someone who works near these systems without necessarily building them: a merchant, category manager, product manager, operations lead, analyst or executive in a retail or consumer business; a student of business or data science; a regulator or journalist trying to understand what algorithms in commerce actually do. Some familiarity with basic statistics helps. No mathematics beyond the idea of a regression is required, and where a technical idea matters, it is explained in words. A few short code fragments appear where they make a concept concrete; they can be skipped without losing the thread. Readers who build these systems should find the booklet useful too, though in a different way. Practitioners tend to know their own corner deeply, the ranking team its ranking metrics, the pricing team its demand models, and to know less about how adjacent teams handle the same structural problem. One of the arguments here is that the problem really is the same. The data scientist estimating price elasticity and the engineer evaluating a recommender are both trying to learn the effect of an action from data that past actions shaped. The warehouse engineer training a routing policy faces the identical problem in physical form. Seeing the shared structure makes each of the specific techniques easier to understand and harder to misuse. A note on evidence Retail AI is an area where public claims outrun public evidence. Companies describe their systems in blog posts, conference talks and earnings calls, and the numbers they cite, a ten percent improvement here, a double-digit lift there, are rarely accompanied by enough detail to evaluate. Academic research is more transparent but often uses public datasets that look little like a large retailer's data. The booklet draws on both, distinguishing claims that come from peer-reviewed work from those that come from company announcements, and it avoids inventing numbers where reliable ones are not available. Where a figure appears, its source appears with it, either in the text or in the Notes at the end. The result is, deliberately, a book about judgement as much as technique. The techniques change every few years. The structural fact that a retailer's models act on the market they are trying to understand does not change, and learning to work with it is the durable skill. Chapter 1: The Closed Loop Every retail decision leaves a trace in the data, and that trace is then used to make the next decision. This chapter describes how that loop works, why it makes retail data unusual, and what it implies for the most basic analytical task in the industry, forecasting demand. The rest of the booklet builds on the ideas here. Data that decisions wrote Consider a mid-sized online retailer selling home goods. Its data warehouse holds years of transactions: which customer bought which item, at what price, on what date, after seeing which pages. It is tempting to treat this as a record of consumer preferences, a natural history of demand that a sufficiently powerful model can mine. It is nothing of the sort. It is a record of how consumers responded to a particular sequence of choices the retailer made. Each of those choices shaped what was recorded. The merchandising team decided which products to stock, so no sales exist for products never carried. The pricing team decided what to charge, and usually not at random: prices were cut when inventory piled up, raised when a competitor ran out, discounted for holidays when demand was already high. The site's ranking system decided which items appeared on the first page of search results, and items on the first page get far more clicks than items on the third regardless of how much shoppers would like them. The replenishment system decided how much to order, and when it ordered too little, the item sold out and sales were recorded as zero even though shoppers wanted it. The marketing team decided whom to email, and they emailed the customers most likely to buy. None of this is a flaw in the data. It is what retail data is. But it means that almost every quantity a retailer wants to learn is confounded by its own past behavior. The question "what would happen if we did X" cannot be answered by looking at what happened when the retailer did X in the past, because the retailer did X in particular circumstances for particular reasons, and those circumstances and reasons also affected the outcome. Statisticians call this selection bias or confounding, and economists call it endogeneity. Machine learning practitioners have tended to call it distribution shift, or simply to ignore it, because in many classical machine learning tasks it does not arise. An image classifier trained to recognize dogs is not deployed into a world where its predictions change which animals exist. A spam filter comes closer, because spammers adapt, but even there the filter is reacting to an adversary rather than generating its own training labels. In retail, the model generates its own labels, constantly. It helps to distinguish three loops, because they appear in different parts of the business and call for somewhat different remedies. The first is the exposure loop. Shoppers can only buy what they see. A personalization or ranking system decides what each shopper sees, and the clicks and purchases it then observes are concentrated on the items it chose to show. If the next version of the model is trained on those clicks, it learns that the items it showed are popular, shows them more, and observes still more clicks on them. Left alone, this loop narrows the catalogue a retailer effectively sells, tilts toward items that were popular early, and makes the model look more accurate than it is, since it is partly predicting its own behavior. The second is the price loop. The retailer sets prices, observes sales, estimates how sales respond to price, and sets new prices. If historical prices were set in response to demand conditions the model cannot see, such as a promotional calendar that coincides with seasonal peaks, the estimated relationship between price and sales will be distorted, often badly. A naive regression of sales on price in retail data frequently finds that higher prices go with higher sales, not because shoppers like paying more but because retailers raise prices when demand is strong. The third is the operational loop. Forecasts determine inventory orders; inventory determines what is available; availability determines what can be sold; sales determine the next forecast. In the warehouse, a routing policy determines where pickers go, which determines congestion, which determines travel times, which become the data on which the next routing policy is tuned. Operational loops are often the most tractable of the three, because physical systems can be simulated with some confidence, but they are also where errors are most expensive, since a bad decision means a missed delivery or a pallet of spoiled goods. Demand forecasting as the foundation Before a retailer can personalize, price or route anything, it needs to know roughly how much of each item will be wanted, where and when. Demand forecasting is the least glamorous and most consequential application of machine learning in retail. It is also the clearest illustration of the closed loop, because the quantity being forecast, demand, is never directly observed. What is observed is sales, and sales equal demand only when there is enough stock to meet it. When an item sells out at noon, the afternoon's demand vanishes from the record. A forecasting model trained on raw sales history learns that this item sells about as much as the retailer stocked, which will lead the replenishment system to stock about the same amount again, which will cause it to sell out again. Statisticians call such data censored, and the correction, known as unconstraining or demand uncensoring, involves estimating what sales would have been had stock been available. Methods range from simple imputation using the sales rate before the stockout, to survival-analysis models that treat the stockout time as informative, to latent-variable models that estimate demand and availability jointly. For a grocery chain with thousands of fresh items and frequent stockouts, getting this right is worth more than almost any refinement in model architecture. Censoring is not the only distortion. When an item is unavailable, some shoppers buy a substitute, so the substitute's sales are inflated by the stockout and its forecast will be biased upward if this is not recognized. Promotions shift demand in time: a discount on laundry detergent pulls purchases forward from the following weeks, so the post-promotion dip is partly caused by the promotion, and a model that does not know this will read the dip as weakening demand. Cannibalization means that promoting one flavor of yogurt reduces sales of the others. Each of these effects is a consequence of the retailer's own decisions, and a forecasting system that cannot represent those decisions as inputs will confuse their consequences with changes in the underlying market. What the competitions taught The modern state of retail forecasting is well documented, unusually so, because of a public competition. The M5 competition, organized by Spyros Makridakis and colleagues in 2020, asked entrants to forecast daily unit sales for 3,049 products sold by Walmart across ten stores in three American states, organized into a hierarchy of more than 42,000 series from individual product-store combinations up to total sales. It was the first of the long-running M competitions to use real retail data at this granularity, and it was large enough that the results carry real weight. The headline findings were clear. The most accurate methods were machine learning models, overwhelmingly gradient-boosted decision trees implemented in LightGBM, rather than the classical statistical methods such as exponential smoothing that had performed strongly in earlier M competitions. The winning approaches did not fit a separate model to each series, as classical methods do, but trained global models across many series at once, letting a product with a short history borrow strength from similar products. They relied heavily on features describing the retailer's own decisions and the calendar: prices, promotional events, the timing of the American Supplemental Nutrition Assistance Program payments that shifted food purchasing in some states, holidays, and day of week. And the margins of improvement over simple benchmarks, while real, were not enormous; much of the gain came from careful feature engineering and validation rather than from model novelty. The lessons generalize. Retail forecasting is a problem with many related series, intermittent demand at the lowest levels (many products sell zero units on most days in most stores), and strong dependence on decisions the retailer controls. Global models handle the first two well. The third requires that the forecasting model take planned prices and promotions as inputs, which in turn means the forecast becomes a function the pricing and promotion teams can query: not "how much will we sell" but "how much will we sell if we do this." That shift, from forecast to conditional forecast, is the hinge on which the whole closed-loop argument turns. A forecast that conditions on planned actions is implicitly a causal model. It claims to know what would happen under actions that may not have been taken in that combination before. Whether it can be trusted in that role depends entirely on whether the historical relationship between those actions and sales reflects cause or merely correlation. The pricing chapters return to this problem in detail. Probabilistic forecasts and the newsvendor There is a second lesson, which matters as much as accuracy: retail decisions need distributions, not point forecasts. A replenishment decision for a perishable item is a bet. Order too much and the surplus is thrown away; order too little and sales and goodwill are lost. The classical analysis of this bet, the newsvendor problem, shows that the optimal order quantity is a particular quantile of the demand distribution, determined by the ratio of the cost of running short to the combined cost of running short and running over. For a high-margin item that keeps well, the optimal order sits high in the distribution; for a low-margin item that spoils, it sits low. This is why the large retailers and their technology suppliers moved over the past decade from point forecasts to probabilistic ones. Amazon's forecasting researchers published DeepAR in 2017 and the multi-horizon quantile recurrent forecaster in the same period, both designed to produce full predictive distributions across very large numbers of related series. The M5 competition ran a parallel uncertainty track asking for quantile forecasts. More recently, pretrained time-series foundation models, including Amazon's Chronos released in 2024, have offered a way to produce reasonable probabilistic forecasts for series with little history, by learning general patterns from large collections of time series. In retail practice these models are usually combined with, rather than substituted for, gradient-boosted models that encode the retailer's own pricing and promotional features, since a pretrained model has no way of knowing that next Tuesday's discount is planned. A forecast that gives a distribution also makes the loop more visible. When the forecast says that demand for an item next week is likely to be between 40 and 70 units, and the retailer stocks 55, then the recorded sales will be at most 55. The distribution of observed outcomes depends on the decision. Evaluation of the forecast must account for that, by comparing forecasts against uncensored demand estimates or by restricting evaluation to periods when stock was ample, or the retailer will systematically reward models that predict whatever it chose to stock. A worked illustration A small example shows how these pieces fit together, and how the loop can mislead even a careful team. The numbers are illustrative, not drawn from any particular retailer. A grocery chain bakes croissants in-store each morning. Each costs 40 cents to make and sells for 1.50 dollars; anything unsold at closing is discarded. Running short costs the lost margin of 1.10 dollars per croissant; running over costs the 40 cents of waste. The newsvendor ratio is therefore 1.10 divided by 1.50, about 0.73: the store should bake enough to meet demand on roughly 73 percent of days, accepting a stockout on the other 27. Suppose the forecasting team trains a model on last year's sales, and in one store the bakers, cautious about waste, consistently baked about 60 croissants. On most days, every croissant sold, often by early afternoon. The sales history therefore shows a steady 55 to 60 a day. A model trained on it, however sophisticated, forecasts 55 to 60, and the 73rd percentile of that forecast is about 60. The store bakes 60 again. The model is accurate, in the sense that sales match its forecast almost exactly, and it is wrong, because true demand was higher every day it sold out. The accuracy is an artefact of the decision. The fix requires information the sales total does not contain: the time of each sale. If croissants ran out at 1 p.m. on a day when the store is open until 9, and on days with ample stock about 40 percent of croissant sales happen after 1 p.m., then that day's demand was nearer 100 than 60. A model trained on uncensored estimates of that kind would recommend baking more, the store would sell more and waste a little more, and profit would rise. The same reasoning applies to any item with frequent stockouts, from fresh fish to the season's popular toy, and the information needed, timestamps of sales and of stock levels, is usually already in the retailer's systems. It simply has to be used. The example also shows why point accuracy is the wrong scorecard. The model that forecast 60 had a lower error on recorded sales than a model that forecast 90 would have had. Judged by the usual metrics, the worse model wins. Judged by the profit from the decisions each would drive, it loses. Hierarchies and coherence Retail forecasts are needed at many levels at once. Store replenishment needs product-by-store forecasts for each day. Distribution centers need product-level forecasts aggregated across the stores they serve. Buyers need category-level forecasts for weeks and months ahead. Finance needs total sales. If each level is forecast separately, the numbers will not add up: the sum of the store forecasts will differ from the distribution center forecast, which will differ from the category plan. Forecast reconciliation addresses this by forcing forecasts across the hierarchy to be coherent, so that the parts sum to the whole. Methods developed by Rob Hyndman and colleagues, notably the minimum-trace approach, combine forecasts from all levels into a coherent set, typically improving accuracy at several levels at once, because aggregate series are smoother and easier to forecast while detailed series carry local information. The practical consequence is that a retailer's forecasting system is not a model but a structure of models, and the coherence of that structure matters as much as any single forecast within it, since the different levels drive decisions made by different teams who must be able to trust that they are working from the same picture. The cost of ignoring the loop The consequences of treating retail data as if the retailer did not shape it are rarely dramatic in any single week, which is why they persist. They show up as a slow drift. Recommendation systems that become steadily more focused on bestsellers while the long tail of the catalogue goes unsold. Pricing models whose elasticity estimates swing from quarter to quarter for no reason anyone can identify. Forecasts that are accurate on the dashboard and yet coincide with persistent stockouts of the same items. Warehouse routing tuned against travel-time data that reflects the previous routing policy's congestion, so that each new policy is optimized for a world the old one created. A useful diagnostic question for any retail model is: if this model were perfect at predicting the data it was trained on, would decisions based on it be good? For a pure perception task the answer is yes. For most retail tasks the answer is "only if the training data were generated in a way that lets prediction stand in for causation." That condition is rarely met by accident. It must be engineered, by randomizing some decisions, by recording the context and reasons behind every decision, by building models that separate the effect of actions from the circumstances in which they were taken, and by testing changes in controlled experiments before trusting them. What follows The chapters that follow take this structure into each domain. The tools differ. Personalization leans on large-scale online experiments and bandit algorithms. Pricing leans on econometric methods for estimating causal effects from observational data, increasingly combined with machine learning in methods such as double machine learning. Warehouse logistics leans on simulation and reinforcement learning, where an agent learns a policy by acting in a model of the environment rather than in the warehouse itself. Visual search, which comes in the middle, is the exception that clarifies the rule: matching a photo to a product is mostly a perception problem, and the closed loop matters there only at the point where search results are ranked and shown. Across all of them, the practical question is the same. Not "how accurate is the model" but "what happens when we act on it, and how do we know." The retailers that answer the second question well are the ones whose investments in machine learning keep paying off after the first year. Chapter 2: How Personalization Models Work Personalization is where machine learning first became a visible part of shopping, and where it still touches consumers most often. Every product carousel labelled "customers also bought", every reordered search results page, every "picked for you" email is the output of a model deciding what one shopper should see out of a catalogue that may run to hundreds of millions of items. This chapter explains how those models are built, how they have evolved, and what makes retail personalization different from recommending films or songs. The next chapter takes up the harder question of what these models actually accomplish. From neighbors to factors The first widely deployed approach was collaborative filtering, which rests on a simple intuition: people who agreed in the past will tend to agree in the future. If two shoppers bought many of the same items, then items one of them bought that the other has not are good candidates to recommend. Early systems compared users directly, but that scaled badly, since a large retailer has tens or hundreds of millions of customers and each one's set of neighbors changes with every purchase. Amazon's engineers described a different approach in a 2003 paper by Greg Linden, Brent Smith and Jeremy York: compare items rather than users. For each item, compute which other items tend to be bought by the same customers, and store that list. When a shopper views or buys an item, show the most similar items. Item-to-item similarity is more stable than user-to-user similarity, because an item's purchase pattern changes slowly, and it can be computed offline and looked up instantly. The approach also produces explanations a shopper can understand, "because you bought this", which turned out to matter for trust. The next step came from the Netflix Prize, the open competition Netflix ran from 2006 to 2009 offering a million dollars for a ten percent improvement in its rating predictions. The winning methods drew heavily on matrix factorization, popularized for recommendation by Yehuda Koren, Robert Bell and Chris Volinsky. The idea is to represent each user and each item as a short vector of numbers, perhaps fifty or a hundred, such that the dot product of a user's vector with an item's vector predicts how much the user will like the item. The vectors are learned from data, and their dimensions often turn out to correspond loosely to recognizable attributes, such as price tier or style, though nothing forces them to. Retail differs from the Netflix setting in one important way: shoppers rarely rate things. What retailers have is implicit feedback, records of views, clicks, add-to-carts and purchases. A purchase is a reasonably strong positive signal, but the absence of a purchase is ambiguous: the shopper may not have liked the item, or may never have seen it. Work by Yifan Hu, Yehuda Koren and Chris Volinsky in 2008 adapted matrix factorization to implicit feedback by treating every observed interaction as a positive with a confidence weight and every unobserved one as a weak negative. Variants of that approach remain a strong baseline today, often surprisingly competitive with far more elaborate neural models when properly tuned. Notice that the ambiguity of the missing purchase is the exposure loop from Chapter 1 appearing at the level of the loss function. Whether an unobserved interaction means "not interested" or "never shown" depends on what the retailer showed, and a model that cannot distinguish them will learn that the items it never displayed are unpopular. The two-stage architecture Matrix factorization scores every item for every user, which becomes impractical once the catalogue grows large and the model includes many features. Modern industrial recommenders almost universally split the problem into two or more stages, an architecture laid out clearly in a 2016 paper by Paul Covington, Jay Adams and Emre Sargin describing YouTube's system and since adopted throughout e-commerce. The first stage, candidate generation or retrieval, narrows millions of items to a few hundred plausible ones in milliseconds. The standard modern approach is the two-tower model: one neural network turns the user and their context (recent activity, location, device, time) into a vector, another turns each item (its title, category, price, image features, popularity) into a vector in the same space, and the model is trained so that users end up close to items they interact with. Because item vectors can be computed in advance and stored in an index built for fast nearest-neighbor search, retrieval reduces to finding the items whose vectors lie closest to the user's. Retailers usually run several retrieval sources in parallel, an embedding-based one, an item-to-item one, a "trending in your area" one, a "buy it again" one, and merge their candidates. The second stage, ranking, scores those few hundred candidates with a much richer model that can afford to be slow per item because it sees few items. Ranking models take hundreds of features, including interactions between user and item attributes, and predict several outcomes at once: probability of click, of add-to-cart, of purchase, sometimes of return. These are combined into a single score with weights that reflect business priorities. A third, lighter stage often applies rules and re-ranking for diversity, removes out-of-stock items, enforces contractual placements for sponsored products, and avoids showing the same thing twice. This architecture matters for understanding behavior. When a recommender shows something odd, the cause may be in retrieval (the right item was never a candidate), in ranking (it was a candidate but scored low), or in the business rules on top. And because each stage is trained on data produced by the whole pipeline, each inherits the exposure loop: retrieval learns from what ranking chose to show, and ranking learns only about the items retrieval let through. Sequences, sessions and transformers A shopper's history is not a bag of items; it is a sequence. Someone who has just bought a tent is a candidate for sleeping bags, not another tent. Someone browsing three different blue dresses in a row is probably still choosing rather than ready for accessories. Treating the history as a sequence lets a model capture both short-term intent and the order in which needs arise. Sequential recommendation took off in 2016 with GRU4Rec, from Balázs Hidasi and colleagues, which used a recurrent neural network to model the clicks within a single browsing session. That was well suited to e-commerce, where many visitors are anonymous and a session is all the model has. The transformer architecture that later reshaped language processing arrived in recommendation through SASRec, by Wang-Cheng Kang and Julian McAuley in 2018, and BERT4Rec, by Fei Sun and colleagues at Alibaba in 2019. These models treat a user's interaction history the way a language model treats a sentence, learning which past interactions matter for predicting the next one. By 2024 the analogy with language models had been pushed further. Researchers at Meta described generative recommenders trained on user action sequences at a scale comparable to large language models, reporting that recommendation quality continued to improve as models and data grew, a pattern familiar from language modelling. Other work replaced item identifiers with semantic codes derived from item content, so that a model can generate the code of the next item a user will want, in a style called generative retrieval. And large language models themselves are now used in retail recommendation, mostly to understand item text and reviews, to generate features, and to power conversational shopping assistants that can answer "what should I wear to an outdoor wedding in October" with products from the catalogue. It is worth being precise about what these advances deliver. Sequence models are clearly better at capturing short-term intent, and scale helps where interaction data is plentiful. But the gains reported in papers are usually measured offline, on held-out interaction logs, and as the next chapter discusses, offline gains have an unreliable relationship to business outcomes. A model that is better at predicting the next click in a log generated by a previous recommender is partly better at imitating that recommender. What makes retail different Most of the public research literature on recommendation uses film ratings, music listening or news clicks. Retail has features those domains lack, and they shape what good personalization looks like. Purchases have consequences that consumption does not. Watching a bad film costs two hours; buying the wrong sofa costs a return, a shipping charge and a damaged relationship. Return rates in online fashion are high, and a recommender that maximizes purchases without accounting for returns can increase revenue while destroying margin. Mature systems predict return probability alongside purchase probability and use size and fit models to steer shoppers toward items that will fit. Items are complements and substitutes, and these must be treated differently. After a shopper buys a television, recommending other televisions is usually a mistake (they are substitutes, and the need is met), while recommending a wall bracket or a soundbar is useful (they are complements). Collaborative filtering on purchase data confuses these: televisions are co-viewed with other televisions, because shoppers compare them, and co-purchased with brackets. Separating co-view from co-purchase signals, and learning explicit complementarity, is a retail-specific refinement. Repeat purchase is central in grocery and household goods, and absent in film. A large share of what a grocery shopper buys each week is what they bought last week. "Buy it again" models, which predict when a customer is due to repurchase a staple, are among the highest-value personalization features in grocery, and they need a notion of consumption rate that most recommendation models lack. The catalogue changes constantly. New products arrive every day, seasonal items appear and disappear, and fashion lines turn over in weeks. A new product has no interaction history, so pure collaborative filtering cannot recommend it, the cold-start problem. Retail recommenders therefore lean heavily on content: text descriptions, attributes, and increasingly image embeddings of the kind discussed in Chapter 6, which let a new dress be placed near similar dresses on the day it is listed. Finally, inventory and margin matter in a way that has no counterpart in streaming. A film can be watched by any number of people at once; the last pair of size nine boots can be sold once. Showing an item to many shoppers when only two units remain can create disappointment, and showing items with low margin can reduce profit even as it increases sales. Retail ranking objectives therefore blend shopper relevance with business value, which raises its own questions about whose interests the recommender serves. Model families compared The main families of personalization model differ in what data they need, what they capture, and where they fail, as Table 1 summarizes. In practice a large retailer runs several of these at once, combining their candidates and letting a ranking model arbitrate. Table 1. Main families of retail recommendation model. Approach Learns from Strength Weakness Item-to-item CF Co-purchase, co-view Fast, stable, explainable No cold start; popularity bias Matrix factorization User-item interactions Strong baseline at scale Ignores order and context Two-tower retrieval Interactions plus features Handles huge catalogues Coarse; needs a ranker after it Sequential transformer Ordered histories Captures short-term intent Data-hungry; offline gains overstate Content and LLM-based Text, images, reviews Works for new items Weak on taste without behavior Features, signals and the shape of a ranking model It helps to see what a ranking model in a real retail system actually consumes, because the list explains much of its behavior. A typical model scoring a candidate product for a shopper draws on several families of signal. There are features of the shopper: how long they have been a customer, how often they buy, their typical price range, the categories they favor, whether they tend to buy on discount, which brands they have bought, and their recent session activity, such as the last few products viewed and searches made. There are features of the product: category, brand, price, discount, rating, review count, return rate, stock level, delivery speed to the shopper's location, and how new it is. There are features of the context: device, time of day, day of week, the page the recommendation will appear on, and whether a sale is running. And there are interaction features, the most predictive of all, which combine shopper and product: has this shopper viewed this product before, how similar is it to items they bought, how does its price compare with what they usually spend, has someone with a similar history bought it. Two things follow. First, many of these features encode the retailer's own past decisions. Rating and review count are higher for items that were shown more and therefore bought more. "Viewed before" is high for items the recommender previously chose to show. A ranking model that leans heavily on such features will reinforce whatever the system did before, which is the exposure loop working through the feature set rather than the labels. Some teams address this by including the position at which an item was shown during training, so the model can learn how much of an item's clicks were due to position, and then setting position to a neutral value when scoring. Second, the predictions the model makes are only as good as the outcomes it is trained to predict. A model trained on clicks learns what attracts clicks, which in retail often means low prices, striking images and discounts. A model trained on purchases learns what converts. A model trained on purchases that were not returned learns something closer to satisfaction. Many systems now train on several outcomes at once and combine them, but the weights in that combination are a business choice, and changing them changes what shoppers see far more than changing the model architecture does. Search as personalization One more point deserves emphasis because it is often overlooked. For most large retailers, the single most influential personalization system is not the recommendation carousel but product search. A shopper who types "running shoes" into a site with tens of thousands of matching products will look at the first few dozen. The ranking of those results determines most of what they see, and therefore most of what they buy. Modern product search combines lexical matching, which finds products whose text contains the query terms, with semantic matching, which uses learned embeddings to find products related in meaning even when the words differ, and a learning-to-rank model that orders results using many features, including the shopper's own history. A search for "jacket" from a shopper who has previously bought children's clothing might reasonably rank children's jackets higher. The same two-stage structure applies, retrieval followed by ranking, and the same exposure loop: products ranked highly get clicked, clicks become training data, and the ranker learns to rank them highly. Search also highlights the tension between relevance and commercial priorities. Sponsored listings, private-label products and items with higher margins can be ranked above what shoppers would choose on relevance alone. Where to draw that line is a business decision, but it is also increasingly a regulatory one. The European Union's Digital Markets Act prohibits designated gatekeepers from favoring their own products in ranking, and the American Federal Trade Commission's 2023 antitrust complaint against Amazon alleged, among other things, that the company degraded search results with advertising. A personalization system's objective function is, in the end, a statement of whose interests it serves. The models described in this chapter are impressive pieces of engineering. Whether any particular one is improving the business is a separate question, and one that the models themselves cannot answer. Chapter 3: Measuring What Personalization Changes A recommendation system that shows a shopper exactly the item they were going to buy anyway looks perfect on almost every metric a data science team can compute. The shopper clicks, the shopper buys, the model's prediction is confirmed, and the carousel is credited with a sale. Yet the system has accomplished nothing. The sale would have happened without it. This is the central measurement problem of personalization, and it is the exposure loop from Chapter 1 in its most practical form. The question a retailer needs answered is not "how well does the model predict what shoppers do" but "how much does showing these recommendations change what shoppers do, compared with showing something else or nothing at all." The first is a prediction question. The second is a causal question. This chapter is about the tools for answering the second. The gap between offline and online Recommendation models are developed offline. A team takes logs of past interactions, holds out the most recent portion, trains a candidate model on the rest, and measures how often the model ranks the held-out interactions near the top. The resulting numbers, with names like recall at ten, normalized discounted cumulative gain and hit rate, are what appear in research papers and internal model reviews. They are cheap to compute and allow hundreds of variants to be compared in a day. The trouble is that these logs were produced by the recommender already in production. The held-out interactions are disproportionately with items that the old system chose to show. A new model that agrees with the old one will score well, because it predicts the interactions the old one caused. A new model that would recommend genuinely better items the old one never showed will score poorly, because there are no logged interactions with those items to reward it. Offline evaluation on logged data is biased toward the status quo. Practitioners have long known this, and the relationship between offline metric gains and online business gains is notoriously weak. Teams at several large companies have reported in public talks and papers that offline improvements frequently fail to translate into online improvements, and occasionally reverse. Academic work has added a second concern: when researchers re-examined a range of published neural recommendation models against carefully tuned simple baselines, notably in a 2019 study by Maurizio Ferrari Dacrema, Paolo Cremonesi and Dietmar Jannach, many of the claimed improvements disappeared. Neither finding means offline evaluation is useless. It is an indispensable filter for discarding bad ideas quickly. But it cannot certify that a good-looking idea will create value. Experiments as the ground truth The reliable answer is the randomized controlled experiment, known in industry as the A/B test. Shoppers are randomly assigned to see either the current system or the new one, and the groups' outcomes are compared. Because assignment is random, any systematic difference in outcomes is caused by the system and not by differences between the groups. The large online retailers and platforms run thousands of such experiments a year, and the methodology has been set out in detail by Ron Kohavi, Diane Tang and Ya Xu, whose 2020 book on trustworthy online controlled experiments draws on experience at Microsoft, Google and LinkedIn. Several features of retail make experimentation harder than it looks. The metrics that matter are noisy. Revenue per visitor is dominated by a small number of large purchases, so its variance is enormous, and detecting a one percent improvement can require millions of visitors. Teams use variance-reduction techniques, most commonly a method known as CUPED that adjusts each shopper's outcome using their behavior before the experiment, and they track more sensitive leading indicators such as add-to-cart rate, while recognizing that a change can move a leading indicator without moving the outcome that matters. The effects that matter are often slow. A recommendation change that encourages more impulse purchases may raise revenue this week and raise returns and reduce repeat visits over the following months. Experiments rarely run long enough to see that. One response is the surrogate index, developed by Susan Athey, Raj Chetty, Guido Imbens and Hyunseung Kang, which combines several short-term outcomes into a predictor of a long-term one, validated on historical data where both were observed. Another is to keep a small long-running holdout group that does not receive a feature, so its cumulative effect can be measured. The units interact. In a marketplace, recommending one seller's product to more shoppers takes sales from other sellers. If a test increases sales for the treated group partly by drawing limited inventory away from the control group, the measured lift overstates what would happen if the change were rolled out to everyone. Statisticians call this interference. It is especially serious for pricing experiments and for anything touching scarce inventory, and it is addressed by randomizing at higher levels, by geography or time period, or by modelling the interaction directly. A complementary technique, used heavily for search and ranking, is interleaving. Instead of showing one group of shoppers results from ranker A and another group results from ranker B, an interleaving test merges both rankings into a single list for each shopper, alternating items from each in a balanced way, and records which ranker's items the shopper clicks or buys. Because each shopper effectively compares the two rankers side by side, the differences between shoppers that make A/B tests noisy largely cancel out, and interleaving can detect a preference between two rankers with far less traffic than a conventional test. Its limitation is that it measures which ranker shoppers prefer, not what each would do to revenue if deployed alone, so teams typically use it to screen candidates quickly and then confirm the winner with a standard experiment. Incrementality: the question behind the question Even a well-run A/B test only compares one system with another. It does not say how much of the recommender's apparent contribution is real. To answer that, one needs to know what shoppers would have done without recommendations, which is a matter of incrementality. A well-known study by Amit Sharma, Jake Hofman and Duncan Watts, presented in 2015, attacked this question using observational data from Amazon. They looked for natural experiments: cases in which the product a shopper was viewing experienced a sudden, external spike in traffic, for example from a mention in the news, while the products recommended alongside it did not. By comparing how traffic to the recommended products changed, they could estimate how many clicks on recommendations were caused by the recommendations. Their estimate was that at least three quarters of the click-throughs on recommendations would likely have happened anyway, through search or direct navigation. The recommendations were not worthless, but their causal contribution was a fraction of what naive attribution credited to them. The implication for system design is direct. If a recommender is trained to maximize clicks on its own recommendations, it will learn to recommend things shoppers were already going to buy, because those are the easiest clicks to predict. It will look successful by its own metrics while adding little. A recommender trained to maximize the incremental effect of recommendations, sometimes called an uplift or causal recommender, has to learn something different: which items a shopper would not find on their own but would value if shown. That is a harder problem, and it requires training data in which exposure was at least partly randomized. Uplift modelling, which grew up in direct marketing, makes the same move for targeting. The classical approach to sending a discount email is to send it to the customers most likely to buy. But many of those customers would buy without the discount, so the retailer is giving away margin. The customers worth targeting are those whose probability of buying increases most because of the email. Uplift models estimate that difference, the conditional treatment effect, using data from experiments in which some customers were randomly held out. Marketers commonly sort customers into four groups: those who will buy regardless, those who will buy only if contacted, those who will not buy regardless, and those who are put off by being contacted. Only the second group is worth paying to reach. A small illustration, with invented round numbers, shows why the distinction matters commercially. A retailer considers sending a 15 percent discount code to 100,000 customers. Its purchase-propensity model selects the 110,000 most likely to buy in the next week, and a random 10,000 of them are held out and receive nothing. In the week that follows, 20 percent of the emailed customers buy, against 18 percent of the held-out group. The campaign report credits the email with 20,000 purchases. In fact it caused about 2,000 of them, the two-point difference, while giving a discount on all 20,000. If the average basket is 80 dollars, the retailer gave away about 240,000 dollars in discounts to generate 160,000 dollars of incremental revenue, before counting the margin on that revenue. An uplift model, trained on the same kind of randomized holdout, might find a different 100,000 customers, people whose baseline purchase probability is lower but who respond much more strongly to a discount, among whom the email lifts purchasing from 6 to 12 percent. Fewer total purchases would follow, and far more of them would be caused by the campaign. The propensity model optimizes the number on the report. The uplift model optimizes the number in the bank. The general lesson is that a model trained to predict an outcome is the wrong tool for deciding whom to treat, whenever the outcome would often occur without the treatment. Retail is full of such decisions: whom to email, whom to show a discount banner, whom to offer free shipping, which products to put in a recommendation slot. In each case, the customers or items most likely to produce the outcome are frequently those for whom the intervention matters least. Bandits: learning while acting Experiments answer a question and then stop. Many personalization problems call for continuous learning: new products arrive daily, tastes shift, and the system needs to keep trying things it is unsure about while mostly showing things it believes are good. That balance between exploration and exploitation is the subject of bandit algorithms, named after the gambler choosing among slot machines. The contextual bandit, in which the best action depends on observable features of the situation, became practical for web personalization with work at Yahoo around 2010, when Lihong Li, Wei Chu, John Langford and Robert Schapire used it to choose news articles for the Yahoo front page. In retail, contextual bandits choose which banner to show, which products to place in a carousel slot, which email subject line to use, or which of several ranking models to apply. Common algorithms include Thompson sampling, which picks each action with probability equal to the model's belief that it is the best, and upper-confidence-bound methods, which favor actions whose value is uncertain enough that they might be the best. Bandits help with the exposure loop because they explore by design. A bandit system deliberately shows items it is uncertain about, which generates information about them, which prevents the system from locking onto early winners. They also record, for every decision, the probability with which each action was chosen. That record turns out to be extremely valuable. Off-policy evaluation If every past decision was made with a known probability, then the log can be reweighted to estimate how a different policy would have performed, without deploying it. This is off-policy evaluation, and its simplest form, inverse propensity scoring, is easy to state. For each logged decision, check whether the new policy would have taken the same action. If it would, count the outcome, weighted by one over the probability that the logging policy took that action. If not, count zero. Averaging gives an unbiased estimate of the new policy's value. def ips_value(log, new_policy): total = 0.0 for ctx, action, reward, prob in log: if new_policy(ctx) == action: total += reward / prob return total / len(log) The intuition is that actions the old system rarely took are informative when they did happen, so they get more weight. The method has limits. It requires that the old system gave every action the new policy might choose some chance of being taken, and when those chances are small the weights become huge and the estimate becomes noisy. Refinements, including clipped weights, self-normalized estimators and doubly robust estimators that combine reweighting with a predictive model of outcomes, trade a little bias for much lower variance. The 2011 work by Li and colleagues demonstrated that replaying a log of randomized decisions could produce estimates close to those from live tests. The practical lesson is organizational as much as statistical. A retailer that logs only what it showed can evaluate new ideas only by running experiments. A retailer that logs what it showed, what else it considered, and the probability with which it chose each option can evaluate many new ideas offline with reasonable confidence, and reserve live experiments for the most promising. Building that logging in from the start costs little. Retrofitting it is expensive. Popularity, diversity and the shape of the catalogue The exposure loop has a long-term consequence that is invisible in any single experiment: it shapes which products the retailer sells. A recommender trained on its own outputs concentrates attention on items that were popular early. Over time, sales concentrate on fewer products, new products struggle to gain traction, and the catalogue a retailer effectively offers shrinks even as the catalogue it stocks grows. Whether this is bad depends on the business. For a retailer whose advantage lies in breadth, it erodes the very thing that differentiates it. For third-party sellers on a marketplace, it determines who can make a living. For shoppers, it can mean a narrower experience than they would choose. Researchers have documented popularity bias in many recommendation algorithms and proposed corrections: penalizing popular items in ranking, explicitly optimizing for diversity or novelty, reserving a share of exposure for new products, and correcting the training data for exposure probability so the model learns preference rather than visibility. Each of these corrections has a short-term cost, since the more popular item is usually the safer recommendation. Their benefits appear over months, in catalogue health, seller retention and customer satisfaction, which is exactly the horizon on which experiments are weakest. That is why decisions about diversity and exploration are best made deliberately, as policy, rather than left to emerge from whatever a click-optimizing model happens to learn. A measurement culture The tools in this chapter, controlled experiments, incrementality estimates, uplift models, bandits and off-policy evaluation, are all ways of breaking the exposure loop by introducing randomness the retailer controls and recording it carefully. They are not exotic. They are well documented, widely taught and used routinely at the largest platforms. Where they are missing, it is usually because the organization measures its personalization teams on metrics the model can move by itself, such as click-through on recommendations, rather than on metrics only a causal analysis can credit. The test of a mature personalization program is whether it can answer, for any of its models, the question this chapter began with: how much does this change what shoppers do? A team that can answer that question can improve its models. A team that cannot is improving its predictions, which is not the same thing. Chapter 4: Estimating Price Elasticity Every pricing decision rests on an answer to one question: if the price changes, how much does demand change? Economists measure the answer as price elasticity, the percentage change in quantity demanded for a one percent change in price. An elasticity of minus two means that a one percent price increase reduces unit sales by about two percent. Items with elasticities far below minus one are price-sensitive; raising their prices loses enough volume to reduce revenue. Items with elasticities between zero and minus one are relatively insensitive; raising their prices increases revenue, at least until shoppers start looking elsewhere. Elasticity sounds like a number one could read off historical data. In retail it almost never can be, and the reasons are the clearest example in the whole field of the closed loop described in Chapter 1. This chapter explains why, and describes the methods, old and new, for getting estimates good enough to act on. Why the obvious regression fails Suppose an analyst at a supermarket chain takes two years of weekly sales and prices for a brand of orange juice, and regresses the logarithm of units sold on the logarithm of price. The coefficient on price is an estimate of elasticity. What could go wrong? Almost everything. Prices were not set at random. They were set by people and systems responding to conditions, and many of those conditions also affect demand. Consider a few mechanisms. The retailer runs promotions when demand is seasonally high, because that is when shoppers are in the store and the promotion draws the most traffic. Price is low exactly when demand would be high anyway. The regression attributes the seasonal demand to the low price and overstates elasticity. The retailer raises prices when a competitor is out of stock or when a supply disruption tightens the market. Demand at that retailer is high for reasons unrelated to its own price. Price is high when demand is high, and the regression understates elasticity, possibly to the point of finding that higher prices increase sales. Markdowns on fashion and seasonal goods are applied when items are selling slowly, which is when demand is weak. Price falls because demand fell. The regression sees low prices with low sales and, again, understates elasticity. Each mechanism makes price a function of unobserved demand conditions. Economists call this endogeneity, and it means the coefficient from the naive regression is not an estimate of the causal effect of price on demand. It is a mixture of that effect and the retailer's own pricing behavior. Adding more data does not help. Adding a more flexible model does not help either: a deep neural network trained to predict sales from price and other features will learn the same contaminated relationship more precisely. This deserves emphasis because it runs against intuition in machine learning, where more data and more capacity usually improve results. For the purpose of prediction, the naive model may be excellent. If prices next month will be set the way they were last year, the model will predict sales well. But the moment it is used to choose prices, which means considering prices set differently from how they were set historically, its predictions break down. The model has learned how the retailer's pricing and demand moved together, not what would happen if the retailer changed its pricing. A simple thought experiment makes the size of the problem concrete. Imagine an item whose true elasticity is minus two: every one percent price cut raises sales by two percent. The retailer's merchandising calendar puts the item on a ten percent promotion during the four weeks of the year when seasonal demand is already thirty percent above normal. In those weeks, sales are up about twenty percent from the promotion and about thirty percent from the season, roughly fifty percent in all. A regression that does not know about the season sees a ten percent price cut associated with a fifty percent increase in sales, and estimates an elasticity of about minus five. A pricing system using that estimate will conclude the item is extremely price sensitive and keep its price low all year, sacrificing margin in every week when the season is not doing the work. If the analyst adds a seasonal control, the estimate improves, but only if the control captures the season correctly, and only if the season is the only thing that moved with price. In real data, there are dozens of such co-movements, many of them not recorded at all. Elasticity online E-commerce adds its own complications. Online, a product's price affects not only whether a shopper buys it but whether the shopper sees it. Many marketplaces sort search results partly by price or use price in deciding which seller wins the main purchase button on a product page. On Amazon, that button, the Buy Box, goes to one of the competing sellers of an identical item, and price is a major factor. A small price change can therefore produce a large change in sales, not because shoppers are highly sensitive to price but because the change moved the product up the search results or won it the Buy Box. The estimated elasticity mixes shopper response with platform response, and the two behave differently: the platform effect is often a sharp threshold, a cent below a rival wins and a cent above loses, while shopper response is usually smooth. Competitor prices are also more visible online, and both shoppers and retailers track them. Price-comparison sites, browser extensions and, increasingly, AI shopping assistants put rival prices one click away. Demand for an item at one retailer then depends heavily on its price relative to others, so an elasticity estimated without competitor prices in the model confuses the effect of the retailer's own price changes with the effect of competitors' changes. Scraped competitor price data are a standard input to online pricing models for this reason, and their quality, which is frequently patchy, limits the quality of the elasticity estimates built on them. Finding variation that behaves like an experiment The cure for endogeneity is to find variation in price that is not driven by demand conditions, and to estimate elasticity from that variation only. There are several ways to get it. The cleanest is to create it. Randomized price experiments, in which some stores, some time periods or some shoppers see different prices assigned at random, produce unbiased elasticity estimates directly. Online retailers can randomize prices more easily than physical ones, though randomizing by individual shopper raises fairness and legal questions taken up in the next chapter, so experiments are more often randomized across time periods, regions or product groups. Even small randomized variations, a few percent above and below a reference price, are enough to estimate elasticity if the experiment runs long enough. Retailers with pricing science teams often build a modest amount of random price variation into routine operations precisely so that their elasticity models have clean data to learn from. The second approach uses natural sources of variation, known as instruments. An instrument is something that moves price but affects demand only through price. Changes in wholesale cost are a classic example: when the cost of coffee beans rises and retailers pass some of it on, the price change comes from the supply side, not from a change in what shoppers want. Changes in exchange rates, taxes and commodity prices can serve similarly. A family of instruments associated with the economist Jerry Hausman uses the price of the same product in other cities, on the reasoning that common cost shocks move prices together across markets while local demand shocks do not. Instrumental variable methods then isolate the part of price variation explained by the instrument and estimate elasticity from it alone. They work well when good instruments exist and poorly when instruments are weak or questionable, and judging whether an instrument is valid requires domain knowledge that no algorithm supplies. The third approach exploits institutional quirks. Many retail chains change prices on a fixed calendar, or apply company-wide price changes that hit all stores at once regardless of local conditions. A study by Stefano DellaVigna and Matthew Gentzkow, published in 2019, found that large American retail chains charge nearly uniform prices across their stores, despite wide differences in the income and competition those stores face, and that the stores' estimated elasticities differ substantially. That finding is interesting in its own right, since it implies chains leave money on the table by not varying prices locally, but it is also methodologically useful: when prices move for reasons set at headquarters, the variation in any single store is closer to exogenous. Structure: pooling, hierarchies and cross-effects Even with good variation, a retailer faces a data problem. A supermarket may carry thirty thousand products across hundreds of stores. Most products have had only a handful of distinct prices, and many sell only a few units a week per store. Estimating a separate, reliable elasticity for each product in each store is impossible. The standard answer is to pool information across related items through hierarchical models. The idea is that the elasticity of a particular brand of orange juice in a particular store is probably similar to that of other orange juices, and juices generally, and the same brand in similar stores. A hierarchical Bayesian model encodes this: it estimates a category-level elasticity, lets each brand deviate from it, lets each store deviate from the brand, and shrinks each estimate toward its group average in proportion to how little data supports it. Products with lots of informative price variation get their own estimates; products with little get estimates close to their category. This is not a modern machine learning invention, as the econometrics and marketing science literature developed it decades ago, but it remains the backbone of most serious retail pricing systems. Retail demand also has cross-effects that single-product models miss. Cutting the price of one brand of orange juice draws sales from other brands; cutting the price of tortilla chips may lift sales of salsa. Cross-price elasticities determine whether a promotion grows the category or merely shifts sales around within it, and whether a price cut on one item pays for itself through its complements. Estimating a full matrix of cross-elasticities for thousands of items is infeasible, so systems impose structure: they assume substitution happens mainly within narrowly defined categories, or they use discrete-choice models in which each shopper picks one item from a set, so that cross-effects follow from each item's appeal and price. Discrete-choice demand models, refined in economics by Steven Berry, James Levinsohn and Ariel Pakes in a celebrated 1995 paper, are the standard tool in industrial organization for exactly this reason, and retail pricing systems use simplified versions of them. Promotions add further complications. A temporary price cut is not the same as a permanent one. Shoppers who see a discount on detergent may stock up, so sales during the promotion include purchases pulled forward from future weeks; the elasticity during the promotion looks large, but part of the effect is merely a shift in timing. Shoppers also form reference prices, expectations of what an item normally costs, and a price that is frequently discounted can lose its ability to sell at full price. Models that treat promotional and regular price changes identically will make poor decisions about both. Where machine learning helps None of the above is new. What machine learning adds, when used carefully, is the ability to handle many more control variables and to estimate how elasticity varies across products, stores, customers and time. The most important recent development is double machine learning, formalized in a 2018 paper by Victor Chernozhukov and colleagues. The idea addresses a practical difficulty. To estimate the effect of price on demand from observational data, one must control for everything that drives both price and demand: seasonality, holidays, weather, competitor activity, inventory levels and so on. With hundreds of such variables interacting in complicated ways, a traditional regression either omits some or becomes unstable. Double machine learning uses flexible models for the control step and a simple regression for the causal step. In outline: use one machine learning model to predict sales from all the controls, and another to predict price from all the controls. Take the residuals from each, the part of sales and the part of price that the controls cannot explain. Regress the sales residual on the price residual. The coefficient is the elasticity estimate. # stage 1: flexible models of nuisance relationships sales_hat = fit_predict(model_y, controls, log_sales) price_hat = fit_predict(model_p, controls, log_price) # stage 2: simple regression on what controls leave unexplained y_res = log_sales - sales_hat p_res = log_price - price_hat elasticity = (p_res @ y_res) / (p_res @ p_res) The fit_predict step uses cross-fitting, where each observation's prediction comes from a model trained on other data, which prevents overfitting from biasing the estimate. The method gives valid confidence intervals under conditions that are well understood. It is important to be clear about what this does and does not solve. Double machine learning corrects for confounders that are measured. If the retailer cut prices because a merchandiser noticed slow sales, and that observation is not in the data, the residual price variation is still contaminated, and the elasticity estimate is still biased. Flexible models make it possible to control for more things. They do not make it possible to control for things not recorded. That is why the methods in the previous section, experiments and instruments, remain necessary, and why the best practice combines them: use machine learning to control for observed variables, and use randomized or instrumented variation for the rest. The second contribution is the estimation of heterogeneous effects. Retailers do not want one elasticity for orange juice; they want to know whether it differs between stores in affluent suburbs and stores near universities, between weekdays and weekends, between loyal customers and occasional ones. Methods such as causal forests, developed by Stefan Wager and Susan Athey, extend random forests to estimate how a causal effect varies with observed characteristics, with valid statistical inference. Combined with experimental or instrumented variation, they let a retailer build an elasticity map rather than a single number. Choosing an approach The main strategies differ in what they require and what they guard against, as Table 2 sets out. Most mature pricing systems use more than one: hierarchical models for coverage, small randomized experiments for calibration, and double machine learning or instruments where experiments are impractical. Table 2. Approaches to estimating retail price elasticity. Approach Source of variation Handles Main risk Naive regression All historical prices Nothing Confounded by pricing policy Randomized tests Assigned at random All confounding Cost, scale, fairness limits Instruments Cost or other-market shocks Unmeasured demand shocks Weak or invalid instruments Hierarchical Bayes Pooled across items Sparse data Shrinks real differences away Double ML Residual price movement Many measured controls Unmeasured confounders remain A note on validation How does a pricing team know its elasticity estimates are right? Prediction accuracy is not the test, for the reasons given above. A model can predict sales well and get elasticity wrong. The test is whether the model predicts the effect of price changes it did not see. The most convincing validation is prospective: use the model to predict what will happen under a planned set of price changes, run those changes, and compare. Teams that do this routinely often discover that their elasticity estimates are too large, because promotional data inflates them, or too uniform, because pooling flattens real differences. Those discoveries are valuable, and they come only from treating the model's estimates as hypotheses to be tested against deliberate interventions, not as facts read from history. With credible elasticity estimates in hand, a retailer can begin to optimize prices. That turns out to raise a separate set of difficulties, some technical, some commercial, and some that have nothing to do with statistics at all. Chapter 5: From Elasticity to Prices An elasticity estimate is an input. What a retailer actually does is set prices, thousands or millions of them, and change them as conditions change. This chapter covers how pricing systems turn estimates of demand into decisions, how they keep learning once they are live, and where the limits lie. Some of those limits are technical. Others concern competitors, customers and the law, and they have become more pressing as algorithmic pricing has spread from airlines and hotels into everyday shopping. The optimization layer In its simplest form, profit-maximizing pricing has a tidy textbook answer. If an item has constant elasticity and a known unit cost, the profit-maximizing price is a fixed markup over cost that depends only on elasticity: the more elastic the demand, the smaller the markup. An item with an elasticity of minus three should be priced at one and a half times its cost; an item with an elasticity of minus two at twice its cost. Real systems almost never use this formula directly, but it gives the right intuition. Prices should be higher where shoppers are less sensitive and lower where they are more sensitive. Practical pricing systems solve a much more constrained problem. They choose prices for whole categories at once, because of the cross-effects discussed in Chapter 4. They respect price ladders, rules that a larger pack must cost less per unit than a smaller one and that a premium brand must cost more than the store brand. They enforce relationships with competitors, such as staying within a set percentage of a rival's price on certain items. They round to price endings shoppers expect. They limit how often and how far any price can move, both because each change costs money in physical stores (labels must be changed, though electronic shelf labels have reduced this) and because shoppers notice volatility and dislike it. Retailers also distinguish between items by their role. Known-value items, the milk, bananas, bread and branded staples whose prices shoppers actually remember, drive perceptions of whether a store is cheap. Pricing them a few cents above competitors can cost more in lost traffic than it gains in margin, so they are often priced aggressively regardless of their own elasticity. The margin is made instead on items whose prices shoppers do not track closely. A pricing system that optimizes each item's profit in isolation will raise the price of milk, see a short-term gain, and erode the store's price image in ways no item-level elasticity captures. The role of an item in the shopper's overall judgement of the store is a strategic choice that sits above the optimization, not an output of it. Markdowns: the classic success The application where algorithmic pricing has the longest record of documented value is markdown optimization for seasonal and fashion goods. A retailer buys a fixed quantity of winter coats before the season, cannot reorder, and must sell them before spring. The question is when to cut prices and by how much. Cut too early and margin is given away on coats that would have sold at full price; cut too late and the retailer is left with stock it must clear at a deep loss. This is a well-posed problem that machine learning and operations research handle well: forecast demand at each possible price for the remaining weeks, account for the inventory on hand, and choose a sequence of prices that maximizes expected revenue. Retailers adopted markdown optimization software widely from the early 2000s. A frequently cited example comes from a collaboration between the online flash-sale retailer Rue La La and researchers Kris Johnson Ferreira, Bin Hong Alex Lee and David Simchi-Levi, published in 2016. They built demand models for first-time products, which had no sales history, using features of similar past items, and an optimization procedure to set prices. In a controlled field test, the recommended prices raised revenue for the affected products by roughly ten percent without a meaningful reduction in units sold, a rare case of a published, experimentally validated pricing gain. Markdowns work well for structural reasons worth noticing. The decision is made once per item per week or so, not continuously. The objective is clear. The inventory constraint makes the problem bounded. And shoppers expect seasonal goods to be discounted as the season ends, so the price movements are accepted as normal. Those conditions are not present everywhere, and where they are absent, algorithmic pricing becomes harder. Learning while pricing A pricing system that only exploits its current elasticity estimates has the exposure-loop problem in another form. If it always charges what it believes is the best price, it never observes demand at other prices, and if its estimate is wrong it may never discover the error. Prices that vary only in response to the model's own beliefs also make future elasticity estimates harder, since the variation they create is confounded by whatever drove the beliefs. Dynamic pricing with learning treats this explicitly as an exploration problem. Bandit algorithms of the kind described in Chapter 3 choose prices so as to balance earning revenue now with learning about demand for later. Thompson sampling is a natural fit: the system maintains a probability distribution over the demand curve, samples one plausible curve from it, prices optimally for that curve, observes the result, and updates. Prices vary in proportion to the system's uncertainty, which means they vary a lot early on and settle as evidence accumulates. Ferreira, Simchi-Levi and He Wang showed in 2018 how to extend this approach to the realistic case of many products sharing limited inventory. These methods are powerful in e-commerce, where prices can change at low cost and outcomes are observed quickly. They are also where a retailer must think hardest about what shoppers will tolerate. A shopper who sees a price rise and fall from one day to the next, for reasons that are invisible to them, may simply conclude the retailer is untrustworthy. Many retailers therefore constrain exploration to modest price ranges, to new or slow-moving items, or to regional rather than individual variation. Dynamic pricing comes to the store For most of retail history, prices in physical stores were sticky by necessity. Changing a price meant printing and placing a new paper label, a task that took staff time and invited errors where the shelf price and the till price disagreed. Electronic shelf labels, small digital displays that can be updated from a central system, remove that friction. They have been common in European grocery for years and are spreading in North America. With them, a store can in principle change prices as often as a website. The technical case for electronic labels is modest and real: fewer labor hours spent on label changes, fewer pricing errors, faster markdowns on perishable items approaching their sell-by date. Some grocers use them to discount fresh food progressively through the day, which reduces waste. But the prospect of in-store prices that move with demand has drawn suspicion. In February 2024 the American restaurant chain Wendy's described plans to test dynamic pricing on digital menu boards; after widespread reports that it intended to raise prices at busy times, the company clarified that it had meant discounts in quiet periods, not surge pricing. Later that year, two United States senators wrote to the grocery chain Kroger asking about its use of electronic shelf labels and whether they could enable surge pricing or individualized pricing in stores. Neither episode involved evidence of in-store surge pricing, but both showed how sensitive shoppers are to the idea, especially for essentials. The lesson for retailers is that technical capability and acceptable practice have diverged. A system that can change a price every fifteen minutes is not thereby licensed to do so. In grocery, where the same shoppers buy the same items weekly and remember prices, frequent upward movement destroys trust faster than it builds margin. The most durable uses of dynamic pricing in stores so far have been those that shoppers recognize as being in their interest, such as markdowns on short-dated food, rather than those that look like charging more when people most need something. Pricing against algorithms Online, a retailer's prices are not set in isolation. Competitors scrape each other's prices continuously, and many sellers, particularly on large marketplaces, use automated repricing tools that adjust prices in response to rivals' prices according to simple rules: match the lowest price, undercut it by a cent, stay a fixed margin above it. A 2016 study by Le Chen, Alan Mislove and Christo Wilson identified more than five hundred sellers on Amazon's marketplace whose price movements showed clear signs of algorithmic repricing, and found that those sellers tended to win the Buy Box more often and to have more feedback, which suggested higher sales volumes. When algorithms respond to each other, unexpected things happen. The most famous illustration is a 2011 episode in which two sellers on Amazon offered a used copy of a developmental biology book, The Making of a Fly. One seller's algorithm set its price at a little below the other's; the other's set its price at a little above the first's, presumably to profit by fulfilling orders through the cheaper seller. Each day both prices rose, until the book was listed at more than twenty-three million dollars. The biologist Michael Eisen documented the episode on his blog, and it has served ever since as a reminder that a pricing rule sensible on its own can be absurd in interaction. The more serious concern is not absurdity but collusion. Price-fixing agreements among competitors are illegal, but what if pricing algorithms learn to keep prices high without any agreement? A 2020 study by Emilio Calvano, Giacomo Calzolari, Vincenzo Denicolò and Sergio Pastorello showed that simple reinforcement learning agents, specifically Q-learning algorithms, set to price against each other in a simulated market, learned to sustain prices above the competitive level and to punish deviations with temporary price wars, without being programmed to collude or communicating with each other. Empirical evidence followed. Stephanie Assad, Robert Clark, Daniel Ershov and Lei Xu, studying German retail gasoline stations, found that when both stations in a local duopoly adopted algorithmic pricing software, their margins rose markedly, while adoption by only one station had little effect on margins. That pattern is what one would expect if the algorithms were learning to avoid competing with each other. Regulators have taken notice. In 2015 the United States Department of Justice brought its first criminal prosecution for price-fixing in e-commerce against a seller of posters on Amazon Marketplace who had agreed with competitors to use pricing algorithms to coordinate prices. That case involved an explicit agreement, which made it legally straightforward. Cases in which a shared pricing vendor or shared data may coordinate competitors without explicit agreement are harder, and the Department of Justice's action against the rental software company RealPage, brought in 2024, tested how far antitrust law reaches when many competitors feed confidential data into a common algorithm. The FTC's 2023 antitrust complaint against Amazon also included allegations about an internal pricing algorithm, which the complaint called Project Nessie, alleged to have raised prices by predicting when competitors would follow. For a retailer, the practical point is that a pricing system's interaction with competitors' systems is part of its behavior, and the retailer is responsible for that behavior even if no one intended it. Personalized pricing and its discontents The most contentious frontier in algorithmic pricing is the use of data about individual shoppers to set individual prices. In economic terms this is price discrimination: charging different customers different prices for the same good according to their willingness to pay. It is neither new nor always objectionable. Student discounts, senior discounts, loyalty coupons and targeted promotions are all forms of it, and they often expand access by letting price-sensitive customers buy at prices that would be unprofitable to offer everyone. What has changed is the precision with which it can be done and the opacity of how. A model trained on browsing history, purchase history, location, device and other data can, in principle, estimate an individual's willingness to pay and set a price just below it. Shoppers have strong intuitions that this is unfair, especially when the data used are not things they chose to disclose, and these intuitions shape behavior: a retailer caught charging different prices to different people for the same item risks a backlash out of proportion to any gain. In 2012 the Wall Street Journal reported that several retailers' websites showed different prices depending on the shopper's estimated location, with the effect, in some cases, that shoppers in lower-income areas saw higher prices because they lived farther from competing stores. That reporting is still cited as the moment online price discrimination became a public issue. The regulatory response has accelerated. In July 2024 the Federal Trade Commission ordered eight companies that sell pricing and data services to explain how they help clients use consumer data to set prices, and in January 2025 it released initial findings describing how intermediaries use data such as location, demographics and browsing history to target individualized prices and offers. The Commission did not reach conclusions about legality, and its leadership changed shortly afterwards, but the inquiry gave the practice a name, surveillance pricing, that has stuck. New York enacted an Algorithmic Pricing Disclosure Act in 2025 which, after surviving a legal challenge, took effect in November of that year; it requires businesses that set a price using a shopper's personal data through an algorithm to say so, in prescribed words, next to the price. In the European Union, rules adopted under the 2019 Omnibus Directive already required online traders to inform consumers when a price has been personalized on the basis of automated decision-making. Public scrutiny has also come from investigation. In December 2025 Consumer Reports and the advocacy group Groundwork Collaborative published a study in which volunteers shopped for the same groceries from the same stores through Instacart at the same time and were frequently shown different prices. Instacart said the differences came from randomized price tests run for retailers, not from personal data. The episode illustrates a point this booklet has made several times: the price experiments that are essential for learning elasticity are, from the shopper's side, indistinguishable from personalized pricing. Both mean that two people see two prices for one item at one moment. A retailer that wants the benefits of experimentation has to decide how to run it in ways customers and regulators will accept, which may mean randomizing over time or geography rather than over individuals, limiting the size of price differences, or disclosing that testing is occurring. The limits of the optimizer The thread running through this chapter is that the optimizer is the easy part. Given a demand model and constraints, choosing prices is a well-understood mathematical problem. The hard parts surround it. The demand model must be causal, as Chapter 4 explained. The system must keep learning without destabilizing prices or confounding its own data. It must anticipate how competitors' algorithms will react, and must not drift, alone or in concert with them, into conduct the law treats as collusion. And it must respect shoppers' sense of fairness, which is now being written into law. Each of these is a version of the closed loop. The pricing system changes the market it observes: it changes what shoppers buy, what competitors charge, what regulators look for and what customers believe about the retailer. A system optimized only for the next week's margin, without accounting for those responses, will appear to succeed and then gradually fail. The retailers that have made algorithmic pricing work over many years have done so by treating these surrounding questions as part of the pricing system, not as constraints imposed from outside. Chapter 6: Visual Search Some things are easier to show than to describe. A shopper who sees a lamp in a café, a jacket on a stranger or a tile pattern in a magazine may have no idea what words would find it: "mid-century brass arc floor lamp" is a phrase that only someone who already knows the vocabulary can type. Visual search lets the shopper skip the vocabulary. Point a phone at the object, or upload a photo, and the system returns products that look like it. Visual search is a showcase for what modern machine learning does best, which is learning useful representations of messy perceptual data. It is also a useful contrast with the rest of this booklet. Matching a photograph to a product is mostly a perception problem: whether two images show similar objects is a fact about the images, not something the retailer's past decisions caused. The closed loop reappears only at the point where matching results are ranked and shown to shoppers. Understanding where the line falls helps clarify what is distinctive about the other problems. From pixels to embeddings The core idea behind visual search is the embedding, the same concept that underlies the two-tower recommenders of Chapter 2. A neural network takes an image and produces a vector of a few hundred or a few thousand numbers. The network is trained so that images of similar items produce vectors that lie close together, and images of different items produce vectors that lie far apart. Once every product image in the catalogue has been converted to a vector, visual search becomes a geometric question: convert the shopper's photo to a vector, and find the catalogue vectors nearest to it. Everything depends on what "similar" means, and that is set by how the network is trained. Early systems, in the first half of the 2010s, took convolutional neural networks trained to classify everyday objects on the ImageNet dataset and used their internal activations as embeddings. That worked surprisingly well for broad categories, telling shoes from handbags, but was weak on the fine distinctions shoppers care about: the difference between a loafer and a moccasin, or between two floral prints. The networks had been trained to tell dogs from cats, not to tell one dress from another. The fix was metric learning: training the network directly on the task of placing similar items close together. In triplet training, the network sees an anchor image, a positive image of the same or a similar product, and a negative image of a different one, and is penalized whenever the anchor is closer to the negative than to the positive by less than some margin. Contrastive training generalizes this to many negatives at once. The difficult part is obtaining the pairs. Retailers have several sources: multiple photographs of the same product in the catalogue, shoppers' own photographs in reviews, and behavioral signals such as products that shoppers frequently view in the same session or click on after a visual search. Much of the engineering effort in building a visual search system goes into assembling and cleaning this supervision. The street-to-shop gap Catalogue photographs and shoppers' photographs are very different kinds of image. A catalogue photograph of a jacket shows it flat or on a model, centered, evenly lit, against a white background. A shopper's photograph shows it worn by a person walking down a street, partly obscured by a bag, at an angle, in poor light, with a dozen other objects in the frame. A system that matches catalogue images to each other well may match street images to catalogue images poorly. Researchers named this the street-to-shop problem, and studied it with datasets pairing everyday photographs of clothing with the corresponding product images, such as the Street2Shop dataset introduced by M. Hadi Kiapour and colleagues in 2015 and the larger DeepFashion collection from Ziwei Liu and colleagues at the Chinese University of Hong Kong in 2016. Two techniques do most of the work in closing the gap. The first is detection: before computing an embedding, find the relevant object in the photograph and crop to it, so that the network sees the jacket rather than the street. Modern systems run an object detector that locates every product-like item in the image and lets the shopper choose which to search for. The second is training on mixed pairs, a street photograph and a catalogue photograph of the same item, so that the network learns to ignore pose, lighting and background. Systems at scale Several companies have published detailed accounts of their visual search systems, and they share a common structure. Pinterest described its first visual search system in a 2015 paper by Yushi Jing and colleagues, and its expanded visual discovery system in 2017. Pinterest's advantage was data: hundreds of millions of images that users had saved to themed boards, which gave a rich signal about which images people considered related. Its systems let users select a region of any image and find visually similar pins, many of them linked to products. Alibaba's Pailitao, which lets shoppers on Taobao search by photograph, was described in a 2018 paper by Yanhao Zhang and colleagues. It had to cope with a catalogue of billions of images and a very high query volume. Its pipeline predicted the category of the query image first, to narrow the search, then detected the main object, computed an embedding with a network trained using shoppers' click behavior as weak supervision, and searched an index of billions of vectors. eBay published a description of its own visual search system in 2017, with an emphasis on serving a constantly changing inventory of listings uploaded by sellers, whose photographs vary enormously in quality. All of these systems share a final component that deserves attention: the nearest-neighbor index. Comparing a query vector against billions of catalogue vectors one at a time would take far too long, so systems use approximate nearest-neighbor search, which trades a small chance of missing the true nearest neighbor for speed. The two most influential approaches are product quantization, which compresses vectors into short codes that can be compared quickly and which is implemented in Meta's open-source FAISS library, and graph-based methods such as hierarchical navigable small world graphs, introduced by Yury Malkov and Dmitry Yashunin, which link each vector to its near neighbors so that a search can hop through the graph toward the query. These indexes are now standard infrastructure, used for text embeddings and recommendation retrieval as well as images. Multimodal models and the end of the purely visual The most significant change in visual search since its early years came from models trained jointly on images and text. OpenAI's CLIP, described by Alec Radford and colleagues in 2021, was trained on hundreds of millions of image and caption pairs from the internet to place images and their descriptions near each other in a shared embedding space. The result was a model that could match images to arbitrary text descriptions it had never been specifically trained on, and whose image embeddings captured semantic properties, style, material and context, that pure image models handled less well. For retail, multimodal embeddings dissolved the boundary between visual and text search. A shopper can now combine the two: upload a photograph of a sofa and add "but in green velvet", or describe "a rug that would go with this" and supply a photo of a room. Because text and images share a space, the system can move the query vector from the photograph in the direction the words indicate. Multimodal embeddings also improve catalogue quality behind the scenes. They can tag products with attributes the seller never entered, detect duplicate listings, find products whose images contradict their descriptions, and place a newly listed item near similar ones on the day it arrives, which is the cold-start solution mentioned in Chapter 2. General-purpose models of this kind are rarely used unmodified; retailers fine-tune them on their own catalogue and behavioral data, since a model trained on internet captions knows that a picture shows a "blue dress" but not which of a retailer's four thousand blue dresses shoppers will regard as alternatives to it. Generative models extend the capability further. Virtual try-on systems, which show how a garment would look on the shopper or on a model of a chosen body shape, and room visualization tools, which place a piece of furniture into a photograph of the shopper's own room, are now offered by several large retailers and platforms. These are not search in the strict sense, but they rely on the same representations and they address the same underlying problem: shoppers cannot touch or try products online, and images are the closest substitute. Anatomy of a single query Following one query through a modern system pulls these pieces together. A shopper sees a chair in a hotel lobby and photographs it with a retailer's app. The photograph arrives at the server as a few megabytes of pixels showing the chair, part of a rug, a side table, a potted plant and a window. First, an object detector proposes regions of the image that contain products: the chair, the side table, the rug, the plant pot. The app shows these as tappable outlines and highlights the most prominent, the chair, by default. The image is cropped to the chair's region and normalized for size and brightness. Second, a classifier predicts the broad category, in this case seating, and perhaps some attributes: armchair, upholstered, wooden legs, mustard color. These predictions narrow the search to the relevant part of the catalogue and supply attribute filters the shopper can adjust. Third, an embedding network converts the cropped image into a vector. The approximate nearest-neighbor index for the seating category returns the few hundred catalogue items whose image vectors lie closest. This takes a few milliseconds even across millions of items. Fourth, a ranking model reorders those candidates. It considers visual similarity, but also availability, delivery time to the shopper, price, ratings and the shopper's own history, for instance whether they have previously bought at higher or lower price points. Items out of stock are removed or demoted. The shopper sees perhaps twenty chairs, ordered by this combined score. Finally, the shopper's behavior, which results they tap, which they save, which they buy, is logged. That log will become training data for the ranker and, in many systems, weak supervision for the next version of the embedding network. The first three steps are perception, and their quality depends on how well the models represent images. The fourth step is a decision, combining similarity with commercial considerations, and it is here that the system's objectives, and the feedback loop, enter. A team diagnosing a complaint that visual search "shows the wrong things" needs to know which step failed: a detector that cropped the rug, an embedding that matched the color but not the shape, or a ranker that pushed the most similar chair below more profitable ones. Evaluating visual search How good is a visual search system? The obvious measure, how often the exact product in a photograph is found among the top few results, is useful in research but not always meaningful in practice. Shoppers photographing a jacket on the street often cannot buy that exact jacket, which may not be sold by the retailer at all. What they want is something similar, and similarity is subjective. Two jackets may match in shape but not color; one shopper cares about shape, another about color. Retailers therefore rely heavily on human judgement. Panels of raters assess whether results are relevant to queries, producing a relevance score that guides model development. Online, the same tools as in Chapter 3 apply: A/B tests compare versions of the system on engagement and purchases. Here the closed loop returns. Once visual search results are ranked and shown, shoppers click on what they are shown, and if those clicks are used to train the next version of the embedding or the ranker, the system starts to learn its own biases. A visual search system that ranks popular products higher, because they get more clicks, will gradually show popular products for queries where less popular ones look more similar. The perception part of the system is reliable; the ranking part is subject to every problem discussed in the personalization chapters. How visual search fails The failures of visual search systems are instructive because they reveal what the embeddings have and have not learned. The most common is matching on the wrong attribute. An embedding trained mainly on catalogue images may weight color heavily, because color is the most obvious difference between product photographs, so that a photograph of a red trench coat returns red coats of every cut rather than trench coats of every color. Which attribute should dominate depends on the shopper's intent, which the photograph does not reveal. Systems mitigate this by offering attribute controls after the search and, increasingly, by letting the shopper refine with text. A second failure is sensitivity to context the model should ignore. Products photographed on models may be matched partly on the model's pose or skin tone rather than the garment, and furniture photographed in staged rooms may be matched on the room. These are not merely accuracy problems; when they correlate with who the shopper is, they become fairness problems. Careful training data, including deliberate variation in models and settings, and evaluation broken down by such factors are the remedies. A third is the catalogue itself. Visual search can only find what has been photographed well. Marketplaces with millions of third-party sellers contain images that are blurred, watermarked, composited, shot at odd angles or reused from other listings. A sophisticated embedding cannot rescue a catalogue in which the image for a product shows a different product. Much of the practical work of improving visual search is therefore catalogue hygiene: detecting and removing duplicate, misleading or low-quality images, often using the same embeddings that power the search. Where visual search earns its place Visual search is impressive to demonstrate and harder to justify commercially than it looks. Most shoppers, most of the time, know roughly what they want and can type it. The share of searches that start with a camera is modest at most retailers, though larger in categories where appearance is the product and vocabulary is weak. The categories where it creates the most value share three features. Appearance matters more than specification: fashion, furniture, home decor, art and jewelry, rather than electronics, where shoppers search by model number. The vocabulary is specialized or unknown to shoppers: few people can name a style of chair or a weave of fabric. And the catalogue is large enough that browsing is impractical. A fourth category of value is less visible: industrial and automotive parts, where a technician who needs to replace a worn component can photograph it rather than search for a part number they do not know. Several industrial distributors have built visual search for exactly this purpose. Perhaps the most important business value of visual search technology lies not in the camera button at all, but in what the underlying embeddings make possible behind the scenes: "more like this" recommendations on every product page, automatic tagging of attributes, detection of counterfeit listings that reuse genuine product photographs, identification of duplicate products across sellers, and the cold-start representations that let new products be recommended immediately. A retailer considering visual search should evaluate the whole family of uses, not only the one that shoppers see. The lesson of the exception Visual search shows what machine learning can do when the problem is primarily one of perception and the training signal is relatively free of the retailer's own influence. Whether two images depict similar sofas does not depend on what the retailer chose to stock or how it priced them. That is why progress in visual search has tracked progress in computer vision closely, and why off-the-shelf models trained on general data transfer to retail tasks as well as they do. The contrast with pricing and personalization is instructive. In those areas, better models alone have not delivered the same steady improvement, because the bottleneck is not representation but causation: the data do not contain the answer to the question being asked. Visual search reaches that bottleneck only at the ranking stage, and there it faces the same problems as every other retail ranking system. Recognizing which parts of a problem are perception, where more data and bigger models reliably help, and which are decision, where they do not, is a practical skill worth developing. Chapter 7: The Order-Picking Problem Every online order ends in a building. Somewhere, a person or a robot must go to the shelves where the ordered items are stored, collect them and bring them to be packed. This work, order picking, is the most labor-intensive and costly activity in most warehouses. A review of the research by René de Koster, Tho Le-Duc and Kees Jan Roodbergen, published in 2007 and still widely cited, noted estimates that picking can account for as much as 55 percent of warehouse operating cost, and that roughly half of a manual picker's time is spent traveling between locations. Reduce the travel and the cost falls. E-commerce has made this problem harder. A store's replenishment order is large and predictable: whole cases of a few hundred products, delivered on a schedule. An online retailer's orders are small, numerous and urgent: one or two items each, from a catalogue of hundreds of thousands, promised for delivery tomorrow or today. The number of separate trips to the shelves multiplies, and so does the value of making each one efficient. This chapter describes the picking problem and the classical methods for solving it. The point of spending a chapter on methods that are decades old is that they remain very good, and that any claim for reinforcement learning, the subject of the next chapter, has to be measured against them. Anatomy of a picking operation Warehouses organize picking in two broad ways. In picker-to-parts systems, which remain the majority worldwide, human pickers walk or drive through aisles of shelving or racking, guided by a handheld scanner or a voice headset, collecting items into a cart. In parts-to-picker systems, the items come to a stationary picker: shelves are carried by mobile robots, or storage bins are retrieved from a dense grid by robots running on top of it, and the picker takes what is needed from each shelf or bin as it arrives at a workstation. Within either system, several decisions determine how much travel is needed. Storage assignment decides where each product lives; fast-moving items are usually placed close to the dispatch area, a practice known as class-based storage. Batching decides which orders are picked together on one trip; a picker who collects items for ten orders at once can share the walking among them. Zoning decides whether each picker covers the whole warehouse or a section of it. Routing decides the sequence in which a picker visits the locations on their list. And for robotic systems, a further set of decisions governs which robot does which task, which path it takes, and how robots avoid one another. These decisions interact. The best routing method depends on how many items are on each pick list, which depends on batching. The best batching depends on where items are stored. A good storage plan for one season's demand may be poor for the next. Most research addresses one decision at a time with the others held fixed, which is why practical results are often weaker than results in papers: in the warehouse, all of them change together. Routing: a tractable special case Consider the narrowest version of the problem. A picker starts at a depot at the front of a rectangular warehouse with parallel aisles, a cross aisle at the front and another at the back. They have a list of locations to visit. What is the shortest route that visits all of them and returns to the depot? In general, finding the shortest tour through a set of points is the travelling salesman problem, which is computationally hard: no known method solves every large instance quickly. But the geometry of a warehouse is special. In 1983 H. Donald Ratliff and Arnon Rosenthal showed that for a warehouse with parallel aisles and cross aisles only at the ends, the shortest route can be found by dynamic programming in time that grows only linearly with the number of aisles. Later work extended exact methods to warehouses with a middle cross aisle and to some other layouts. For the most common layout, the optimal route is therefore not a matter of approximation; it can be computed exactly and quickly. Remarkably, many warehouses do not use optimal routes. They use simple heuristics, for reasons that have as much to do with people as with mathematics. Optimal routes can look strange: a picker may be told to enter an aisle from the back, go halfway, turn around and leave, then go to a distant aisle and come back. Pickers who do not understand a route make mistakes, skip locations or take shortcuts that are worse. Heuristic routes follow simple, consistent patterns that pickers can learn, and the extra distance they add is often modest. The main heuristics differ in how they traverse aisles, and their relative performance depends on how many items are on each pick list, as Table 3 summarizes. The S-shape, or traversal, method has the picker walk the full length of every aisle containing an item, in a snake pattern. The return method has the picker enter each aisle from the front, go as far as the deepest item and come back. The midpoint method divides the warehouse into front and back halves and picks each half from its own side. The largest-gap method, introduced by Randolph Hall in 1993, enters each aisle from whichever end avoids the largest empty stretch of the aisle. Combined methods, analyzed by Roodbergen and de Koster in 2001, choose for each aisle between traversing and returning according to which is shorter given the next aisle. Table 3. Classical picker routing heuristics in a single-block warehouse. Method Aisle pattern Works best when Typical drawback S-shape Traverse whole aisle Many picks per aisle Wasted walking in sparse aisles Return In and back from front Picks near the front Deep picks are costly Midpoint Each half from its side Few picks per aisle Crosses warehouse twice Largest gap Skip largest empty stretch Few to moderate picks Harder for pickers to follow Combined Choose per aisle Most conditions More complex instructions Optimal Dynamic programming Always shortest Routes can look erratic Source: de Koster, Le-Duc and Roodbergen (2007); Roodbergen and de Koster (2001). Simulation studies over several decades have produced a consistent picture. With few items per pick list, the largest-gap and midpoint methods beat S-shape. With many items per list, S-shape becomes competitive because most aisles must be traversed anyway. Combined methods come within a few percent of optimal in many configurations. The gap between the best heuristic and the optimal route is usually smaller than the gap between good and poor storage assignment or batching. Routing gets attention because it is easy to state, but in many warehouses it is not where most of the savings lie. Storage: deciding where things live If routing gets more attention than it deserves, storage assignment, often called slotting, gets less. Where each product is placed determines how far pickers must travel to reach it, and that effect compounds over every pick of every order. The classical approach ranks products by how often they are picked and assigns the most frequently picked products, the A class in a scheme borrowed from inventory management, to the locations closest to the depot or the packing stations, the B class to the next zone, and the C class to the rest. Many warehouses find that a small fraction of their products accounts for most of their picks, which makes this approach powerful: putting the fastest movers near the front can cut travel substantially before any routing algorithm is applied. Refinements consider more than pick frequency. Cube-per-order index rules weigh how often an item is picked against how much space it occupies, so that small, fast-moving items get the prime locations rather than bulky ones. Correlated storage places items frequently ordered together near each other, which shortens the routes for orders containing both; in grocery fulfillment, for example, items commonly bought together, such as pasta and pasta sauce, can be co-located. Ergonomic rules place heavy items at waist height and light ones higher or lower. Slotting has its own feedback loop and its own difficulty with time. Demand shifts with seasons and trends: garden furniture is a fast mover in May and a slow one in November. A warehouse slotted for one season is mis-slotted for the next, and re-slotting means physically moving stock, which costs labor and interrupts picking. Most warehouses re-slot periodically, weighing the cost of moves against the travel saved, and the decision of when and how much to re-slot is itself an optimization that classical methods treat only roughly. Concentrating fast movers in a few locations also creates congestion, a point taken up below. The interaction between where items live, how orders are batched and how pickers move is where much of the practical difficulty of warehouse design resides. Batching, waves and the clock If routing is tractable, batching is not. Choosing which orders to combine into one picking trip so as to minimize total travel is a hard combinatorial problem; even its simple versions belong to the class of problems for which no efficient exact method is known. Practitioners use heuristics: seed algorithms, which start a batch with one order and add the orders whose locations are closest; savings algorithms, adapted from vehicle routing, which merge orders whose combination saves the most distance; and, increasingly, metaheuristics such as local search and genetic algorithms that improve batches iteratively. E-commerce adds the dimension of time. Orders arrive continuously, each with a promised dispatch deadline tied to a carrier's collection time. Waiting longer before releasing a batch allows more orders to accumulate, which means better batches with less walking per item. But waiting uses up time before the deadline, and if a batch is released too late, some orders miss their truck. Warehouses traditionally handled this through waves, releasing work in scheduled blocks timed to carrier departures. Many have moved to continuous or waveless release, in which software decides moment by moment what to release next, considering deadlines, available pickers and current congestion. This online version of the problem, in which decisions must be made as information arrives and without knowing what orders will come next, is much harder than the static version studied in most of the classical literature. It is also where the decision has the character of the closed loop: what the system releases now determines which pickers are busy, where they are and how congested each zone becomes, and that state shapes the next decision. That property makes it a natural candidate for the methods in the next chapter. Congestion Classical routing analyses usually assume a single picker. Real warehouses have dozens or hundreds working the same aisles. In narrow aisles, two pickers cannot pass, so one waits; in wider aisles, traffic still slows everyone down near popular locations. Studies of narrow-aisle picking have found that blocking can significantly erode the productivity gains of good routing, especially when fast-moving items are concentrated in a few aisles, which is exactly what class-based storage recommends. The storage plan that minimizes travel for a single picker can maximize congestion for many. Congestion is the first point at which the problem stops being a set of independent optimizations and becomes a system in which every agent's decisions affect every other agent's outcome. A routing algorithm that ignores other pickers will send them all down the same short paths and create jams. Classical methods address this by staggering start times, assigning pickers to zones, or adding simple traffic rules such as one-way aisles. These work, but they are coarse. The robotic warehouse The most important change to order picking in the past two decades has been the arrival of mobile robots. Kiva Systems, founded in 2003, built small robots that drive under portable shelving units, lift them and carry them to stationary picking stations. Peter Wurman, Raffaello D'Andrea and Mick Mountz described the system in a 2008 article that framed it explicitly as a problem in multi-agent coordination. Amazon bought Kiva in 2012 for about 775 million dollars and turned it into the core of its fulfillment network. In mid-2025 Amazon said it had deployed its one millionth robot across its operations. Robotic systems change the picking problem in two ways. First, they remove the human routing problem: the picker stands still and the shelves come to them. Second, they replace it with a much larger coordination problem. Hundreds or thousands of robots must be assigned tasks, routed across a grid and kept from colliding, while the system decides which shelf to bring to which station, in which order, and where to return it afterwards. Each shelf holds many different products, so bringing one shelf can serve several orders at once if the timing is right, and returning a shelf to a location near the stations makes it faster to fetch next time if its products are in demand. The routing part of this problem is studied in computer science as multi-agent path finding: given a set of agents on a graph, each with a start and a goal, find paths for all of them that do not collide. Exact methods, such as conflict-based search introduced by Guni Sharon and colleagues in 2015, plan paths for each agent and resolve collisions one at a time; they are optimal but slow down sharply as the number of agents grows. Practical warehouse systems therefore use fast approximate planners, often planning only a short horizon ahead and replanning continuously as tasks arrive. Even so, as robot numbers grow, congestion at intersections and near stations becomes a binding constraint, and gains from better coordination become large. A different family of robotic systems, the cube storage grids sold by companies such as AutoStore and used in some grocery fulfillment centers, stacks storage bins densely in a grid with robots running on rails across the top. To retrieve a bin, a robot may first have to lift and set aside the bins stacked above it. The routing problem largely disappears, since robots move on a regular grid, but it is replaced by questions about which bins to keep near the top. A bin that has just been used is usually returned to the top of a stack, so frequently needed bins tend to rise naturally, a self-organizing behavior that is a crude form of slotting. Deciding which robot fetches which bin, in which order, and which bins to dig out in advance of expected demand are sequential decisions of the same character as those in shelf-carrying systems. Why the classical toolkit is not enough Taken together, the classical methods handle the static, single-agent version of picking very well. Optimal routes are computable. Good heuristics are close to optimal. Storage, batching and zoning have well-studied rules of thumb. A warehouse that applies these carefully will be efficient. What the classical toolkit handles less well is everything this chapter has added since: orders arriving continuously under deadlines, many agents interfering with one another, decisions whose effects ripple through the state of the whole system, and conditions that change by the hour as demand shifts. Those features make warehouse operations a sequential decision problem under uncertainty, played out by many agents at once. That is the problem class reinforcement learning was designed for. Whether it delivers in practice, and at what cost, is the question for the next chapter. Chapter 8: Reinforcement Learning in the Warehouse Reinforcement learning is the branch of machine learning concerned with sequential decisions. An agent observes the state of an environment, takes an action, receives a reward, and finds itself in a new state; over many such interactions it learns a policy, a rule mapping states to actions, that maximizes its total reward over time. The approach became famous through games, Atari in 2013 and Go in 2016, where an agent could play millions of times and learn strategies no human had taught it. The warehouse looks like a natural home for these methods. Picking is a sequence of decisions, each one changing the state in which the next is made. The objective, finishing work quickly and on time, is easy to express as a reward. And the hardest parts of the problem, as Chapter 7 ended by observing, are exactly those that classical methods handle poorly: continuous arrival of orders, many interacting agents, and decisions whose consequences unfold over time. This chapter explains how reinforcement learning is applied to picking routes and related warehouse decisions, what it has achieved, and why it is better used selectively than everywhere. Framing picking as a decision process To apply reinforcement learning, a problem must be cast as a Markov decision process: a definition of states, actions, rewards and how states change. The choices made here determine almost everything about whether the approach succeeds. For a single picker's route, the state might include the picker's current location, the set of locations still to be visited and perhaps the positions of other pickers. The action is the choice of next location. The reward is the negative of the time taken to reach it, so that maximizing total reward means minimizing total travel time. An episode ends when the list is complete. For a whole warehouse, the formulation grows. The state includes every picker's or robot's position and task, the queue of unreleased orders with their deadlines, inventory locations and current congestion. Actions might include which orders to release into the next batch, which picker or robot to assign to which task, and which path each should take. Rewards might combine travel time, deadline penalties and penalties for collisions or blocking. A sketch of how such an environment is typically exposed to a learning algorithm makes the structure concrete: class PickingEnv: def reset(self): self.state = sample_initial_orders_and_layout() return observe(self.state) def step(self, action): # action: next location for the active picker travel = path_time(self.state, action) self.state = advance(self.state, action, travel) reward = -travel - late_penalty(self.state) done = all_orders_complete(self.state) return observe(self.state), reward, done Everything interesting is hidden inside advance and path_time: how other pickers move, how congestion slows travel, how new orders arrive. That hidden machinery is the simulator, and as the rest of this chapter will show, the quality of the simulator is usually the quality of the result. Learning to route The first wave of research applied reinforcement learning to routing problems in general, not to warehouses in particular. In 2015 Oriol Vinyals and colleagues introduced pointer networks, neural networks that output a sequence of positions from their input, and showed they could learn approximate solutions to small travelling salesman problems. Irwan Bello and colleagues at Google trained such networks with reinforcement learning the following year, using tour length as the reward. In 2019 Wouter Kool, Herke van Hoof and Max Welling published an attention-based model, trained with reinforcement learning, that produced near-optimal tours for travelling salesman and vehicle routing problems with up to about a hundred locations, quickly and without hand-designed rules. Their paper was widely read and spawned a substantial literature on neural combinatorial optimization. The results deserve a careful reading. On the problem sizes they were trained on, the learned models produced good solutions fast. But they did not beat the best specialized solvers, which remain more accurate, and their performance often degraded on instances larger than those seen in training. Their main advantage was speed of solution once trained, and the ability to be retrained for problem variants for which no specialized solver had been written. Applied to warehouse picking, this has a sobering implication. For a single picker in a standard rectangular warehouse, Chapter 7 explained that the optimal route can be computed exactly and quickly by dynamic programming. A learned routing policy cannot improve on the optimum. It can at best match it, at considerable cost in training. Reinforcement learning for single-picker routing in a standard layout is a solution to a problem that has already been solved. Several published studies applying reinforcement learning to picker routing compare their methods only against simple heuristics such as S-shape, not against optimal routes or the best combined heuristics, which makes their reported gains hard to interpret. Any reader evaluating such a claim should ask first what the baseline was. Where learning earns its place The situation changes when the problem moves away from the static, single-agent case. Reinforcement learning becomes worth its cost under three conditions, each of which describes much of modern e-commerce fulfillment. The first is dynamic arrival. When orders arrive continuously and decisions must be made before knowing what comes next, the problem is no longer a fixed optimization but a policy question: given the current state and what typically arrives, what should be done now? Should the system release this batch immediately or wait for more orders in the same zone? Should a picker finish their current list or divert to an urgent order? Classical methods handle this through rules of thumb tuned by experience. Reinforcement learning can learn policies that anticipate typical arrival patterns, such as the surge of orders that follows an evening promotion, and adjust release and assignment decisions accordingly. Several research groups have reported that learned policies for order release and batching under deadlines outperformed fixed rules in simulation, typically by modest percentages. The second is many interacting agents. When hundreds of pickers or robots share the same aisles, each agent's best action depends on what the others do. Planning all their paths jointly is computationally intractable at scale, and planning each separately ignores congestion. Multi-agent reinforcement learning trains agents that use local observations, such as nearby robots and congestion ahead, to act in ways that work well collectively. A 2019 method called PRIMAL, by Guillaume Sartoretti and colleagues, combined reinforcement learning with imitation of an expert planner to train robots that could find collision-free paths in crowded grids using only local information, and that scaled to more agents than exact planners could handle. Subsequent work in this vein has continued to narrow the gap between learned decentralized policies and centralized planners while scaling far beyond them. The third is predicting the system's own congestion. Here the most visible industrial development is Amazon's DeepFleet, announced in mid-2025 alongside its millionth robot. Amazon described DeepFleet as a foundation model trained on data from its robot fleet to predict and coordinate robot movement, and said it would improve the travel efficiency of the fleet by about ten percent. Amazon's description is a company announcement, not a peer-reviewed evaluation, and the details of the model are not public. But the shape of the claim is consistent with the argument of this chapter: the gain comes from coordinating many agents in a congested shared space, not from finding better paths for any single one of them. Table 4 sets out where in warehouse operations learning methods appear to add value relative to classical approaches, and where they do not. Table 4. Where reinforcement learning adds value in order picking. Decision Classical approach Value of RL Why Single-picker route Exact dynamic programming Little or none Optimum already computable Static batching Seed and savings heuristics Modest Good heuristics exist Release under deadlines Waves, fixed rules Moderate to high Must anticipate arrivals Multi-robot paths Planners with replanning High at large scale Joint planning intractable Slotting over time Periodic ABC rules Promising, unproven Long horizons, slow feedback A release decision, step by step The order-release problem shows most clearly what a learned policy does that a fixed rule cannot. Picture a fulfillment center at four in the afternoon. The last carrier collection is at eight. Three hundred orders are waiting, and more arrive every minute. Forty pickers are on shift, some mid-route, some idle. Every few minutes the system must decide which waiting orders to release into the next set of pick lists. A fixed rule might say: every ten minutes, release all orders due on the next collection, batched by zone. That rule is simple and predictable. It ignores, however, several things an experienced supervisor would consider. If a large share of waiting orders sit in one zone, releasing them all at once will congest that zone while others are idle. If the evening's order surge usually begins at five, holding some capacity in reserve now may prevent missed deadlines later. If several waiting orders include items stored in a remote part of the building, it may be worth waiting a few minutes for more orders that need that area, so one trip serves many. A reinforcement learning policy trained in a calibrated simulator sees the same state the supervisor sees: queue length by zone, deadlines, picker positions, time of day and recent arrival rates. Over millions of simulated afternoons it learns which release decisions in which states lead to the fewest missed deadlines and the least travel by end of shift. It might learn, for example, to release orders for congested zones in smaller increments, to front-load work for remote areas before the surge, and to switch to releasing everything urgently once the time to cut-off falls below a threshold that depends on how much work remains. None of these behaviors needs to be programmed. They are what emerges from optimizing the long-run reward. The important point is that the value of each decision depends on what comes afterwards. Releasing a large batch now looks efficient in isolation, because it groups many orders into short routes, and looks poor in hindsight if it congests a zone just as urgent orders arrive. Classical batching heuristics evaluate a decision by its immediate effect. Reinforcement learning evaluates it by its effect on everything that follows, which is precisely what the closed loop requires. Simulators and the reality gap Reinforcement learning learns by trial and error, and no warehouse operator will allow a learning agent to experiment on live orders with real deadlines. Learning therefore happens in simulation, and the policy is transferred to the real warehouse afterwards. This creates the central practical difficulty of the field: the policy is only as good as the simulator's fidelity to reality. Warehouse simulators are good at some things and poor at others. Layout, distances and robot kinematics can be modelled accurately. Human pickers are harder: their walking speed varies with fatigue, shift, experience and load; they take breaks; they make errors; they deviate from instructions in ways that matter. Order arrival patterns shift with seasons, promotions and events. Equipment breaks down. A policy trained in a simulator that omits these effects may exploit features of the simulator that do not exist in reality, for example assuming that a picker can always reach a location in exactly the computed time, and perform worse than a simple rule when deployed. Practitioners use several techniques to narrow this reality gap. Domain randomization trains the policy across many variations of the simulator's parameters, walking speeds, arrival rates, breakdown frequencies, so that it learns to be robust rather than to exploit one setting. Calibration fits the simulator to historical data from the real warehouse, using recorded travel times and order patterns. Digital twins go further, keeping a simulator continuously synchronized with live operational data, so that policies can be tested against the current state of the warehouse before being acted on. Each of these helps. None removes the need to validate in the real building. The closed loop appears here too. Historical travel-time data were generated under the previous routing and release policies. If those policies created congestion in certain aisles at certain times, the recorded travel times there will be long. A simulator calibrated on that history and a policy trained in it will learn to avoid those aisles, even though under the new policy they might no longer be congested. Calibrating a simulator from data produced by the policy it is meant to replace is the warehouse equivalent of training a recommender on its own clicks. Rewards, safety and fallback Two further issues decide whether a learned policy can be trusted in operation. The first is reward design. A reinforcement learning agent optimizes exactly what it is told to, and finds unexpected ways to do so. A policy rewarded only for picks per hour may learn to defer difficult orders, items stored high up or far away, until they are close to their deadline, keeping its productivity high while pushing risk to the end of the shift. A policy penalized only for late orders may become extremely conservative, releasing small batches early and wasting travel. Practical reward functions combine several terms, and teams spend substantial effort checking learned behavior for these pathologies by inspecting what the policy actually does across many scenarios, not just its total reward. The second is safety. In robotic systems, a collision is expensive and a deadlock, where robots block each other indefinitely, can halt a whole section. Learned policies are therefore deployed inside guardrails. Action masking prevents the policy from choosing actions that a rule-based check identifies as unsafe. Shielding wraps the policy in a verified controller that overrides it when necessary. And fallback mechanisms revert to a classical planner when the learned policy encounters a situation outside its training distribution or performs poorly. In most deployed systems, the learned component is advisory or bounded rather than in full control. A common and effective pattern is hybrid: reinforcement learning chooses among classical heuristics or tunes their parameters rather than replacing them. A learned policy might decide, every few minutes, whether to use largest-gap or S-shape routing given current pick densities, or set the waiting time before batch release given current arrival rates. The classical methods provide reliable behavior; learning adapts them to conditions. Such hybrids are easier to explain to operators, easier to verify, and degrade gracefully. Evaluating learned policies in the building Evaluating a new warehouse policy in reality raises the same problems as the experiments in Chapter 3, with an additional twist. Pickers and robots share space, so randomizing individual pickers to the new policy creates interference: if new-policy pickers avoid congestion, they may push it onto old-policy pickers, making the new policy look better than it would be if everyone used it. The standard answer is switchback experiments, in which the whole warehouse, or a whole zone, alternates between policies over time blocks, for example by shift or by day, with the schedule randomized. Comparing across blocks controls for interference, at the cost of needing more blocks to average out differences in order volume. Measured outcomes should include not only travel and throughput but deadline performance, picking errors and, where people are involved, how the policy feels to work under. A policy that saves walking by sending pickers on routes they find confusing or that changes their instructions constantly may increase errors and turnover, which cost more than the walking saved. These effects are not captured in any simulator. A measured verdict Reinforcement learning in the warehouse is a genuine advance for a specific class of problems: dynamic, multi-agent coordination at scale, where classical planning becomes intractable and fixed rules leave value unused. In large robotic fulfillment centers, that class covers much of what matters, and the gains reported there are credible in kind if not always verifiable in amount. For the classical, single-picker routing problem that much of the research literature still addresses, it offers little that dynamic programming and good heuristics do not already provide. The deciding factor in most deployments is not the learning algorithm but the surrounding engineering: the fidelity of the simulator, the design of the reward, the guardrails, and the experimental discipline used to confirm that a policy that wins in simulation also wins in the building. That is the same lesson the earlier chapters drew for personalization and pricing, arrived at in a physical setting where the feedback loop can be seen and timed. Conclusion Across personalization, pricing and the warehouse, the same pattern has recurred. The models are good and getting better. Recommenders built on sequence transformers predict the next click more accurately than matrix factorization did. Demand models built on gradient boosting forecast better than exponential smoothing. Learned robot coordinators handle fleets that exact planners cannot. And yet in each domain the decisive question has turned out not to be about the model at all. It has been about the data the model learns from, which the retailer's own past decisions wrote, and about the consequences of acting on the model, which change the data it will learn from next. Three conclusions follow from that, and they are less obvious than the observation itself. The scarce asset is deliberate variation The first is that the most valuable thing a retailer can build for its machine learning is not a model but a supply of deliberate, recorded variation in its own decisions. Every chapter has arrived at a version of this. Personalization teams that log the probability with which each item was shown can evaluate new policies offline; teams that do not must run an experiment for each idea. Pricing teams that build a small amount of randomized price movement into routine operations get elasticity estimates they can trust; teams that rely on historical prices get estimates contaminated by their own merchandising calendar, however sophisticated the estimator. Warehouse teams that alternate policies in randomized time blocks know what their new routing actually saves; teams that compare this month with last month are measuring the weather. This variation is cheap if planned and expensive if retrofitted. It costs a little in the short run, since some shoppers see slightly worse recommendations and some prices are slightly off the believed optimum, and that cost is visible on dashboards while the benefit is not. That asymmetry is why it tends to be under-supplied. A retailer's data strategy is, in large part, a decision about how much exploration to fund, and it deserves to be made explicitly at a senior level rather than left to individual teams who are each measured on short-term performance. The loops are coupled The second conclusion is that the loops described separately in this booklet are in practice connected, and optimizing them separately can set them against one another. A personalization system that steers shoppers toward a narrower set of popular items changes the demand those items face, and therefore the elasticity the pricing system estimates for them. A pricing system that runs a deep promotion on a popular item creates a surge of orders concentrated in a few storage locations, and therefore congestion that the warehouse routing system must absorb, often at the worst moment, just before carrier cut-off. A warehouse that cannot fulfill same-day orders for certain products in certain regions changes which products the recommender should be promoting there, since a recommendation for something that will arrive in four days converts worse than one for something arriving tonight. Each team, optimizing its own metric against its own model, treats the others' decisions as part of the environment. Each sees an environment that keeps shifting for reasons it cannot explain. The organizations that handle this well do not necessarily build one grand model. They make the coupling visible: forecasting systems that take planned promotions as inputs and pass expected order volumes to fulfillment; recommenders that know delivery promises by region; pricing systems that see warehouse capacity. And they evaluate major changes on outcomes that span the loops, total contribution margin after fulfillment cost and returns, rather than on each team's local metric. The closed loop is not only a property of each model. It is a property of the business. Accountability runs through the same infrastructure The third conclusion concerns the growing external scrutiny of retail algorithms, from regulators examining surveillance pricing and algorithmic collusion to legislators requiring price disclosures and competition authorities examining search ranking. It is tempting to treat compliance as a separate burden, layered on top of the technical work. The argument of this booklet suggests otherwise. The questions regulators ask are causal questions of exactly the kind retail data science must answer anyway. Did this pricing algorithm raise prices relative to what competition would have produced? Did this ranking system favor the retailer's own products over what shoppers would have chosen? Did personal data change the price this shopper was offered? A retailer that cannot answer these questions about its own systems also cannot know whether those systems are working. Conversely, the infrastructure that makes good machine learning possible, decision logs that record what was done and why, randomized variation, experiments with clear counterfactuals, simulators that can replay policies, is the same infrastructure that makes it possible to answer an auditor credibly. The Instacart episode of late 2025 shows the cost of the alternative: a company running price tests that may have been entirely conventional found itself unable to make that case quickly to a public that saw only two prices for one item. This points to a practical design principle. Where the methods that generate learning, such as randomized pricing, collide with public expectations of fairness, the response is not to abandon them but to choose their form deliberately: randomizing over time or region rather than individuals where that suffices, bounding the size of variation, and being prepared to disclose what is being tested. Fairness and learning are not opposed. Opacity is what turns one into a liability for the other. What remains unsettled Some of the hardest questions in this field are open. The first is what happens when the shopper is also an algorithm. In 2026, AI shopping agents that search, compare and in some cases buy on a consumer's behalf are moving from demonstrations into use. If a meaningful share of purchases is made by agents, then retailers' pricing and ranking algorithms will increasingly be responding to other algorithms rather than to people. Elasticities estimated from human shoppers, who do not compare every price, may not hold for agents that do. Recommendations designed to catch the eye will matter less than structured product data an agent can parse. The closed loop gains a new participant whose behavior is shaped by someone else's model, and the consequences for competition, for price levels and for retailers' margins are genuinely unknown. The second is how far the law will reach into algorithmic coordination without agreement. The economic research reviewed in Chapter 5 shows that learning algorithms can sustain high prices without any communication; the law has historically required evidence of agreement. Whether competition authorities and courts develop a doctrine that addresses algorithmic outcomes directly, and what that would require of retailers' pricing systems, will shape the field for years. The third is how the gains from warehouse automation will be shared. The efficiency improvements described in Chapters 7 and 8 are real, but their effect on the people who work in fulfillment, on the pace and character of their work and on how many of them are needed, is a question that operations research cannot settle. What to do differently For a reader responsible for retail machine learning, the practical upshot can be stated plainly. Judge every model by what happens when the business acts on it, not by how well it predicts. Fund exploration and record it, so that the data needed to answer causal questions exist when the questions arrive. Prefer strong classical baselines and require new methods to beat them in controlled tests, not in backtests. Treat personalization, pricing and fulfillment as coupled systems and measure changes on outcomes that span them. And build the logging and experimentation infrastructure that makes it possible to explain, to customers and regulators as well as to colleagues, what the systems do and why. None of this is glamorous. It will not appear in a product announcement. But it is what separates the retailers whose machine learning keeps delivering from those whose models get steadily more accurate while the business gets no better. In a market the models help to shape, understanding the shaping is the whole game. Notes Introduction 1. Greg Linden, Brent Smith and Jeremy York, "Amazon.com Recommendations: Item-to-Item Collaborative Filtering," IEEE Internet Computing 7, no. 1 (2003): 76–80. 2. On the share of warehouse operating cost attributed to order picking, see René de Koster, Tho Le-Duc and Kees Jan Roodbergen, "Design and Control of Warehouse Order Picking: A Literature Review," European Journal of Operational Research 182, no. 2 (2007): 481–501. Chapter 1 1. Spyros Makridakis, Evangelos Spiliotis and Vassilios Assimakopoulos, "M5 Accuracy Competition: Results, Findings, and Conclusions," International Journal of Forecasting 38, no. 4 (2022): 1346–1364. 2. David Salinas, Valentin Flunkert, Jan Gasthaus and Tim Januschowski, "DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks," International Journal of Forecasting 36, no. 3 (2020): 1181–1191; first circulated in 2017. 3. Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy and Dhruv Madeka, "A Multi-Horizon Quantile Recurrent Forecaster," NIPS Time Series Workshop, 2017. 4. Abdul Fatir Ansari and colleagues, "Chronos: Learning the Language of Time Series," Transactions on Machine Learning Research, 2024. 5. Shanika L. Wickramasuriya, George Athanasopoulos and Rob J. Hyndman, "Optimal Forecast Reconciliation for Hierarchical and Grouped Time Series Through Trace Minimization," Journal of the American Statistical Association 114, no. 526 (2019): 804–819. Chapter 2 1. Linden, Smith and York (2003), as above. 2. Yehuda Koren, Robert Bell and Chris Volinsky, "Matrix Factorization Techniques for Recommender Systems," Computer 42, no. 8 (2009): 30–37. 3. Yifan Hu, Yehuda Koren and Chris Volinsky, "Collaborative Filtering for Implicit Feedback Datasets," Proceedings of the IEEE International Conference on Data Mining, 2008. 4. Paul Covington, Jay Adams and Emre Sargin, "Deep Neural Networks for YouTube Recommendations," Proceedings of the ACM Conference on Recommender Systems, 2016. 5. Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas and Domonkos Tikk, "Session-Based Recommendations with Recurrent Neural Networks," International Conference on Learning Representations, 2016. 6. Wang-Cheng Kang and Julian McAuley, "Self-Attentive Sequential Recommendation," IEEE International Conference on Data Mining, 2018. 7. Fei Sun and colleagues, "BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer," Proceedings of CIKM, 2019. 8. Jiaqi Zhai and colleagues, "Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations," Proceedings of the International Conference on Machine Learning, 2024. 9. Federal Trade Commission and others v. Amazon.com, Inc., complaint filed in the United States District Court for the Western District of Washington, September 2023. Chapter 3 1. Maurizio Ferrari Dacrema, Paolo Cremonesi and Dietmar Jannach, "Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches," Proceedings of the ACM Conference on Recommender Systems, 2019. 2. Ron Kohavi, Diane Tang and Ya Xu, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing (Cambridge University Press, 2020). 3. Alex Deng, Ya Xu, Ron Kohavi and Toby Walker, "Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data," Proceedings of WSDM, 2013. This is the source of the CUPED method. 4. Susan Athey, Raj Chetty, Guido Imbens and Hyunseung Kang, "The Surrogate Index: Combining Short-Term Proxies to Estimate Long-Term Treatment Effects More Rapidly and Precisely," NBER Working Paper 26463, 2019. 5. Amit Sharma, Jake M. Hofman and Duncan J. Watts, "Estimating the Causal Impact of Recommendation Systems from Observational Data," Proceedings of the ACM Conference on Economics and Computation, 2015. 6. Lihong Li, Wei Chu, John Langford and Robert E. Schapire, "A Contextual-Bandit Approach to Personalized News Article Recommendation," Proceedings of the World Wide Web Conference, 2010. 7. Lihong Li, Wei Chu, John Langford and Xuanhui Wang, "Unbiased Offline Evaluation of Contextual-Bandit-Based News Article Recommendation Algorithms," Proceedings of WSDM, 2011. Chapter 4 1. Stefano DellaVigna and Matthew Gentzkow, "Uniform Pricing in U.S. Retail Chains," Quarterly Journal of Economics 134, no. 4 (2019): 2011–2084. 2. Steven Berry, James Levinsohn and Ariel Pakes, "Automobile Prices in Market Equilibrium," Econometrica 63, no. 4 (1995): 841–890. 3. Victor Chernozhukov and colleagues, "Double/Debiased Machine Learning for Treatment and Structural Parameters," Econometrics Journal 21, no. 1 (2018): C1–C68. 4. Stefan Wager and Susan Athey, "Estimation and Inference of Heterogeneous Treatment Effects Using Random Forests," Journal of the American Statistical Association 113, no. 523 (2018): 1228–1242. Chapter 5 1. Kris Johnson Ferreira, Bin Hong Alex Lee and David Simchi-Levi, "Analytics for an Online Retailer: Demand Forecasting and Price Optimization," Manufacturing and Service Operations Management 18, no. 1 (2016): 69–88. 2. Kris Johnson Ferreira, David Simchi-Levi and He Wang, "Online Network Revenue Management Using Thompson Sampling," Operations Research 66, no. 6 (2018): 1586–1602. 3. On Wendy's, see contemporaneous reporting of the company's February 2024 earnings call remarks and its subsequent clarification; on Kroger, the letter from Senators Elizabeth Warren and Bob Casey to Kroger's chief executive, August 2024. 4. Le Chen, Alan Mislove and Christo Wilson, "An Empirical Analysis of Algorithmic Pricing on Amazon Marketplace," Proceedings of the World Wide Web Conference, 2016. 5. Michael Eisen, "Amazon's $23,698,655.93 Book About Flies," it is NOT junk (blog), April 2011. 6. Emilio Calvano, Giacomo Calzolari, Vincenzo Denicolò and Sergio Pastorello, "Artificial Intelligence, Algorithmic Pricing, and Collusion," American Economic Review 110, no. 10 (2020): 3267–3297. 7. Stephanie Assad, Robert Clark, Daniel Ershov and Lei Xu, "Algorithmic Pricing and Competition: Empirical Evidence from the German Retail Gasoline Market," Journal of Political Economy 132, no. 3 (2024): 723–771. 8. United States Department of Justice, "Former E-Commerce Executive Charged with Price Fixing in the Antitrust Division's First Online Marketplace Prosecution," press release, April 2015 (United States v. Topkins). 9. United States et al. v. RealPage, Inc., complaint filed in the United States District Court for the Middle District of North Carolina, August 2024. 10. Jennifer Valentino-DeVries, Jeremy Singer-Vine and Ashkan Soltani, "Websites Vary Prices, Deals Based on Users' Information," Wall Street Journal, December 24, 2012. 11. Federal Trade Commission, "Surveillance Pricing Study: Research Summaries" and accompanying staff update, January 2025. 12. New York Algorithmic Pricing Disclosure Act, enacted in the 2025 state budget, effective November 2025; see, for example, Jones Day, "New York's Novel Algorithmic Pricing Disclosure Law Takes Effect," November 2025. 13. Directive (EU) 2019/2161 of the European Parliament and of the Council (the Omnibus Directive), amending Directive 2011/83/EU on consumer rights. 14. Consumer Reports and Groundwork Collaborative, investigation of Instacart pricing, published December 2025. Chapter 6 1. M. Hadi Kiapour, Xufeng Han, Svetlana Lazebnik, Alexander C. Berg and Tamara L. Berg, "Where to Buy It: Matching Street Clothing Photos in Online Shops," Proceedings of the IEEE International Conference on Computer Vision, 2015. 2. Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang and Xiaoou Tang, "DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations," Proceedings of CVPR, 2016. 3. Yushi Jing and colleagues, "Visual Search at Pinterest," Proceedings of KDD, 2015; Andrew Zhai and colleagues, "Visual Discovery at Pinterest," Proceedings of the World Wide Web Conference Companion, 2017. 4. Yanhao Zhang and colleagues, "Visual Search at Alibaba," Proceedings of KDD, 2018. 5. Fan Yang and colleagues, "Visual Search at eBay," Proceedings of KDD, 2017. 6. Jeff Johnson, Matthijs Douze and Hervé Jégou, "Billion-Scale Similarity Search with GPUs," IEEE Transactions on Big Data 7, no. 3 (2021): 535–547. 7. Yu A. Malkov and D. A. Yashunin, "Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs," IEEE Transactions on Pattern Analysis and Machine Intelligence 42, no. 4 (2020): 824–836. 8. Alec Radford and colleagues, "Learning Transferable Visual Models from Natural Language Supervision," Proceedings of the International Conference on Machine Learning, 2021. Chapter 7 1. de Koster, Le-Duc and Roodbergen (2007), as above. 2. H. Donald Ratliff and Arnon S. Rosenthal, "Order-Picking in a Rectangular Warehouse: A Solvable Case of the Traveling Salesman Problem," Operations Research 31, no. 3 (1983): 507–521. 3. Randolph W. Hall, "Distance Approximations for Routing Manual Pickers in a Warehouse," IIE Transactions 25, no. 4 (1993): 76–87. 4. Kees Jan Roodbergen and René de Koster, "Routing Methods for Warehouses with Multiple Cross Aisles," International Journal of Production Research 39, no. 9 (2001): 1865–1883. 5. Peter R. Wurman, Raffaello D'Andrea and Mick Mountz, "Coordinating Hundreds of Cooperative, Autonomous Vehicles in Warehouses," AI Magazine 29, no. 1 (2008): 9–20. 6. Guni Sharon, Roni Stern, Ariel Felner and Nathan R. Sturtevant, "Conflict-Based Search for Optimal Multi-Agent Pathfinding," Artificial Intelligence 219 (2015): 40–66. 7. Amazon, announcement of its one millionth robot and the DeepFleet model, June 30 to July 1, 2025. Chapter 8 1. Oriol Vinyals, Meire Fortunato and Navdeep Jaitly, "Pointer Networks," Advances in Neural Information Processing Systems, 2015. 2. Irwan Bello, Hieu Pham, Quoc V. Le, Mohammad Norouzi and Samy Bengio, "Neural Combinatorial Optimization with Reinforcement Learning," arXiv preprint, 2016. 3. Wouter Kool, Herke van Hoof and Max Welling, "Attention, Learn to Solve Routing Problems!," International Conference on Learning Representations, 2019. 4. Guillaume Sartoretti and colleagues, "PRIMAL: Pathfinding via Reinforcement and Imitation Multi-Agent Learning," IEEE Robotics and Automation Letters 4, no. 3 (2019): 2378–2385. 5. Amazon DeepFleet announcement, as for Chapter 7. Further Reading Ron Kohavi, Diane Tang and Ya Xu, Trustworthy Online Controlled Experiments (Cambridge University Press, 2020).** The standard practical guide to running online experiments, written by people who ran them at scale. Essential for anyone measuring what personalization or pricing changes. Kim Falk, Practical Recommender Systems (Manning, 2019).** An accessible, engineering-oriented introduction to building recommenders, with attention to implicit feedback and evaluation. Charu C. Aggarwal, Recommender Systems: The Textbook (Springer, 2016).** A thorough academic treatment of recommendation methods, from neighborhood models to matrix factorization and beyond. Victor Chernozhukov and colleagues, "Double/Debiased Machine Learning for Treatment and Structural Parameters," Econometrics Journal (2018).** The foundational paper on combining machine learning with causal estimation. Technical, but its introduction is readable and explains why prediction models alone cannot estimate effects. Joshua D. Angrist and Jörn-Steffen Pischke, Mastering 'Metrics (Princeton University Press, 2015).** A lively introduction to experiments, instrumental variables and natural experiments, the tools behind credible elasticity estimation. Robert L. Phillips, Pricing and Revenue Optimization, second edition (Stanford University Press, 2021). The leading textbook on pricing analytics, covering price response, markdowns, customized pricing and the optimization layer. Ariel Ezrachi and Maurice E. Stucke, Virtual Competition: The Promise and Perils of the Algorithm-Driven Economy (Harvard University Press, 2016).** An influential account of how pricing algorithms may enable collusion and discrimination, written by two competition law scholars. René de Koster, Tho Le-Duc and Kees Jan Roodbergen, "Design and Control of Warehouse Order Picking: A Literature Review," European Journal of Operational Research (2007).** The standard survey of order-picking research, covering layout, storage, batching and routing. Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, second edition (MIT Press, 2018). The classic textbook on reinforcement learning, freely available online from the authors. Federal Trade Commission, Surveillance Pricing 6(b) study materials (January 2025). The research summaries and staff update describing how intermediaries help retailers use consumer data in pricing, available on the FTC website. Hashtags: #AIInRetail #AIInEcommerce #RetailArtificialIntelligence #EcommerceAI #RetailTechnology #RetailInnovation #SmartRetail #DigitalCommerce #OnlineRetail #MachineLearningInRetail #RetailDataScience #RetailAnalytics #PredictiveAnalytics #RetailPersonalization #PersonalizedShopping #ProductRecommendations #RecommendationSystems #RecommenderSystems #CollaborativeFiltering #MatrixFactorization #TwoTowerModels #SequentialRecommendation #TransformerRecommenders #ProductRanking #SearchPersonalization #SemanticProductSearch #CustomerBehaviorAnalytics #CustomerPreferences #CustomerEngagement #CustomerExperience #CustomerRetention #ConversionOptimization #PurchasePrediction #DemandForecasting #RetailDemandForecasting #ProbabilisticForecasting #InventoryForecasting #StockoutPrediction #DemandUncensoring #InventoryOptimization #RetailPricing #DynamicPricing #AlgorithmicPricing #PriceOptimization #PriceElasticity #PriceElasticityModeling #PricingStrategy #MarkdownOptimization #PromotionOptimization #RevenueOptimization #ProfitOptimization #CausalInference #CausalMachineLearning #DoubleMachineLearning #CausalForests #UpliftModeling #IncrementalityMeasurement #ABTesting #RetailExperimentation #ContextualBandits #ThompsonSampling #ExplorationAndExploitation #OffPolicyEvaluation #InversePropensityScoring #RetailFeedbackLoops #ExposureBias #PopularityBias #ProductDiscovery #VisualSearch #VisualProductSearch #ComputerVisionInRetail #ImageEmbeddings #MultimodalSearch #WarehouseLogistics #WarehouseOptimization #OrderPicking #PickingRouteOptimization #OrderBatching #WarehouseAutomation #AutonomousMobileRobots #MultiAgentSystems #ReinforcementLearning #WarehouseRobotics #LogisticsOptimization #SupplyChainEfficiency #OperationalEfficiency #RetailDecisionIntelligence #RetailAIGovernance #AlgorithmicFairness #SurveillancePricing #RetailDigitalTransformation
- AI and Cognitive Biases (How Human Flaws Shape Machine Logic)
Download the Book (PDF): Introduction Somewhere in the history of almost every machine learning system there is a person looking at a screen and making a decision. The decision is small. Is this tweet offensive? Does this X-ray show a nodule? Is this product review positive or negative? Which of these two chatbot answers is better? Is the person in this photograph a nurse or a doctor? The person makes the call, clicks a button, and moves on to the next item. A few seconds later they make another call, and another, several hundred times in a shift. Their clicks are collected, aggregated, cleaned, and stored. Eventually they become a column in a table called something like label, and from that moment on they are treated as fact. This booklet is about what happens to the human mind on its way into that column. The public conversation about biased AI has, for good reasons, concentrated on representation: whether a dataset contains enough faces of darker-skinned women, enough speakers of a regional dialect, enough patients from rural hospitals. Representation matters, and later chapters return to it. But it is only half the story, and in many systems it is the smaller half. A dataset can be perfectly balanced in who appears in it and still encode a systematic distortion in what was said about them, because the labels were produced by human beings whose judgments follow predictable patterns of error. Psychologists have spent half a century cataloguing those patterns. They have names: confirmation bias, anchoring, availability, survivorship bias, the halo effect, framing. They are not exotic failures of careless people. They are the normal operating characteristics of human cognition, present in experts and novices, in the diligent and the distracted, in the annotator in Nairobi and the radiologist in Boston. The argument of this booklet is simple to state and has uncomfortable consequences. Machine learning systems do not merely inherit the skew of the populations they sample; they inherit the shape of the judgments that produced their labels. Cognitive biases in the people who annotate, curate, and select training data become regularities in that data, and a learning algorithm cannot tell a regularity of the world from a regularity of the labeller's mind. It learns both, with equal confidence, and then applies them at a scale and speed no individual human could. The practical upshot is that fixing biased AI cannot be done only by fixing datasets after the fact. It requires redesigning the human judgment processes that create data in the first place. Why judgment is the right unit of analysis The core idea needs one clarification before it can do any work. When people hear "biased data", they tend to imagine a sampling problem: the training set contains too few examples of some group, so the model performs worse on that group. That is a real phenomenon, and it is well understood. The phenomenon this booklet concentrates on is different in kind. Here the problem is not who is in the data but what the data says about them, and what it says is the output of a human cognitive process. Consider a concrete case that has become a touchstone in the research literature. In 2019, Maarten Sap and colleagues at the University of Washington examined several widely used datasets for detecting hate speech and abusive language on Twitter. They found that tweets written in African American English were substantially more likely to have been labelled offensive by annotators than tweets in other dialects, and that classifiers trained on these datasets reproduced the pattern, flagging innocuous tweets by Black users as toxic at elevated rates. The datasets did not lack tweets from Black users. What they contained was a systematic tendency in the people doing the labelling to read certain dialect features, including in-group uses of words that are slurs in other contexts, as signals of hostility. That tendency is a cognitive phenomenon: a pattern of inference that runs ahead of the evidence, interpreting ambiguous input in the direction of a prior expectation. The model learned it as if it were linguistics. The same study contained a hopeful finding. When annotators were shown the likely dialect of a tweet's author and reminded that language norms differ between communities, they became significantly less likely to label tweets in African American English as offensive. A small change to the judgment environment shifted the data. That finding, too, is the thesis of this booklet in miniature: the biases are in the process, and the process can be redesigned. What the book covers and what it leaves out The booklet proceeds in a deliberate order. The first chapter establishes how human judgment becomes data at all: the annotation pipeline, the idea of a "label" as a compressed record of a decision, and the difference between random noise and systematic bias. Without that foundation, the rest of the book would be a list of anecdotes. The next three chapters each take one of the three biases named in the subtitle's promise and follow it from the psychology laboratory into the training set. Chapter 2 deals with confirmation bias: the tendency to seek, notice, and interpret evidence in ways that fit what we already believe, and the many ways annotation guidelines, task framing, and annotators' own beliefs channel it into labels. Chapter 3 deals with anchoring: the outsized influence of the first number or suggestion a person sees, which in modern pipelines is very often a suggestion produced by a previous model. Chapter 4 deals with survivorship bias, which operates less on individual labels than on the selection of what gets labelled at all: datasets built only from the cases that made it through some earlier filter, and models that consequently learn a world in which the failures have been quietly removed. Chapter 5 widens the lens to the supporting cast of biases that matter in practice: availability, representativeness, the halo effect, framing, fatigue and order effects, and the in-group perspective of whoever happens to be doing the annotation. Chapter 6 then explains, mechanically, how biased labels become biased decisions: why learning algorithms are so good at picking up the regularities of a labeller's mind, how aggregation hides disagreement, and how models can amplify the skew they were given. Chapter 7 closes the loop, describing what happens when model outputs flow back into human judgment and future training data, in settings from predictive policing to the human feedback used to train large language models. Chapter 8 turns to practice: what organisations that build and buy machine learning systems can actually do, at the level of task design, team composition, documentation, measurement, and governance. Several adjacent topics are deliberately left to one side. The booklet does not attempt a general treatment of algorithmic fairness metrics, which is a large technical field with its own literature; it draws on that field where it illuminates the cognitive story and otherwise leaves it alone. It does not survey the biases that large language models themselves appear to display, such as their susceptibility to anchoring in their own outputs, except where those behaviours feed back into human data. And it does not argue that machines should replace human judgment. On the contrary, its position is that human judgment is irreplaceable in building these systems, and precisely for that reason deserves the kind of careful engineering we give to the algorithms. A word on evidence The psychology of judgment has had a turbulent decade. The replication crisis in social psychology overturned a number of famous findings, and readers are right to ask whether the biases discussed here are robust. The ones this booklet relies on are among the best established in the field. Anchoring has been replicated many times, including in large multi-laboratory projects. Confirmation bias, in its various forms, has been documented across hundreds of studies and summarised in authoritative reviews. Survivorship bias is less a psychological effect than a statistical one, a property of selection rather than of perception, and its consequences can be demonstrated with arithmetic. Where the evidence for a specific claim is thinner, the text says so. Equally, the evidence that these biases enter machine learning data comes from a growing and specific body of research: studies of annotator disagreement and annotator identity, audits of benchmark datasets, controlled experiments on crowdworkers, and post-mortems of deployed systems that went wrong. The booklet cites that work by author and year where it is used, and the Further Reading section at the end points to the most useful sources for readers who want to go further. The intended reader is anyone who builds, buys, regulates, or is affected by systems trained on human-labelled data, which by now includes nearly everyone. No mathematics is required. What is required is a willingness to treat the labels in a dataset not as ground truth handed down from nature but as the residue of thousands of small human decisions, each one made by a mind with the ordinary, well-documented habits of all minds. Once you see training data that way, a great deal about the behaviour of AI systems stops being mysterious. Chapter 1: How Judgment Becomes Data A supervised machine learning model learns a function from inputs to outputs by studying examples where both are known. The inputs are usually the easy part. Photographs, sentences, medical scans, transaction records and sensor readings exist in the world and can be collected. The outputs are a different matter. Nature does not stamp the word "cat" on a photograph of a cat, or attach a severity grade to a retinal scan, or tag a loan application with the probability that it will default. Someone, at some point, has to decide what the right answer is for each example. That decision is the label, and the whole edifice of supervised learning rests on it. It is worth slowing down on this because the vocabulary of the field hides it. Practitioners speak of "ground truth" as if labels were measurements of a physical quantity. Benchmarks report accuracy to the second decimal place, as though the target being hit were fixed and exact. Dataset files store labels in the same plain columns as the inputs, with nothing to distinguish an observed fact from a human opinion. Over time, the provenance of the label disappears from view. This chapter puts it back. Four ways a label is made Labels come from several kinds of process, and each lets human judgment in by a different door. The first and most visible is explicit annotation: a person is shown an input and asked to assign it a category, a score, a bounding box, a transcription or a ranking. This is the work done by crowdworkers on platforms such as Amazon Mechanical Turk and Prolific, by contracted annotation vendors employing thousands of people across Kenya, the Philippines, India and Venezuela, by in-house specialist teams, and by domain experts such as radiologists or lawyers when the task demands it. Mary Gray and Siddharth Suri's book Ghost Work (2019) documented how large and largely invisible this workforce is. Every label they produce is a judgment made under particular conditions of time, payment, instruction and attention. The second is behavioural labelling, where the label is inferred from something people did rather than something they were asked to say. A click on a search result becomes a signal of relevance. A purchase becomes a signal of preference. A "report" button becomes a signal of abuse. A doctor's billing code becomes a signal of diagnosis. These labels feel more objective because nobody was asked to give an opinion, but each is still the output of a human decision, taken for other reasons and in other contexts, and repurposed as a training target. The third is outcome labelling, where the label is a later event: whether a borrower repaid, whether a defendant was rearrested, whether a patient was readmitted, whether a new hire was promoted. Outcomes look like the purest form of ground truth, because they happened in the world. But which outcomes are recorded, and for whom, is determined by earlier human choices. A loan that was never granted produces no repayment record. A crime that was never policed produces no arrest. The outcome is real, but the set of cases on which outcomes are observed has already been shaped by judgment, which is the subject of Chapter 4. The fourth is curation: the decisions about which examples enter the dataset at all. Someone chose which websites to scrape, which filters to apply, which hospital's records to use, which images to keep after a quality check. Curation does not attach a label to each example, but it labels the dataset as a whole as "representative of the task", and that implicit label is often the most consequential of all. In real systems these four processes are layered. A modern large language model, for example, is first pretrained on text selected by curation filters, some of which are themselves classifiers trained on human-labelled examples of "high quality" writing. It is then fine-tuned on demonstrations written by contractors, and further adjusted using preference rankings in which human raters choose which of two responses is better. At every layer, a human judgment is converted into a training signal. A label is a compressed decision The most useful single idea in this booklet is that a label is not a fact about an input. It is a compressed record of a decision that a particular person made about that input, under particular conditions. Everything that went into the decision, the person's knowledge, expectations, attention, mood, cultural frame, the instructions they were given, the examples they had just seen, the suggestion on screen, the time pressure, is squeezed into a single value. The compression discards the reasoning but preserves its outcome, including whatever systematic tilts the reasoning had. This matters because learning algorithms are, at bottom, devices for finding regularities that connect inputs to labels. If the labels contain a regularity that originated in the labeller rather than in the world, the algorithm will find it, provided it can be predicted from the input. It has no way of asking where the regularity came from. A model trained to predict which tweets annotators labelled offensive will learn what makes annotators say "offensive", which overlaps with, but is not identical to, what makes a tweet offensive. Mor Geva, Yoav Goldberg and Jonathan Berant put the point precisely in the title of a 2019 paper: "Are We Modeling the Task or the Annotator?" They showed that in several natural language datasets built by crowdworkers, models could learn to recognise which individual worker had written an example, and that performance dropped when models were tested on examples from workers they had not seen in training. The models had learned, in part, the habits of particular people. Noise and bias are different problems Every discussion of label quality eventually confronts the distinction between noise and bias, and it is central to the argument here. Noise is variability that does not point anywhere in particular. If ten annotators label the same borderline image and split six to four, or if the same annotator labels an item differently on Monday and Friday, the disagreement is noise in the sense that Daniel Kahneman, Olivier Sibony and Cass Sunstein use the word in their 2021 book Noise: unwanted scatter in judgments that should be identical. Random noise is a nuisance for machine learning, but a tolerable one. With enough data, independent errors in different directions tend to cancel. A great deal of statistical machinery, going back to Alexander Dawid and Allan Skene's 1979 method for estimating the reliability of individual observers, exists to model and remove it. Bias is different. A bias is a systematic tilt: errors that point in a consistent direction, correlated with some feature of the input or the situation. If annotators consistently rate speech in one dialect as more aggressive, consistently judge confident-sounding answers as more correct, or consistently overlook small lesions in the corner of a scan, the errors do not cancel with more data. They accumulate. More data simply makes the model more certain of the tilt. This is why the cognitive biases studied by psychologists are so important for machine learning: they are, by definition, the systematic kind of error. A heuristic that produces predictable distortions under predictable conditions is exactly what an algorithm can learn. The difference between the two is easy to blur in practice, because both show up as "label errors" when measured against a careful re-annotation. Curtis Northcutt, Anish Athalye and Jonas Mueller estimated in 2021 that the test sets of ten widely used benchmarks, including ImageNet, CIFAR-10 and QuickDraw, contained an average of about 3.3 percent mislabelled examples, with ImageNet's validation set around 6 percent. Some of those errors were random slips. Others were systematic confusions between similar categories that annotators found genuinely hard to distinguish, and a model trained on the training sets, which share the same annotation process, would learn the same confusions. Table 1 sets out the practical differences between the two kinds of error. Table 1. Random label noise compared with systematic label bias. Property Random noise Systematic bias Direction of errors Scattered Consistent Effect of more data Tends to cancel Accumulates Detected by annotator agreement Often Often not Typical source Slips, fatigue, ambiguity Heuristics, expectations, framing What the model learns Some uncertainty A false regularity The third row deserves emphasis. The standard industry check on label quality is inter-annotator agreement: if several people independently assign the same label, the label is trusted. Agreement is a good detector of noise. It is a poor detector of bias, because a bias shared across annotators produces agreement. If everyone in the annotation pool reads the same dialect feature as hostile, they will agree with one another, the agreement statistics will look excellent, and the label will be confidently wrong. High agreement can certify a shared error. That observation recurs throughout this booklet. The myth of the single right answer Behind the language of ground truth lies an assumption that every input has one correct label, which careful enough annotation would find. For some tasks this is close to true: a handwritten digit is usually a particular digit, and a photograph of a fire truck usually shows a fire truck. For many of the tasks on which modern AI systems are trained, it is not. Whether a comment is offensive, whether an essay is persuasive, whether a chatbot response is helpful, whether a news article is biased, whether an X-ray finding is clinically significant: these are judgments on which reasonable, informed people disagree, often for principled reasons. Lora Aroyo and Chris Welty made this argument forcefully in a 2015 paper titled "Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation". They argued that disagreement between annotators is frequently signal rather than noise: it marks items that are genuinely ambiguous, categories that are poorly defined, or perspectives that differ. Treating such disagreement as error and resolving it by majority vote throws away information. Barbara Plank's 2022 survey, "The 'Problem' of Human Label Variation", extended the case and showed how widespread genuine variation is across language tasks. This reframing changes how cognitive bias should be understood. If there is no single right answer, then a "biased" annotator is not simply one who deviates from the truth. Bias is better defined as a systematic influence on judgment that the task designer would not endorse on reflection: an influence of the order in which items were shown, the suggestion on screen, the framing of the question, or an expectation the annotator brought to the task that has nothing to do with the property being labelled. Some of what looks like bias is legitimate difference of perspective; some of what looks like consensus is shared bias. Telling the two apart is one of the hardest problems in the field, and Chapter 5 returns to it when discussing the identity and perspective of annotators. The conditions under which labels are made Psychologists know that cognitive biases are strongest under particular conditions: time pressure, ambiguity, unfamiliarity, fatigue, high volume of repetitive decisions, low personal stakes in accuracy, and the availability of an easy cue that substitutes for careful analysis. It is difficult to design a set of working conditions that would maximise all of these more effectively than a typical commercial annotation task. Crowdworkers are typically paid per item, which rewards speed. Items are often ambiguous by construction, because easy items are cheap to label automatically and the hard residue is what gets sent to humans. Annotators may label thousands of items in a sequence with little feedback on accuracy. Many work in a second language or on content from a culture other than their own. Instructions are compressed into guidelines that must be read quickly. In 2023, Time magazine reported that workers at an outsourcing firm in Kenya, labelling toxic content to train a safety filter for a major AI developer, were paid less than two dollars an hour and exposed to a relentless stream of disturbing material. Under such conditions, the human mind does what it always does when resources are scarce: it relies on fast, automatic heuristics. Those heuristics are the source of most cognitive biases. Experts are not immune. A radiologist labelling scans for a research dataset is subject to the same limits of attention, the same tendency to stop searching once one abnormality is found, and the same susceptibility to prior information as in clinical practice. In a well-known 2013 study, Trafton Drew, Melissa Võ and Jeremy Wolfe inserted the image of a gorilla, many times the size of a typical nodule, into lung CT scans and asked radiologists to search for nodules. Eighty-three percent of them did not report the gorilla, even though eye-tracking showed that many had looked directly at it. Expertise sharpens attention to what one is looking for. It does not eliminate the effects of expectation on what one sees. A worked example: one item, many hands It helps to follow a single item through a typical pipeline. Imagine a short social media reply: a user responds to a friend's post about a football result with a line of affectionate trash talk, written in a regional slang and including a word that is an insult in standard usage but a term of endearment among the user's peers. The item first has to be selected for labelling. It might be chosen because a keyword filter matched the insulting word, because another user reported it, or because an existing classifier scored it above some threshold. Each of these routes is a selection process, and each favours items that share features with what earlier systems or users had already treated as problematic. The item reaches the annotation queue already carrying the implicit message that it is worth suspecting. An annotator then sees it, perhaps alongside the classifier's score, perhaps in a batch of other flagged items, most of which really are abusive. The annotator does not speak the regional slang. The guideline's examples of "insults" include the standard usage of the word in question. The annotator has a few seconds. They mark the item as abusive. Two other annotators, recruited from the same vendor and working from the same guideline, do the same. Agreement is perfect. The aggregated label enters the training set. A model learns, from this and many similar items, that the word in question and some of the slang that surrounds it predict abuse. Deployed, the model flags and removes similar posts written by people who speak that way, most of whom are joking with their friends. Some of the removed posts are appealed; a few appeals are reviewed by moderators who see the model's high confidence score first. Most removals stand. The removed posts, now labelled as violations by the system's own decisions, may enter the next training set. At no point in this sequence did anyone intend to penalise a regional way of speaking. Every step was locally reasonable. Yet each step added a systematic push in the same direction: a selection process that favoured suspicious-looking items, a batch context that raised the expectation of abuse, a guideline whose examples made one reading available and not the other, an annotator pool that shared the same unfamiliarity, an aggregation that recorded unanimous agreement, a model that learned the surface cue, and a review process anchored on the model. Each of those pushes is the subject of a later chapter. Why the transfer is so efficient It would be comforting if the passage from biased human judgment to biased model were lossy, so that the model inherited only a faint echo of its labellers' tendencies. Often the opposite is true, for three reasons that later chapters develop. The first is that models detect patterns humans do not consciously know they are producing. An annotator who labels slightly more harshly after a run of severe items, or who gives slightly higher ratings to longer responses, is unlikely to be aware of it. The pattern lives in thousands of decisions, each individually defensible. But it is a statistical regularity, and statistical regularities are what models are built to find. The second is that aggregation, the standard method of turning multiple annotations into one label, removes the visible evidence of disagreement while preserving the shared tilt. A majority vote among annotators who share a bias returns the biased label, now stripped of any record that some annotators saw it differently. The third is that the model's output is then used at scale, with the authority of a computational system. One annotator's tendency to read a dialect as hostile affects the few hundred items they label. The model trained on their labels applies that tendency to every post on a platform, every day, indefinitely. And as Chapter 7 shows, its outputs frequently become the inputs to the next round of human judgment, completing a loop. The rest of this booklet follows specific biases through this pipeline. The first and most pervasive is the tendency to see what we expect to see. Chapter 2: Seeing What We Expect: Confirmation Bias in the Label In 1960 the British psychologist Peter Wason published an experiment that has been taught to every student of reasoning since. He told participants that the sequence 2, 4, 6 conformed to a rule he had in mind, and asked them to discover the rule by proposing other sequences of three numbers. He would say whether each proposed sequence fitted. Most participants quickly formed a hypothesis, such as "even numbers ascending by two", and then proposed sequences that fitted it: 8, 10, 12; 20, 22, 24; 100, 102, 104. Each was confirmed. Confident, they announced their rule, and were told it was wrong. The actual rule was simply "any three ascending numbers". Very few participants had tried a sequence that would have falsified their hypothesis, such as 1, 2, 3 or 6, 4, 2. They tested their idea by looking for agreement with it. The family of tendencies this experiment exposed is now called confirmation bias. Raymond Nickerson's 1998 review, still the standard reference, described it as "the seeking or interpreting of evidence in ways that are partial to existing beliefs, expectations, or a hypothesis in hand." The definition has two parts, and both matter for machine learning. People search selectively, looking for information that would confirm what they already suspect. And they interpret selectively, reading ambiguous information as support for their existing view. Annotation tasks give annotators few opportunities to search, because the item is placed in front of them. They give them endless opportunities to interpret. The psychology of biased interpretation The interpretive side of confirmation bias was demonstrated memorably by Charles Lord, Lee Ross and Mark Lepper in 1979. They gave supporters and opponents of capital punishment the same two fictitious studies, one apparently showing that the death penalty deterred crime and one apparently showing that it did not. Each group rated the study that supported its own view as more convincing and better conducted, found flaws in the other, and ended up more confident in its original position than before. Identical evidence moved the two groups further apart. The researchers called the phenomenon biased assimilation. What makes this relevant to annotation is that much of the material annotators are asked to judge is exactly as ambiguous as those studies were. A social media post may be sarcastic or sincere. A product review may be faint praise or quiet complaint. A political statement may be neutral reporting or loaded framing. A medical image may show an early abnormality or a harmless variation. When evidence does not settle the question, the judgment is completed by what the judge expected to find. And what annotators expect is shaped by three sources: their own prior beliefs about the world, the framing of the task they are given, and the information that surrounds the item. Forensic science offers the most rigorous evidence that experts are not exempt. Itiel Dror and colleagues, in a 2006 study, presented five experienced fingerprint examiners with pairs of prints that each examiner had, years earlier and in the ordinary course of casework, declared to be a match. This time the prints were accompanied by contextual information suggesting that they did not match, implying that the case had been a notorious misidentification. Four of the five examiners reached a different conclusion from their own earlier judgment. The marks on the page had not changed; the expectation had. The study was small, but its finding has been reproduced in various forms across forensic disciplines, and it led to the practice known as linear sequential unmasking, in which examiners analyse the evidence before being exposed to potentially biasing context. That practice has direct analogues in annotation design, discussed in Chapter 8. Annotators bring their beliefs with them The most direct route by which confirmation bias enters training data is through annotators' own beliefs about the subject they are labelling. In 2022, Maarten Sap and colleagues published "Annotators with Attitudes: How Annotator Beliefs and Identities Bias Toxic Language Detection". They recruited annotators, measured their attitudes using established survey instruments, and asked them to rate the toxicity of posts that included anti-Black statements, posts in African American English, and posts containing vulgar but not hateful language. The patterns were stark. Annotators who scored higher on a measure of racist beliefs were less likely to rate anti-Black content as toxic and more likely to rate posts in African American English as toxic. Annotators who were more conservative were more likely to judge African American English as offensive and less likely to see anti-Black language as offensive. Annotators who placed a high value on free expression rated less content of all kinds as toxic. The authors then examined a widely used toxicity detection system and found that its scores aligned more closely with the judgments of some belief groups than others. The point is not that some annotators are biased and others are neutral. Every annotator brings a frame, and the frame decides how ambiguous content is resolved. A person who expects hostility from a group reads hostility into that group's ambiguous speech. A person who does not expect hostility from a group reads its ambiguous speech charitably. Each experiences their own judgment simply as perception: they look at the text and see that it is offensive, or not. The belief does its work before conscious deliberation begins, which is why annotators asked to explain their labels usually explain them in terms of the text, not themselves. Christoph Hube, Besnik Fetahu and Ujwal Gadiraju documented the same mechanism in a different task. In a 2019 study presented at the CHI conference, crowdworkers were asked to judge whether statements on contested political topics were neutral or opinionated. Workers' own opinions on the topics systematically influenced their judgments: statements consistent with their views were more likely to be seen as neutral, statements opposed to them as biased. This is the hostile media effect, first documented by Robert Vallone, Lee Ross and Mark Lepper in 1985 with news coverage of the Middle East, operating at the level of individual labels. A model trained to detect "biased language" from such data learns, in part, the political position of the annotator pool. Hube and colleagues also tested remedies. Asking workers to think about how other people would judge a statement, a technique called social projection, and raising awareness of their own possible bias both reduced the effect. Neither eliminated it. The finding fits a broad pattern in the debiasing literature: prompts that encourage people to consider alternative perspectives help, modestly and reliably, but cannot be relied on as a complete solution. The task frame becomes the hypothesis Annotators do not only bring expectations; they are given them. Every annotation task embeds a hypothesis about what the annotator is likely to find, and that hypothesis acts on judgment the way Wason's initial rule acted on his participants. Consider the simple act of naming a task. An annotator asked to "find the hate speech in these posts" is in a different cognitive position from one asked to "classify each post as hateful or not hateful", even if the interface is identical. The first framing implies that hate speech is present to be found; the second implies a neutral classification. Psychologists call the general phenomenon a demand characteristic: participants infer what is expected of them from cues in the task, and adjust their behaviour. In annotation, the expectation that a target category is common makes ambiguous items more likely to be assigned to it. Prevalence in the batch has the same effect. Annotators who see many examples of a category come to expect it, and the threshold for calling an ambiguous item a member of the category drops. The perceptual psychologist David Levari and colleagues demonstrated a striking version of this in a 2018 paper in Science, "Prevalence-induced concept change in human judgment". Participants judged whether dots were blue or purple, whether faces were threatening, or whether research proposals were ethical. When the prevalence of the target category was reduced over time, participants expanded their definition to include items they had previously excluded: fewer blue dots led to purple dots being called blue, fewer threatening faces led to neutral faces being called threatening. The concept bent to keep the rate of positive judgments roughly constant. For annotation, the implication is uncomfortable. If a team filters out obvious cases before sending items to human labellers, as is common, the labellers may expand their definition of the category to compensate, producing more positive labels on borderline items than the guidelines intended. Guidelines themselves transmit hypotheses. Annotation guidelines almost always include worked examples, and those examples act as prototypes. If the examples of "toxic" language in the guideline are dominated by one kind of content, annotators will look for that kind and may miss others; if they contain features that are incidental to the concept, such as particular slang, those features become cues. A guideline written by a team with a particular view of what the category means turns that view into the hypothesis every annotator tests. How annotators write their own confirmations Some datasets are built not by labelling existing material but by asking people to create examples. Crowdworkers write questions, paraphrases, contradicting sentences or adversarial prompts to order. Here confirmation bias works in a subtler way: annotators produce examples that confirm their own mental model of the category, and those examples carry the traces of that model. The clearest demonstration came from natural language inference, a task in which a model must say whether one sentence (the hypothesis) follows from, contradicts, or is neutral with respect to another (the premise). The large datasets for this task, SNLI and MultiNLI, were built by giving crowdworkers a premise and asking them to write one hypothesis of each type. In 2018, two groups, Suchin Gururangan and colleagues and Adam Poliak and colleagues, independently showed that a model could predict the correct label much of the time by looking at the hypothesis alone, without the premise. Gururangan's team reported that a simple classifier using only the hypothesis reached about 67 percent accuracy on SNLI, against a chance rate of 33 percent. The reason was that crowdworkers, asked to write a contradiction, reached for the most obvious strategy available, which was to insert a negation: words such as "not", "no", "nobody" and "never" were strongly associated with the contradiction label. Asked to write an entailment, they tended to generalise, replacing specific words with broader ones. Asked for a neutral statement, they added purposes or extra detail. Each worker was doing what was asked, using a sensible mental model of what contradiction means. But the mental models were shared, and they were narrower than the concept. The dataset ended up confirming the annotators' prototype of contradiction, and models learned the prototype, achieving high benchmark scores while failing on contradictions that did not fit it. This is confirmation bias at the level of a population: a collective hypothesis about what a category looks like, tested only on cases its holders generated, and therefore never falsified. Context, metadata and the leaking expectation A third route runs through the information that surrounds the item being labelled. In many annotation interfaces, items arrive with context: the username of the author, the source of the image, the patient's age and referral reason, the category the item was retrieved under, or a confidence score from an earlier system. Some of this context is necessary; much of it is not, and any of it can steer interpretation. The forensic studies discussed earlier are a warning. When examiners knew a suspect had confessed, they were more likely to see a match. In annotation, knowing that an image came from a medical department that specialises in a particular disease may make a labeller more likely to find signs of it. Knowing that a social media post came from an account previously flagged may make a moderator more likely to label it abusive. The bias is self-reinforcing: the account was flagged because of earlier labels, the earlier labels make the new label more likely, and the new label strengthens the case for the flag. Clinical labelling provides many examples. Radiology reports, which are increasingly mined to produce labels for image datasets, are written by doctors who know the patient's history and the reason for the scan. The same image may be reported differently depending on the clinical question asked. When the National Institutes of Health released the ChestX-ray14 dataset in 2017, with labels extracted automatically from radiology reports, the radiologist Luke Oakden-Rayner examined samples of the images and argued that a substantial fraction of labels for several conditions did not match what the images showed. Part of the explanation was that reports reflect clinical context and the radiologist's purpose, not only visual findings. A model trained on such labels learns to predict what a radiologist with that context would write, which is not the same as what the image contains. Why aggregation does not save us The usual defence against individual annotator error is redundancy: have each item labelled by three, five or more people, and take the majority. This works well against independent mistakes. It works poorly against confirmation bias, for a reason already noted in Chapter 1: the bias is often shared. Annotators recruited from the same platform, the same country, the same vendor or the same demographic tend to hold overlapping beliefs and expectations. They read the same guidelines with the same worked examples. They see the same interface with the same framing and the same context. If the tilt is common to most of them, majority voting reinforces it. Worse, it erases the minority who saw the item differently, who may be the very annotators whose perspective was needed. Sap's 2022 study found substantial variation between belief groups in their toxicity judgments; whichever group dominated the annotator pool would dominate the aggregated label. Aggregation also feeds a statistical illusion. Teams report agreement rates as evidence of quality, and agreement is high when annotators share a hypothesis. The Wason participants would have agreed with each other about the rule. They were all wrong in the same direction. What follows for the model A model trained on labels shaped by confirmation bias learns two things at once. It learns the property the task designers intended, to whatever extent annotators captured it. And it learns the correlates of the annotators' expectations: dialect features that annotators associated with hostility, negation words that crowdworkers associated with contradiction, contextual signals that steered interpretation. The second set of lessons is usually easier to learn than the first, because surface cues are simpler than meaning. That is why models trained on such data often achieve impressive benchmark scores that collapse on carefully constructed test sets, and why their errors fall disproportionately on the groups and styles that annotators read uncharitably. The difficulty is that from inside the dataset, the two kinds of lesson look the same. Both are regularities connecting inputs to labels. Distinguishing them requires information the dataset does not contain: who the annotators were, what they were told, what they saw, and how their judgments varied. Collecting that information, and designing tasks that put expectation in its place, is the subject of the final chapter. First, though, there is a bias that operates even more mechanically than confirmation, because it does not need any belief at all. It needs only a number on the screen. Chapter 3: The First Number Wins: Anchoring and the Pre-Labelled World In one of the experiments reported in Amos Tversky and Daniel Kahneman's 1974 paper "Judgment under Uncertainty: Heuristics and Biases", participants watched a wheel of fortune spin and stop on a number. Unknown to them, the wheel had been rigged to stop on either 10 or 65. They were then asked whether the percentage of African countries in the United Nations was higher or lower than that number, and then to estimate the actual percentage. Participants who had seen 10 gave a median estimate of 25 percent. Those who had seen 65 gave a median estimate of 45 percent. A number that everyone knew to be random, generated by a device with no connection to the question, shifted estimates by twenty percentage points. Tversky and Kahneman called the phenomenon anchoring and adjustment. People faced with an uncertain quantity start from an available value, the anchor, and adjust away from it toward what they believe is the right answer. The adjustment is typically insufficient, so the final estimate remains biased toward the starting point. Later research suggested that adjustment is only part of the story. Fritz Strack and Thomas Mussweiler argued in the 1990s that anchors also work by selective accessibility: considering whether the answer might be close to the anchor brings to mind information consistent with it, which then shapes the estimate. Whatever the precise mechanism, the effect is one of the most robust in the psychology of judgment. It appeared reliably in the Many Labs replication project of 2014, a coordinated attempt by dozens of laboratories to reproduce classic findings, at a time when many other famous effects were failing to replicate. Anchors among experts It would be natural to suppose that anchoring afflicts people guessing at trivia but not trained professionals making judgments in their own field. The evidence says otherwise. Gregory Northcraft and Margaret Neale showed in 1987 that estate agents asked to value a house were influenced by the listing price they were given, even though they denied that it had affected them. Birte Englich, Thomas Mussweiler and Fritz Strack, in a series of studies culminating in a 2006 paper titled "Playing Dice with Criminal Sentences", gave experienced German judges a realistic case file and asked them to decide a sentence. Before doing so, the judges rolled a pair of loaded dice, which produced either a low or a high number, and were asked whether the sentence should be higher or lower than that number. Judges who rolled the high number gave substantially longer sentences than those who rolled the low one. They knew the dice were irrelevant. They were influenced anyway. These findings matter because they dispose of the most common objection to worrying about anchoring in AI data: that annotators, especially expert ones, will simply ignore an irrelevant or unreliable suggestion. People do not ignore anchors, even when they know they should. And in the modern annotation pipeline, anchors are not irrelevant numbers on a wheel. They are plausible suggestions, presented as the output of a capable system, placed directly in the field where the annotator's answer is supposed to go. Pre-annotation: the anchor built into the workflow Labelling data from scratch is slow and expensive. A natural way to speed it up is to have a model make a first attempt and ask human annotators only to check and correct it. The practice goes by many names: pre-annotation, pre-labelling, model-assisted labelling, human-in-the-loop correction, active verification. It is now standard across the industry. Commercial annotation platforms advertise it as a core feature, and it is the default workflow for many tasks, including drawing bounding boxes around objects in images, transcribing speech, tagging parts of speech, segmenting medical images and labelling entities in text. Pre-annotation does make labelling faster, often dramatically. But from the point of view of cognitive psychology, it converts every annotation into an anchoring task. The annotator is no longer asked "what is in this image?" but, in effect, "is this suggestion correct, or how far should it be adjusted?" That is exactly the structure Tversky and Kahneman used. And the predicted result follows: final labels lean toward the pre-annotation. The effect has been measured. Karën Fort and Benoît Sagot, in a 2010 study of part-of-speech tagging, found that pre-annotation sped annotators up and could improve agreement, but also introduced a measurable bias toward the tags the automatic tool had proposed: annotators were more likely to leave an incorrect suggested tag in place than to arrive at the same wrong tag independently. Studies in other language tasks have reported similar patterns, and the general finding is consistent: pre-annotation improves speed and often raises agreement, while systematically pulling labels toward the machine's suggestions. The rise in agreement is itself partly an artefact of the anchor. If all annotators start from the same suggestion and adjust insufficiently, they end up closer to each other, and the agreement statistics improve even if accuracy does not. Spatial tasks show the same pattern in a geometric form. When a model proposes a bounding box around a pedestrian or a tumour, the annotator tends to nudge its edges rather than redraw it. If the model systematically draws boxes slightly too tight, the corrected boxes will be slightly too tight as well, though less so. If the model misses small or partly hidden objects, annotators reviewing its output are more likely to miss them too than annotators labelling from scratch, because the task has shifted from searching to checking. Checking a set of proposed boxes is a quite different cognitive activity from scanning an image for every instance of a category, and it shares the weakness the radiologists showed with the gorilla: attention is directed by what one is shown to look for. Automation bias: anchoring with authority When the anchor comes from a machine, a second, related phenomenon compounds it. Researchers in human factors call it automation bias: the tendency of people working with automated decision aids to over-rely on them, accepting their recommendations without sufficient scrutiny (errors of commission) and failing to notice problems the aid did not flag (errors of omission). Linda Skitka, Kathleen Mosier and Mark Burdick documented the effect in a series of studies with pilots and students in simulated flight tasks in the late 1990s. Participants using an automated monitoring aid that was usually reliable but occasionally wrong missed events the aid failed to signal, and followed its incorrect directives even when other information on their displays contradicted them. Raja Parasuraman and Dietrich Manzey's 2010 review, "Complacency and Bias in Human Use of Automation", synthesised the literature and found that the effects appear in novices and experts alike, are not eliminated by training, and become stronger when people are busy with other tasks. Medicine has generated the most worrying evidence, because computer-aided detection has been used in screening for decades. In mammography, computer-aided detection systems mark regions of an image they consider suspicious, and the radiologist decides what to report. Eugenio Alberdi and colleagues, in a 2004 study, found that when the system failed to prompt on a cancer that was visible in the image, readers were less likely to detect that cancer than when reading without the system at all. The absence of a prompt acted as a reassurance. A large study by Constance Lehman and colleagues, published in JAMA Internal Medicine in 2015 using data from the Breast Cancer Surveillance Consortium, found that computer-aided detection, which had become nearly universal in US mammography screening because insurers reimbursed it, did not improve diagnostic accuracy, and was associated with lower sensitivity among radiologists who read both with and without it. Whatever the full explanation, the case illustrates that placing a machine suggestion in front of an expert changes the expert's judgment in ways that are not reliably beneficial. For annotation, automation bias means that pre-annotations do not merely anchor; they also carry implied authority. An annotator who sees a suggestion produced by "the model" may assume, often correctly, that it is right most of the time, and so set a high bar for overruling it. When the suggestion is right, this saves effort. When it is wrong in a systematic way, the error passes into the data with a human signature on it. How the anchor closes the loop The most important consequence of anchoring in annotation is structural. Pre-annotations are produced by a model, and the corrected labels are used to train the next version of that model, or a different model, or to evaluate one. The model's errors are partly passed back to it as ground truth. Consider a sequence that is common in practice. A team trains an initial object detector on a modest hand-labelled dataset. They use it to pre-annotate a much larger set of images. Annotators correct the pre-annotations, working quickly because they are paid per image and the suggestions are usually good. The corrected set trains a better detector, which is used to pre-annotate the next batch. At each stage, the detector's systematic tendencies, such as missing small objects, missing objects in poor lighting, or drawing particular shapes of box, are partly corrected by annotators and partly preserved. What is preserved enters the training data, and the next detector learns it. Over several rounds, the dataset converges toward something that is a blend of the world and the model's view of the world, and nobody can say precisely how much of each. The same structure appears at the level of evaluation. If annotators are shown a model's output and asked to judge whether it is correct, their judgments are anchored on the output. Systems evaluated this way look more accurate than they would if annotators had first formed their own answer and then compared. Teams that report high human-verified accuracy for model outputs are often, without intending to, reporting the strength of an anchoring effect. There is an awkward fact here for the economics of the industry. Pre-annotation is popular because it is cheap, and the cheapness comes precisely from reducing the cognitive work annotators do on each item. But reduced cognitive work is the condition under which anchors are least effectively adjusted. The cheaper the verification, the more of the model's view survives into the labels. Anchors in ratings and scales Anchoring does not require a pre-annotation. Rating tasks contain anchors of their own, and they are among the most common sources of systematic distortion in preference and quality data. The first items in a batch set a reference point. If an annotator rating the helpfulness of chatbot responses on a scale of one to seven sees three excellent responses first, subsequent average responses will tend to be rated lower than if the first three had been poor. Psychologists have studied these sequential contrast and assimilation effects since the early twentieth century; they are a reason careful survey designers randomise item order. In annotation, items often arrive in an order determined by the data pipeline, such as by source, by date or by a prior model score, which turns ordering into a systematic rather than random influence. The scale itself can anchor. The labels on the points of a rating scale, the default position of a slider, the number of points, and whether the midpoint is labelled all affect where people place their answers. A slider that begins at the midpoint pulls ratings toward the middle; one that begins at zero pulls them toward the low end. A rating interface that pre-fills the previous item's rating encourages repeating it. These effects are individually small, but they are systematic, and when the ratings are used to train a model that predicts quality, the model learns to reproduce their distribution. Comparative ratings, in which annotators choose which of two items is better, are often promoted as a remedy, because they avoid the problem of calibrating an absolute scale. They help, but they introduce their own anchors. The first item shown becomes a reference against which the second is judged, and position effects, a systematic preference for the item in one position regardless of content, have been documented in many choice settings. In the human preference data used to fine-tune large language models, the presentation order of the two responses is normally randomised for exactly this reason. Randomisation turns a systematic bias into noise, which is the right move, but only works if it is done, and only for the biases it targets. The anchor the annotator brings: language models in the crowd Pre-annotation is an anchor the task designer chooses to show. A newer anchor is one annotators supply for themselves. Since general-purpose language models became widely available, many crowdworkers have used them to help with annotation and writing tasks, whether or not the task allowed it. Veniamin Veselovsky, Manoel Horta Ribeiro and Robert West tested how common this was in a 2023 study titled "Artificial Artificial Artificial Intelligence". They posted a summarisation task on Amazon Mechanical Turk, asking workers to summarise medical research abstracts, and used a combination of a trained classifier and logged keystrokes, including copy-and-paste behaviour, to estimate how many submissions had been produced with the help of a language model. Their estimate was that roughly a third to nearly half of the workers had used one. The task they chose was one in which language models are especially helpful, so the figure is not a general rate for all annotation. But the direction is clear, and annotation vendors and research teams have since introduced measures to detect or discourage the practice. From the point of view of anchoring, the consequence is that a model's view can enter human data without anyone in the pipeline intending it. An annotator who asks a chatbot whether a passage is sarcastic, then adjusts its answer slightly, has anchored their own judgment on a model's. Their label looks like an independent human judgment, and is recorded as one, but it shares the model's tendencies. If many annotators consult the same few widely available models, their labels converge on those models' views, inter-annotator agreement rises, and the data appear more reliable than before. As with pre-annotation, the rise in agreement is partly a measurement of a shared anchor. And because the anchor is invisible to the task designer, there is no record of which labels were influenced and which were not. This matters especially for datasets intended to capture human perspectives, such as preference data for training language models or opinion data in social science. A dataset of human judgments that is partly a dataset of model judgments relayed by humans cannot serve as an independent check on models. It is a mirror that has been mistaken for a window. What makes anchoring distinctive Among the biases in this booklet, anchoring has two distinctive features. The first is that it requires no prior belief on the part of the annotator. Confirmation bias needs an expectation; anchoring needs only a number, a box, a tag or a suggestion in front of the person. This makes it easier to induce and easier to control. Because the anchor is a property of the task design rather than of the annotator, the task designer can decide when it appears, what it says and how much prominence it has. Chapter 8 discusses how: by having annotators commit to a judgment before seeing any suggestion, by measuring how often suggestions are overruled, and by inserting known-wrong suggestions to check whether annotators are actually checking. The second is that in contemporary pipelines, the anchor is usually a model. That creates a direct channel through which a model's systematic errors can be laundered into human-verified data. Of all the routes by which cognitive bias enters machine learning, this is the one most clearly growing in importance, because pre-annotation, model-assisted review and AI-generated draft labels are becoming the norm across the industry. A world in which most training data is produced by humans correcting machines is a world in which anchoring determines a good deal of what the machines learn next. The next bias operates at a different level altogether. Confirmation and anchoring both act on the judgment made about a particular item. Survivorship bias acts on which items are available to be judged at all. Chapter 4: Learning Only from the Survivors During the Second World War, the Statistical Research Group at Columbia University, a team of mathematicians working for the United States military, was asked a version of a practical question: given the pattern of damage on bombers returning from missions, where should extra armour be placed? The intuitive answer was to reinforce the areas that showed the most bullet holes. The statistician Abraham Wald saw the flaw. The aircraft being examined were the ones that had come back. Damage in the places that were riddled with holes was, by definition, survivable. The planes hit in the places that showed few holes on the returning aircraft were the planes that had not returned. The armour, Wald argued, should go where the holes were not. The story has been simplified in the retelling, and the details of Wald's actual memoranda were more mathematical and less dramatic than the popular version. But the principle is exact, and it has a name: survivorship bias. It is the error of drawing conclusions from a set of cases that has been filtered by some process related to the outcome of interest, while forgetting that the filter exists. The survivors are visible; the casualties are not. Whatever distinguishes survivors from casualties is invisible in a sample of survivors, and anything inferred from that sample about the whole population is distorted. Survivorship bias is unlike the other biases in this booklet in one respect. It is primarily a property of data, not of perception. A perfectly rational observer given only the returning bombers would still be misled unless they knew to ask about the missing ones. Its psychological component is the failure to ask: the strong human tendency, which Kahneman summarised with the phrase "what you see is all there is", to reason from the evidence in front of us without accounting for evidence that never reached us. In machine learning, that failure has particular force, because a learning algorithm is the purest possible example of a reasoner for whom what it sees is all there is. It has no concept of the cases that are absent. It cannot ask where the other bombers went. The selective labels problem The most important form of survivorship bias in machine learning arises when a model is trained on outcomes that are observed only for cases that passed an earlier human decision. Himabindu Lakkaraju, Jon Kleinberg, Jure Leskovec, Jens Ludwig and Sendhil Mullainathan gave it a precise name in a 2017 paper: the selective labels problem. Their example was pretrial release. In many jurisdictions, a judge decides whether a defendant awaiting trial should be released or held in custody. If the defendant is released, it becomes possible to observe whether they failed to appear in court or were arrested for a new offence. If they are held, no such outcome can occur, so no label is ever recorded. Any model trained to predict pretrial misconduct from historical data is therefore trained only on the defendants judges chose to release. The defendants judges chose to detain, who may differ from the released group in ways visible to the judge but not recorded in the data, are absent from the outcome labels. Worse, the judge's decision was influenced by the judge's own expectations about who was risky, so the labelled set is shaped by human judgment in exactly the way survivorship requires. The same structure appears in lending. A bank can observe whether a borrower repaid only if it granted the loan. Applicants it rejected generate no repayment history. A credit scoring model trained on historical outcomes learns the relationship between applicant characteristics and repayment among people the bank's earlier officers or earlier models chose to approve. If those earlier decisions systematically rejected some kinds of applicant, such as people with thin credit files, recent immigrants or residents of particular neighbourhoods, the model has little or no information about how such applicants would actually have performed. The credit industry has known about the problem for decades and calls the attempt to correct for it reject inference. No statistical method fully solves it, because the missing outcomes are genuinely missing; every correction rests on assumptions about what they would have been. Hiring offers a third and widely reported example. In 2018 Reuters reported that Amazon had abandoned an experimental system for rating job applicants after discovering that it penalised CVs that included the word "women's", as in "women's chess club captain", and downgraded graduates of two all-women's colleges. The system had been trained on CVs submitted to the company over a ten-year period, a set dominated by men, as is common in technology. The case is often described as a straightforward representation problem, but its survivorship dimension is equally important. A hiring model trained on who was hired, or on how well hired employees subsequently performed, learns only from people who made it through a previous selection process run by humans. The qualities of rejected candidates who would have excelled are invisible. Whatever the earlier recruiters overlooked, the model inherits as a blind spot. How human judgment shapes the filter What makes these cases more than a statistical curiosity is that the filter is itself a human judgment, with all the biases discussed so far. The judges who decided pretrial release, the loan officers who approved credit, and the recruiters who shortlisted candidates were subject to confirmation bias, anchoring and the other heuristics catalogued in the next chapter. Their decisions determined which cases produced labels. So survivorship bias acts as a second channel through which human cognitive error reaches the model: not by distorting what a label says, but by determining which cases are labelled at all. The interaction can be subtle. Suppose loan officers, influenced by a stereotype, rejected most applicants from a particular group but approved a few who had unusually strong applications. The approved members of that group would be an exceptionally strong subset, and would likely repay at high rates. A model trained on these outcomes might then conclude that members of the group are good risks, the opposite of the stereotype, or it might find that the group variable is uninformative, because its effect has been cancelled by selection. Either way, its estimate for a typical applicant from that group, who looks nothing like the select few who were approved, would be unreliable. Survivorship does not always produce bias in the direction of the original prejudice. It produces bias in whatever direction the selection process happened to push, which is often unpredictable without knowing how the selection was made. In predictive policing, the filter runs through where police officers chose to patrol. Arrests and recorded crimes are observed where police are present. A model trained on historical arrest data to predict where crime will occur learns, in part, where police have previously looked. Kristian Lum and William Isaac demonstrated this in a 2016 analysis of drug crimes in Oakland, California, comparing police records with public health survey estimates of drug use. Drug use was spread broadly across the city, but recorded drug crimes, and the predictions of a widely used predictive policing algorithm trained on them, were concentrated in predominantly Black and low-income neighbourhoods. The data recorded the places where policing had happened, and the algorithm learned to recommend more policing there. That feedback loop is the subject of Chapter 7. Survivorship in the curation of data Selective labels arise from decisions about individual cases. A second form of survivorship operates at the level of whole datasets: the curation decisions that determine which text, images or records are kept for training and which are discarded. Modern language models are pretrained on enormous collections of web text, and the raw web is too noisy to use without filtering. Developers therefore apply quality filters that retain some documents and discard others. Some filters are simple rules. Others are classifiers trained to recognise "high-quality" text, often by treating a reference corpus such as Wikipedia, published books or web pages linked from a particular site as the positive class. Whatever the filter keeps is what the model learns language from. What it discards is, from the model's point of view, not language at all. In 2022, Suchin Gururangan and colleagues examined what such a filter actually selects. They applied a quality classifier modelled on the one described for GPT-3 to a corpus of articles from American high school newspapers, for which they could link articles to information about the schools' locations. The filter was more likely to rate articles as high quality when they came from schools in larger, wealthier, more educated and more urban areas. It encoded, in the authors' words, a particular language ideology: one variety of English, associated with particular social groups, counted as quality, and others were filtered out. Jesse Dodge and colleagues, documenting the Colossal Clean Crawled Corpus (C4) in 2021, found that a blocklist used to remove offensive content had disproportionately removed text about and by LGBTQ people and text in African American and Hispanic-aligned English, because words that appear in those contexts appeared on the list. These filters are survivorship mechanisms. The model is trained on the documents that survived, and whatever correlates with survival, including dialect, topic and social group, is learned as part of what language is. A model trained only on surviving text will find the excluded varieties less probable, will generate them less fluently, and may treat them as errors. Benchmarks and the hard cases that disappear Survivorship also shapes evaluation. Benchmarks are curated, and curation tends to remove the items that annotators disagreed about, the items that were hard to label, and the items that did not fit the category scheme cleanly. The reasons are understandable: ambiguous items make for noisy evaluation, and a benchmark with clean labels is easier to use. But the result is a test set made of the survivors of a cleaning process, which systematically excludes the cases where real-world deployment is hardest. Recall the observation from Chapter 1 that human label variation is often signal. A benchmark built by keeping only items with high annotator agreement is a benchmark from which the genuinely ambiguous cases have been removed. A model that scores well on it has been shown to handle easy cases. It has not been tested on the cases where the model will most often be wrong in deployment and where users will most often disagree with it. The reverse process can produce survivorship of a different kind. Researchers who found that benchmarks contained easy shortcuts, such as the negation cue described in Chapter 2, developed adversarial filtering techniques that deliberately remove examples a simple model can solve. The resulting datasets are harder. But they are also selected: the surviving examples are those a particular filter model found difficult, and their properties depend on that model's particular weaknesses. Each act of curation leaves fingerprints on what remains. The missing in recommender and medical data Two further domains show how pervasive the problem is. Recommender systems learn from user feedback, such as ratings, clicks and purchases. But users rate items they chose to consume, and they chose to consume items they expected to like or that the platform showed them. Ratings are therefore missing not at random, as Benjamin Marlin and Richard Zemel showed in work on collaborative filtering in the late 2000s: the distribution of ratings users volunteer differs systematically from the distribution they would give if asked to rate randomly chosen items. Items no one was shown have no ratings at all. A recommender trained on the observed data learns a world in which the items that survived earlier recommendation are the only ones that exist. In medicine, the data that reach a training set are those of patients who entered the healthcare system, received a particular test, and had a recorded outcome. Patients who could not access care, were not referred for imaging, or were lost to follow-up are underrepresented or absent. Ziad Obermeyer and colleagues showed in a 2019 study in Science that a widely used commercial algorithm for allocating extra care to high-risk patients in the United States used healthcare costs as a proxy for health need. Because less money had historically been spent on Black patients with the same level of illness, often reflecting unequal access, the algorithm systematically underestimated their needs. At a given risk score, Black patients were considerably sicker than white patients. The authors estimated that correcting the bias would increase the share of Black patients identified for extra help from about 18 percent to about 47 percent. The label, cost, was a record of care actually received, and care received is the survivor of a long chain of human decisions about who gets treatment. Medical imaging datasets carry a related survivorship signature. Images reach research archives because a clinician ordered them, which means the patient had symptoms, sought care, was referred and was scanned. A dermatology dataset assembled from images taken in specialist clinics contains the lesions that general practitioners found worrying enough to refer, photographed on the patients who were referred. Roxana Daneshjou and colleagues reported in 2022, in Science Advances, that several dermatology AI models performed substantially worse on a diverse, carefully curated set of images than on the data they had been developed with, with particular difficulty on darker skin tones and uncommon diseases. Part of the explanation was simple representation, since darker skin was rare in the development data. Part was the filter: the conditions and presentations that reach specialist archives are those that clinicians, with their own trained expectations, recognised as concerning. A lesion that looked unremarkable to a referring doctor never became a training example, however dangerous it later proved to be. Table 2 collects the main forms of survivorship described in this chapter. Table 2. Forms of survivorship bias in machine learning data. Form What filters the data What goes missing Example Selective labels Earlier human decision Outcomes for rejected cases Credit, bail, hiring Exposure-driven Where attention was directed Events in unobserved places Predictive policing Proxy outcome Access to a service Need that did not become cost Care allocation Corpus filtering Quality or safety filter Dialects, topics, groups Web pretraining text Benchmark cleaning Agreement thresholds Ambiguous, hard cases Curated test sets Feedback missingness User choice, prior ranking Unshown or unchosen items Recommender ratings Asking where the bombers went The general lesson of survivorship bias is that every dataset is the output of a selection process, and the selection process is part of what the model learns. Wald's insight was to ask a question that the data could not answer by itself: which aircraft are not here, and why? The equivalent questions for a machine learning dataset are: which cases never received a label, and what decided that? Which documents were discarded, and by whose standard of quality? Which patients never reached the hospital, and which defendants were never released? These questions are hard to answer from the data, because the data, by construction, do not contain the missing cases. They require knowledge of the process that generated the data, which usually lives with the people who ran it: the loan officers, the judges, the engineers who wrote the filter, the clinicians who decided whom to test. It is one of the strongest arguments for documenting datasets in a way that records how they were assembled, a practice discussed in Chapter 8. A model cannot ask where the bombers went. Its builders must. The three biases examined so far, confirmation, anchoring and survivorship, are the most important routes by which human cognition shapes machine learning data. They are not the only ones. The next chapter turns to the supporting cast. Chapter 5: The Supporting Cast: Availability, Halo, Framing and Fatigue Confirmation bias, anchoring and survivorship bias are the main channels by which human cognition shapes training data, but they are not the only ones. The psychology of judgment has identified dozens of systematic effects, and several of them leave recognisable traces in the datasets used to build AI systems. This chapter examines the ones that matter most in practice. They are grouped by how they operate: heuristics that substitute an easy question for a hard one, effects of presentation and sequence, and the perspective of the particular people who happen to be doing the labelling. Availability: judging by what comes to mind In 1973, Tversky and Kahneman described the availability heuristic: people estimate how frequent or likely something is by how easily examples come to mind. In one of their demonstrations, participants were asked whether more English words begin with the letter K or have K as their third letter. Most said words beginning with K were more common, because such words are easier to retrieve from memory, even though words with K in the third position are in fact more numerous. Availability is usually a reasonable guide, because frequent things are usually easy to recall. It goes wrong when ease of recall is driven by something other than frequency: vividness, recency, media coverage, personal experience. In annotation, availability shapes labels whenever a judgment requires an estimate of what is typical or likely. An annotator asked whether a news headline is misleading draws on their memory of misleading headlines, which is dominated by the cases they have heard discussed. An annotator asked to rate how dangerous an activity shown in a video is compares it with dangers that come easily to mind, which are skewed toward the dramatic and the recent. An annotator asked whether an image shows a "typical" member of a category draws on the members most available to them, which depend on where they live and what they have seen. Availability also operates within the annotation session. Items seen recently are the most available reference points, which is one reason that a run of extreme examples changes how subsequent items are judged. Annotators who have just labelled a dozen graphic violent images will find a moderately aggressive comment less striking than annotators who have been labelling cooking recipes. The labels depend partly on the batch, and the batch is rarely random. Representativeness and the stereotype as shortcut Closely related is the representativeness heuristic, which Kahneman and Tversky described in 1972: judging the probability that something belongs to a category by how much it resembles the category's prototype. Their most famous illustration involved a fictional woman named Linda, described as outspoken, bright and concerned with social justice. Asked which was more probable, that Linda was a bank teller or that she was a bank teller active in the feminist movement, most participants chose the second, though it cannot logically be more probable than the first. The description resembled the prototype of a feminist, and resemblance overrode logic. In data labelling, representativeness is the cognitive engine of stereotyping. When an annotator is asked to judge something about a person from limited evidence, such as their occupation from a photograph, their emotional state from a face, or their likely intent from a short text, they compare the evidence with their mental prototypes. Those prototypes are built from experience and culture, and they encode the social stereotypes of that experience and culture. A woman in a hospital setting is more likely to be labelled a nurse than a doctor; a man in a kitchen in a domestic photograph may be labelled as something other than cooking. The ImageNet dataset provides a well-documented case. Its "person" categories were inherited from WordNet, a lexical database built in the 1980s, and were populated by crowdworkers who were shown images retrieved from web searches and asked to confirm whether each image depicted the category. The artist Trevor Paglen and the researcher Kate Crawford, in their 2019 project ImageNet Roulette and accompanying essay "Excavating AI", drew public attention to the results: categories included derogatory and moral judgments, and the images assigned to them reflected crowdworkers' snap judgments about what such people looked like. Kaiyu Yang and colleagues at Princeton and Stanford, who maintained ImageNet, published an analysis in 2020 that identified a large majority of the person categories as potentially offensive or not visually identifiable, and removed them from the version of the dataset they made available. A category such as an occupation or a personality trait cannot be reliably seen in a photograph, so crowdworkers verifying such categories had nothing to go on except resemblance to a prototype. Representativeness filled the gap. Facial expression datasets are vulnerable in a similar way. Many emotion recognition systems are trained on images labelled with emotion categories by annotators who look at a face and judge what the person is feeling. Lisa Feldman Barrett and colleagues, in a major 2019 review in Psychological Science in the Public Interest, concluded that facial configurations are far less reliable indicators of emotional states than the standard categories assume, and that the same configuration can mean different things in different contexts and cultures. Annotators labelling a frowning face as "angry" are matching it to a cultural prototype of anger. The resulting system learns the prototype and presents its output as a reading of internal states. The halo effect and the polish of an answer In 1920, the psychologist Edward Thorndike reported that army officers rating their subordinates on separate qualities, such as intelligence, leadership, loyalty and physical appearance, gave ratings that were far more correlated with one another than the qualities themselves plausibly were. An officer who rated a soldier highly on one quality tended to rate him highly on all of them. Thorndike called it the halo effect: a general impression spills over into judgments of specific attributes. The halo effect has acquired new importance with the use of human ratings to train and evaluate language models. When raters are asked to judge whether a response is accurate, helpful, harmless and well written, these judgments are not independent. A fluent, confident, well-organised response creates a general positive impression that raises ratings on attributes the rater cannot easily check, such as factual accuracy. Several studies have documented this directly. Tom Hosking, Phil Blunsom and Max Bartolo, in a 2023 paper titled "Human Feedback is not Gold Standard", asked crowdworkers to evaluate language model outputs on a range of specific error types, and manipulated how assertive the outputs sounded. More assertive outputs were judged to contain fewer factual errors, even when they contained the same number. Human preference judgments, they concluded, underrepresent factuality and are confounded by style. Prasann Singhal and colleagues, also in 2023, showed that in several widely used preference datasets, length was a strong predictor of which response raters preferred, and that much of the improvement attributed to reinforcement learning from human feedback could be reproduced by optimising for length alone. Raters treated longer answers as better answers, a form of halo in which apparent thoroughness stands in for quality. A further consequence was documented by Mrinank Sharma and colleagues at Anthropic in a 2023 paper, "Towards Understanding Sycophancy in Language Models". They found that several AI assistants trained with human feedback tended to tell users what they appeared to want to hear: agreeing with stated opinions, changing correct answers when challenged, and giving feedback that matched the user's apparent view of their own work. Analysing human preference data, they found that responses matching the user's views were more likely to be preferred, and that both humans and preference models sometimes preferred convincingly written sycophantic responses over correct ones. Agreement is pleasant, and pleasantness casts a halo. The model learned to produce it. Framing: the question shapes the answer Tversky and Kahneman's 1981 paper "The Framing of Decisions and the Psychology of Choice" showed that logically equivalent descriptions of a choice can produce opposite preferences. In their best-known example, participants chose between two programmes to combat a disease expected to kill 600 people. When outcomes were described in terms of lives saved, most chose the certain option; when the same outcomes were described in terms of deaths, most chose the gamble. Every annotation task frames its question, and the frame affects the answers. Asking whether a post "violates the community guidelines" invites a different judgment from asking whether it is "harmful". Asking annotators to rate how "toxic" a comment is, a word with connotations of contamination and danger, elicits different ratings from asking how "rude" or "disrespectful" it is. Asking whether an image is "inappropriate for children" produces different labels from asking whether it is "sexually explicit". The difference between the frames is often not noticed by the people who wrote the task, because to them the words are interchangeable descriptions of the same intended concept. To annotators, who must interpret the words in the moment, they are different questions. Framing also operates through the response options. A binary choice between "toxic" and "not toxic" forces ambiguous items into one bin or the other, and the default or first-listed option tends to collect the ambiguous cases. A scale with an explicit "unsure" option produces different distributions from one without. Including "other" as a category changes what gets assigned to the named categories. Each of these choices is a framing decision, and each leaves a systematic mark on the data. Order, fatigue and the decision-weary annotator Annotators make decisions in long sequences, and the position of an item in the sequence affects how it is judged. Some of the evidence is contested and worth stating carefully. A widely cited 2011 study by Shai Danziger, Jonathan Levav and Liora Avnaim-Pesso reported that Israeli parole board judges granted favourable rulings at a high rate at the start of each session and that the rate declined toward nearly zero before meal breaks, recovering afterward. The authors interpreted this as decision fatigue. Later analyses, including by Keren Weinshall-Margel and John Shapard in the same journal, pointed out that the order of cases was not random: prisoners without legal representation tended to be heard last in each session, and their cases were more likely to be refused. The size of the effect attributable to fatigue is therefore uncertain. The general phenomenon that judgment quality and leniency drift over long sequences of similar decisions is better supported, but this particular study should not be cited as proof of it. More directly relevant evidence comes from annotation itself. Studies of crowdsourcing have repeatedly found that the quality of work changes over the course of a session, with workers becoming faster and, in many tasks, less careful as they proceed. Workers develop shortcuts: after seeing many items they learn which features predict the label most of the time, and begin to rely on those features rather than examining each item fully. This is efficient, and on easy items harmless. On hard items it means that the label reflects the shortcut, and the shortcut becomes a regularity the model can learn. Sequential effects also include the contrast and assimilation discussed in Chapter 3. A moderately offensive item following several highly offensive items will be rated less offensive than the same item following several benign ones. Because items in real annotation queues often arrive grouped by source, topic or prior model score, these sequential influences can be correlated with properties of the items, which converts them from noise into bias. Whose judgment? The perspective of the annotator pool The final source of systematic influence is not, strictly speaking, a cognitive bias. It is the perspective that comes from who the annotators are. Every person's judgments of social content are shaped by their culture, language, experiences and identity. That is not a defect; it is what judgment is. But when a dataset is labelled by a pool of annotators drawn from a narrow population, the perspective of that population becomes the standard against which everyone else's content is judged. This has been measured repeatedly. Hala Al Kuwatly, Maximilian Wich and Georg Groh, in a 2020 study, found that annotators' demographic characteristics, including age, education and first language, affected their hate speech labels, and that models trained on labels from different groups behaved differently. Deepak Kumar and colleagues, in a 2021 study involving over seventeen thousand participants, found substantial disagreement about what counted as toxic content between people of different backgrounds, and argued that a single toxicity score cannot represent a diverse population. Work at Google by Aida Mostafazadeh Davani, Mark Díaz and Vinodkumar Prabhakaran has proposed modelling individual annotators' judgments separately rather than collapsing them into one label, so that the disagreement is preserved and can be examined. Perspective interacts with every bias discussed in this booklet. Confirmation bias is the tendency to see what one expects, and what one expects depends on one's experience. Representativeness relies on prototypes, and prototypes are cultural. Availability depends on which examples are familiar. A pool of annotators who share a background will share their expectations, prototypes and available examples, and their agreement will look like objectivity. Table 3 summarises the effects covered in this chapter and how each tends to appear in data. Table 3. Additional cognitive effects and their traces in training data. Effect Core mechanism Typical trace in data Example task Availability Ease of recall as frequency Labels track recent or vivid cases Risk and harm ratings Representativeness Resemblance to prototype Stereotyped category assignments Occupation, emotion labels Halo Global impression spreads Style raises unrelated ratings Chatbot response ratings Framing Wording changes the question Shifts tied to task phrasing Toxicity versus rudeness Order and fatigue Position in sequence Drift and shortcut reliance Long moderation queues Shared perspective Common background of pool Agreement on a partial view Offensiveness, sentiment The perspective problem is not confined to social content. In medical imaging, labels often come from a small number of specialists at a single institution, trained in a particular school of practice. Inter-reader variability among radiologists and pathologists is well documented for many tasks, such as grading tumours or classifying the density of breast tissue, and some of that variability is systematic: individual readers and institutions have consistent tendencies to grade higher or lower. A model trained on labels from one institution learns that institution's habits of grading, and may disagree with experts elsewhere not because it is wrong by any absolute standard but because it has learned a local convention. The local convention is a shared perspective in exactly the sense that matters here. Interactions and compounding In practice, these effects rarely act alone. An annotator late in a long session, working in a second language, rating chatbot responses presented with a pre-filled score, on a scale labelled with emotive words, is subject to fatigue, anchoring, framing and halo at once. A crowdworker in one country labelling the offensiveness of slang from another is subject to representativeness, availability and confirmation together, each pushing in a direction set by the same cultural distance. Compounding matters because it makes the overall distortion larger than any single experiment would suggest, and because it makes the distortion hard to diagnose. A team that tests for one bias, say position effects in paired comparisons, and corrects it by randomising order may conclude that its data are clean, when the larger influences lie elsewhere. The biases in this chapter are best thought of not as a checklist of separate errors but as a description of the conditions under which human judgment is reliable and the conditions under which it is not. Annotation pipelines that create the second set of conditions will produce biased data regardless of which named bias is responsible. The question that follows is mechanical. Given labels shaped in all these ways, what exactly does a learning algorithm do with them, and why are the resulting models so faithful to their labellers' minds? Chapter 6: From Biased Labels to Skewed Decisions The previous chapters described how human judgment acquires systematic tilts on its way into training data. This chapter follows the data the rest of the way: into a learning algorithm, out of it as a trained model, and into the decisions that model makes. The central question is why the transfer is so faithful. One might hope that a powerful learning system, trained on millions of examples, would see through the quirks of its labellers to the underlying truth. In general it does not, and the reasons are structural. They lie in what learning algorithms are optimised to do, in how labels are aggregated before training, in how models are evaluated, and in a family of effects through which models can end up more biased than the data that trained them. The objective is to agree with the labels A supervised learning algorithm is trained to minimise a loss function, which measures how far its predictions are from the labels. Every widely used loss function, whether cross-entropy for classification or squared error for regression, rewards the model for reproducing the labels as closely as possible. Nothing in the objective rewards the model for being right about the world when the labels are wrong. From the algorithm's point of view, the labels are the world. This is not a defect of any particular method. It is what supervised learning means. A model that reproduced the intended concept perfectly, while disagreeing with the labels wherever the labellers were biased, would score worse on the training objective and worse on any test set labelled the same way. Optimisation pushes the model away from that hypothetical ideal and toward a faithful reproduction of the labelling process, including its systematic tendencies. It follows that the question "will the model learn the annotators' bias?" has a simple answer in principle: it will learn the bias to the extent that the bias can be predicted from the input. If annotators judged tweets in a particular dialect more harshly, and the dialect is visible in the text, the model will learn to judge that dialect harshly, because doing so reduces its loss. If annotators were anchored on a pre-annotation produced by a model with a recognisable pattern of errors, and the inputs that trigger those errors are recognisable, the new model will learn the pattern. If radiologists' labels were influenced by clinical context that left traces in the image, such as markers placed by technicians in particular departments, the model will learn those traces. The gap between the concept and the column There is a useful vocabulary for thinking about this, borrowed from the social sciences. Abigail Jacobs and Hanna Wallach, in a 2021 paper titled "Measurement and Fairness", argued that most of what machine learning systems are asked to predict are unobservable theoretical constructs, such as creditworthiness, toxicity, risk, relevance, quality or need, and that labels are measurements of those constructs, produced by a measurement model that may or may not be valid. When a dataset records "toxic" for a post, it records the output of a measurement process, the annotators and their task, that is supposed to track the construct of toxicity. The question of whether it does is a question of construct validity, the same question a psychologist asks of an intelligence test or an economist of a poverty index. Seen this way, each cognitive bias in the preceding chapters is a specific threat to validity. Confirmation bias makes the measurement depend on the expectations of the measurer. Anchoring makes it depend on the instrument's default reading. Survivorship makes the measurement available only for a filtered subset of cases. Halo makes one dimension of the measurement contaminate another. The obligation that follows is the one any field with serious measurement practices accepts: to establish that the measurement tracks the construct, to understand when it fails, and to report those failures alongside the results. Machine learning has inherited the measurements without, for the most part, inheriting the discipline that accompanies them elsewhere. A model trained on a measurement that does not track its construct will be optimised, with great precision, toward the wrong target. The cost-for-need substitution that Obermeyer and colleagues uncovered, described in Chapter 4, is the clearest example. The developers wanted to predict health need. They measured it with healthcare cost, a column that was available and that correlated with need on average. The correlation was weaker for patients whose access to care had been limited, so the measurement failed systematically for them. Nothing in the model's training or its headline accuracy revealed the failure, because the model predicted cost well. It was the construct that had been lost, between the concept and the column. Shortcuts: the easiest regularity wins That last example points to a further reason the transfer is so efficient. Learning algorithms, especially deep neural networks, have a strong tendency to pick up whichever regularity in the data is simplest to learn, even when a more meaningful regularity is also present. Robert Geirhos and colleagues called this shortcut learning in a widely cited 2020 paper. A model trained to recognise cows may learn to recognise grass; a model trained to detect pneumonia may learn to recognise the hospital where the image was taken. The pneumonia example is real. John Zech and colleagues reported in 2018 that convolutional neural networks trained to detect pneumonia in chest X-rays from several hospital systems could also identify which hospital system an image came from with very high accuracy, and that the models used features associated with the hospital, including metal tokens placed on the patient by technicians, as part of their prediction. Because the prevalence of pneumonia differed between hospitals, knowing the hospital helped predict the label. Alex DeGrave, Joseph Janizek and Su-In Lee found something similar in 2021 with models trained to detect COVID-19 in chest radiographs: many relied on shortcuts such as text markers and patient positioning that correlated with the data source rather than on the pathology itself. Sebastian Lapuschkin and colleagues, in a 2019 paper in Nature Communications, described a model trained on a standard image benchmark that classified pictures of horses partly by detecting a copyright watermark that appeared on many horse images in the dataset. They called such models "Clever Hans" predictors, after the horse that appeared to do arithmetic but was in fact reading its trainer's unconscious cues. The name is apt for this booklet's argument. Clever Hans was not reading the world; he was reading a person. Models trained on human-labelled data frequently do the same. When annotator bias produces a correlation between a surface feature and a label, that correlation is often the simplest regularity in the data, simpler than the concept the task designer intended. The negation cue in natural language inference, the dialect cue in toxicity detection, the length cue in preference data: each is easier to compute than the underlying meaning. The model's tendency to take shortcuts and the annotators' tendency to produce shortcuts reinforce each other. Annotators rely on a heuristic because it saves cognitive effort; the model learns the heuristic because it saves optimisation effort. Aggregation removes the evidence Before most labels reach a model, they pass through aggregation. Multiple annotators label each item, and their judgments are combined, by majority vote or by a statistical model of annotator reliability, into a single training label. Chapter 2 noted that aggregation does not correct a bias shared by most annotators. Its effect on the model deserves a closer look, because aggregation also changes what the model can know about its own uncertainty. Suppose five annotators rate a social media post, and three say it is offensive and two say it is not. Majority vote produces the label "offensive". The model is trained to predict "offensive" with full confidence. It never learns that the item was contested, or that the two dissenting annotators may have shared a background the others lacked. If many items show the same three-to-two pattern along the same lines of disagreement, the model learns the majority's view as a firm rule, and the minority's view vanishes. Its predicted probabilities will be confidently wrong for precisely the population whose judgment was outvoted. Methods exist to preserve disagreement. Training on soft labels, the proportion of annotators who chose each option, rather than on a single hard label, lets the model learn that some items are genuinely contested. Joshua Peterson and colleagues showed in 2019, using a dataset they called CIFAR-10H in which each test image had been labelled by around fifty people, that training on the full distribution of human labels improved models' robustness and generalisation. Multi-annotator models that predict each annotator's label separately, as in the work by Davani and colleagues mentioned in Chapter 5, go further. These approaches are still uncommon in production, partly because most datasets do not record individual annotations. Once aggregated, the disagreement cannot be recovered. Evaluation inherits the same blind spot A natural check on a model is to evaluate it on held-out data. But held-out data is almost always labelled by the same process as the training data, often by the same annotators, under the same guidelines and interface. A model that has faithfully learned the annotators' biases will agree with the test labels wherever the biases are consistent, and its measured accuracy will be high. The problem is circular. The test measures agreement with a biased reference, and the model has been trained to agree with that reference. High accuracy shows that the model has learned the labelling process well. It does not show that the labelling process captures the intended concept. Northcutt's finding that benchmark test sets contain systematic errors, discussed in Chapter 1, has a sharp consequence here: when the researchers corrected the test labels, the ranking of models on some benchmarks changed. Larger, more flexible models, which had fitted the noisy training labels more closely, did worse on the corrected test sets than their scores on the original sets suggested. Models that learned the labellers' errors were being rewarded for it. Breaking the circle requires evaluation data produced by a different process: different annotators, different guidelines, adjudication by experts, or labels derived from a more reliable source such as a later definitive diagnosis. It also requires evaluation broken down by subgroup. A model may be accurate on average while concentrating its errors on a particular dialect, demographic or image type, and those concentrated errors are where the annotators' biases typically live. Joy Buolamwini and Timnit Gebru's 2018 Gender Shades audit of commercial gender classification systems found that error rates for darker-skinned women reached about 35 percent for the worst-performing system, against less than one percent for lighter-skinned men. The systems' headline accuracy hid the gap entirely. How models amplify bias The most troubling possibility is that models do not merely reproduce the biases in their data but strengthen them. There is good evidence that this can happen, through at least three mechanisms. The first is amplification of correlations. Jieyu Zhao and colleagues, in a 2017 paper titled "Men Also Like Shopping", studied models trained to label activities in images, using a dataset in which activities such as cooking were associated with women in the training labels at a rate already skewed by the annotators and the images collected. They found that the trained models predicted the associated gender even more often than the training data did. For cooking, women appeared as the agent in about two-thirds of the training images involving cooking, while the model predicted a woman in over four-fifths of the cooking images in the test set. When a correlation helps the model predict the label, it can be advantageous to over-rely on it, especially for ambiguous inputs where other evidence is weak. The model turns a statistical tendency into something closer to a rule. The second is the effect of thresholds. Many models output a score that is converted into a decision by a threshold: posts scoring above 0.8 are removed, applicants scoring above 0.6 are approved. A small systematic shift in scores, caused by biased labels, can produce a large shift in decisions if many cases sit near the threshold. If a toxicity model assigns posts in one dialect scores slightly higher on average, and the removal threshold sits in the region where those posts cluster, the difference in removal rates can be much larger than the difference in average scores. The third is optimisation pressure on a learned proxy. When a model is trained to maximise a score produced by another model trained on human judgments, as in reinforcement learning from human feedback, it will find and exploit any systematic tendency in that score. If the reward model has learned that raters prefer longer, more confident or more agreeable answers, the policy model will produce longer, more confident and more agreeable answers, pushing well beyond the range of the original data. Leo Gao, John Schulman and Jacob Hilton documented the general phenomenon in 2022 in a study of reward model over-optimisation: as a policy is optimised harder against a learned reward model, its score on the learned reward keeps rising while its quality as measured by a better reference eventually falls. This is Goodhart's law, that a measure which becomes a target ceases to be a good measure, applied to a measure built from human cognitive tendencies. The halo effect and sycophancy described in Chapter 5 are the human tendencies being optimised. From score to decision to consequence A model's output becomes consequential when it feeds a decision. The decision may be fully automated, as when a content filter removes a post, or advisory, as when a risk score is shown to a judge or a clinician. In either case, the biases learned from labels reach real people. In fully automated settings, the scale effect noted in Chapter 1 dominates. A tendency that affected a few hundred labels now affects millions of decisions. Because automated decisions are rarely reviewed individually, errors concentrated in a particular group may go undetected for a long time, surfacing only through external audits or public complaint. In advisory settings, the model's output becomes an anchor for the human decision-maker, completing a circle with Chapter 3. Ben Green and Yiling Chen, in experiments published in 2019, showed participants risk assessments while they made pretrial and lending predictions. Participants did not use the assessments in the way the designers intended; their adherence varied, and they were more likely to deviate from the assessment in ways that disadvantaged Black defendants. The human in the loop does not neutralise the model's biases and may add their own, and the model's output shapes what the human sees as the relevant question. Megan Stevenson's 2018 study of Kentucky's adoption of mandatory pretrial risk assessment found that it produced only a small and short-lived increase in pretrial release, partly because judges overrode the recommendations in many cases. The interaction between human judgment and algorithmic advice is complex, and neither component can be assessed in isolation. Why this chain is hard to see The full path from a cognitive bias in an annotator to a skewed decision about a person passes through many hands: a guideline author, a vendor, a crowd of annotators, an aggregation script, a training run, an evaluation suite, a threshold setting, a product team and a user. At each step, the bias is less visible. The annotator does not notice the influence of their expectations. The aggregation removes the evidence of disagreement. The training process converts the tilt into model weights that cannot be read. The evaluation, using the same labels, reports success. The product team sees a number. The person affected sees a decision with no explanation. This is why the transfer of human cognitive bias to machine decisions is so persistent. It is not hidden by anyone's intention. It is hidden by the structure of the pipeline, in which each stage takes the output of the previous one as given. Interrupting the chain requires knowing what to look for at each stage, and, as the next chapter shows, the chain does not stop at the decision. It loops back. Chapter 7: Closing the Loop: When Machine Output Becomes Human Input The account so far has followed a straight line: human judgment produces labels, labels train a model, the model makes decisions. In reality the line bends back on itself. Model outputs shape what people see, what they do, and what they believe, and those changed human behaviours and judgments become the data on which the next model is trained. Once that loop is closed, the cognitive biases of the original labellers are no longer a one-time contamination. They become self-reinforcing, and they can grow. This chapter looks at four kinds of feedback loop: loops through the world, in which a model's decisions change what data is collected; loops through attention, in which a model shapes what people are exposed to and therefore what they choose; loops through judgment, in which a model's outputs become anchors for the humans who label future data; and loops through generation, in which model-produced content enters the data used to train future models. Loops through the world: policing and lending Predictive policing is the classic case, and Chapter 4 introduced it. Danielle Ensign and colleagues formalised the mechanism in a 2018 paper titled "Runaway Feedback Loops in Predictive Policing". Their model was deliberately simple. A city has two districts with the same underlying crime rate. A system allocates police to the district with more recorded crime, and crime is recorded only where police are present to observe it. If one district starts with slightly more recorded crime, perhaps because of historical patrol patterns, it receives more police, which produces more recorded crime there, which justifies more police. The allocation runs away toward the district that started ahead, regardless of the true crime rates. The authors showed that the loop could be broken by changing how observations were weighted, essentially by correcting for the fact that the system had chosen where to look. The cognitive dimension of this loop lies in how the initial data were generated and how officers respond to the system's outputs. Officers sent to an area labelled "high risk" enter it with an expectation, and expectation shapes perception, as Chapter 2 described. An ambiguous encounter in a designated hotspot is more likely to be interpreted as suspicious, more likely to lead to a stop, and more likely to produce a record. The model's prediction primes confirmation bias in the people who generate its next round of training data. The loop runs through the world and through human minds at once. Lending has a parallel structure, discussed under selective labels in Chapter 4. A credit model that rejects applicants with a particular profile never observes whether they would have repaid. The next model, trained on the outcomes of the first model's approvals, has even less information about that profile, and its uncertainty about such applicants tends to be resolved by rejecting them. Without deliberate exploration, such as approving a random sample of applicants the model would reject, the gap in the data never closes. Banks are understandably reluctant to make loans their model predicts will fail. The result is a self-confirming belief, held by an institution and encoded in a model, that is never tested. Loops through attention: recommendation and popularity Recommender systems learn from what users click, watch, buy and rate. But users can only engage with what the system shows them, and the system shows them what earlier versions predicted they would like. The training data is therefore a record of user choices made within a menu chosen by the model. Allison Chaney, Brandon Stewart and Barbara Engelhardt showed in a 2018 simulation study, "How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility", that training recommenders on data generated under previous recommendations tends to make users' consumption more similar to each other's and can reduce the benefit users get, compared with what an ideal recommender would achieve. Popular items become more popular because they are shown more, and are shown more because they are popular. Items that were never shown accumulate no data and are never learned about. The world of available content, as experienced by users, narrows toward what the system already believes they want. Human cognition contributes at both ends. Users choose from the options presented, and the availability and anchoring effects described in earlier chapters mean that the first, most prominent and most familiar options are favoured. Position effects in search results are large: users click the top results far more often than lower ones, partly because they trust the ranking and partly because they do not scroll. Each click is logged as evidence of relevance, though much of it is evidence of position. Thorsten Joachims and colleagues showed in eye-tracking studies in the mid-2000s that users' clicks were strongly biased by rank, and that clicks were better interpreted as relative preferences between items the user had actually examined than as absolute judgments of relevance. A ranking model trained naively on clicks learns to reproduce its own previous ranking, adding the users' trust in that ranking to its confidence. Loops through judgment: model-assisted annotation at scale Chapter 3 described how pre-annotation anchors human labellers on a model's suggestions, and how corrected labels can carry the model's errors into the next training round. At the scale of a whole industry, this becomes a loop of considerable importance. Increasingly, the human role in data production is to review, correct, rank or approve outputs generated by models. Annotation vendors provide model-generated first drafts. Content moderators review items that automated classifiers have already flagged, sorted by the classifier's confidence. Radiologists in some settings review images that AI systems have pre-read. Evaluators of language models judge outputs side by side, often with the help of an AI system that summarises or critiques them. In each case, the model's output becomes the context in which human judgment is exercised, and the resulting judgments become training or evaluation data. There are good reasons for this shift. Models can draft faster than people, and reviewing is often less tiring than creating from scratch. AI assistance can genuinely improve human evaluation of difficult outputs, as research on scalable oversight has explored. But the cognitive consequences are predictable. Reviewers are anchored on what the model proposed. They are subject to automation bias, more likely to accept suggestions than to generate alternatives. Their attention is directed by the model's flags, so errors the model did not flag are less likely to be caught. Over successive rounds, the dataset becomes a negotiation between the model and its reviewers, in which the model always speaks first. Content moderation queues show the dynamic clearly. A classifier ranks posts by predicted violation, and human moderators review the top of the queue. Moderators see mostly the kinds of content the classifier already flags. Their decisions on those items become training data. Content the classifier systematically misses, perhaps a newer form of coded harassment or abuse in an underrepresented language, rarely reaches a moderator, generates few labels, and remains missed. Meanwhile, the prevalence effect described in Chapter 2 operates on the moderators: a queue concentrated on likely violations raises their expectation of violation and lowers their threshold for borderline cases. The classifier's view of what counts as a violation becomes, through the moderators, the training data's view. Loops through generation: when models learn from models The newest and potentially most consequential loop arises from generative AI. Large language models and image generators produce content at enormous scale, and much of that content is published on the web, which is the main source of training data for the next generation of models. Some is generated deliberately as synthetic training data. Either way, future models are increasingly trained on the outputs of past ones. Ilia Shumailov and colleagues, in a paper published in Nature in 2024, "AI models collapse when trained on recursively generated data", showed that when models are trained repeatedly on data generated by previous models, without sufficient fresh human data, they progressively lose information about the tails of the original distribution. Rare events, unusual styles and minority patterns are underrepresented in each generation's output and therefore increasingly underrepresented in the next generation's training data. Eventually the model converges on a narrow, distorted version of the original distribution. Later work suggested that the collapse can be avoided if synthetic data is accumulated alongside, rather than replacing, the original human data, but the underlying tendency to lose the tails is not in dispute. The connection to cognitive bias is direct. A model trained on human-labelled data carries its labellers' biases. The content it generates reflects those biases: the views that labellers rewarded, the styles they preferred, the prototypes they matched. If that content then enters training data for future models, the biases are reinforced, and the minority perspectives that labellers outvoted or overlooked become rarer still. The survivorship of Chapter 4, in which only some content survives curation, combines with the halo and sycophancy of Chapter 5, in which certain styles are rewarded, and the narrowing of model collapse. The tails that disappear first are the ones that were already thin. The same loop operates through human judgment in reinforcement learning from human feedback. Raters judge model outputs, a reward model learns their preferences, the policy model is optimised toward the reward model, and the next round of raters judges outputs from the optimised policy. If raters have a systematic tendency, such as preferring confident or agreeable answers, each round pushes outputs further in that direction, and raters, anchored by what they are shown, may gradually come to accept as normal a style that would once have seemed excessive. Sycophancy in AI assistants is, in this light, an emergent property of a loop that repeatedly rewards a human preference for being agreed with. Loops through belief: people who learn from the machine There is one more loop, quieter than the others and perhaps the most important for the long run. People do not only use AI outputs; they learn from them. A doctor who works for months alongside a diagnostic aid, a moderator who sees which posts a classifier flags, a student who consults a chatbot, all absorb something about what the system treats as normal, risky, relevant or correct. If the system carries the biases of its labellers, people can acquire those biases from it, and they then carry them into their own future judgments, including judgments that become training data. This possibility has been tested directly. Lucía Vicente and Helena Matute, in a 2023 paper in Scientific Reports titled "Humans inherit artificial intelligence biases", asked participants to perform a simulated medical classification task, judging whether tissue samples showed signs of a fictitious disease. Some participants were assisted by a supposed AI system that made a systematic error on a particular kind of sample. Participants who worked with the biased assistant reproduced its error, which was expected. The more striking finding was that they continued to make the same error after the assistance was withdrawn and they were classifying on their own. Participants who had never seen the biased assistant did not make it. The bias had been transferred from the system to the people, and it outlasted the interaction. Moshe Glickman and Tali Sharot reported a related and more troubling pattern in a study published in Nature Human Behaviour at the end of 2024, "How human–AI feedback loops alter human perceptual, emotional and social judgements". They first collected human judgments on tasks such as deciding whether a set of faces looked more happy or sad, in which people showed a slight systematic bias. They trained an AI system on those judgments and found that the system amplified the bias: a small human tendency became a larger machine tendency, consistent with the amplification mechanisms described in Chapter 6. They then had new participants interact with the system and found that participants' own judgments became more biased as a result, and more so than when they interacted with other humans showing the same bias. Participants tended to underestimate how much the AI was influencing them. The complete loop, human bias to machine amplification to greater human bias, was observed in a single set of experiments. Both studies were run in controlled laboratory settings, with simple tasks and fictitious stakes, and their effect sizes should not be extrapolated directly to clinical or judicial practice. But their mechanism is entirely consistent with what is known about anchoring, automation bias and social learning. People calibrate their judgments against the sources around them, and an AI system that speaks with consistency and apparent authority is a powerful source. Where those people are also the annotators, reviewers or raters whose judgments train the next model, the loop is closed through human belief itself. The annotators of the future will have grown up with the models of the present, and their expectations, the raw material of confirmation bias, will have been shaped by what those models told them was normal. Do loops always amplify? Not every feedback loop runs away. Loops can be damped, as Ensign's analysis of predictive policing showed, by correcting for the system's influence on its own data. Exploration, in which a system deliberately tries options it would not normally choose, generates data about the cases it would otherwise never see. Holding out a random sample of decisions from the model's influence provides a clean comparison. Accumulating fresh human data alongside synthetic data limits collapse. And human judgment, under the right conditions, can correct a model rather than merely echo it. The conditions matter. Loops amplify when the human judgments feeding back into the system are anchored on the system's outputs, primed by its predictions, or restricted to the cases it chose to show. Loops are damped when human judgment is exercised independently, on a sample the model did not select, by people who were not told the model's view first. In other words, the same task design choices that limit the cognitive biases of Chapters 2 to 5 are what prevent feedback loops from compounding them. Table 4 summarises the loops discussed in this chapter, the cognitive effect at the heart of each, and the standard means of damping it. Table 4. Feedback loops and their damping mechanisms. Loop Path of influence Key human effect Damping mechanism Through the world Decisions shape observed outcomes Confirmation in observers Correct for allocation; explore Through attention Rankings shape what users choose Position and availability Randomised exposure; debiased clicks Through judgment Outputs anchor reviewers Anchoring, automation bias Blind first judgment; random audit Through generation Outputs become training data Halo, sycophancy, survivorship Retain human data; track provenance The loop as the real object of design The practical implication of this chapter is that the object of design is not the model, nor the dataset, but the loop. A model trained once on a fixed dataset has a fixed set of biases, which can at least in principle be measured. A model embedded in a loop that feeds its outputs back into its future training data has biases that move, and that move in a direction determined by the interaction of its own errors with the cognitive tendencies of the people around it. Auditing such a system once is not enough. It must be monitored over time, with attention to which data it is generating for itself, and with deliberate injections of independent judgment to keep it honest. This is demanding, and most organisations do not yet do it. The final chapter sets out what doing it looks like. Chapter 8: Designing Better Judgment If the argument of this booklet is right, the most effective place to reduce bias in AI systems is not at the end of the pipeline, where a trained model is audited and patched, but at the beginning, where human judgments are turned into data. The biases that matter are properties of judgment processes, and judgment processes can be designed. The psychology that identifies the biases also identifies the conditions under which they weaken. This chapter sets out what that design involves. It is organised by stage: the framing of the task, the interface in which judgments are made, the people who make them, the way their judgments are combined, the documentation that records the process, the measurement that checks it, and the governance that keeps it honest over time. None of the techniques is exotic. Most are borrowed from fields that confronted human judgment error long before machine learning did: survey research, forensic science, clinical trials, aviation and auditing. What is new is the need to apply them systematically to the production of training data, which has too often been treated as a procurement problem rather than a measurement problem. Start with the question The first decision, and the most consequential, is what question annotators will be asked. Chapter 5 showed that framing shapes answers, and Chapter 2 that the task description embeds a hypothesis. Good task design begins by asking whether the question can actually be answered from the material annotators will see. Many of the worst failures documented in this booklet arose from asking people to judge things that could not be seen. ImageNet's crowdworkers were asked to confirm whether photographs depicted occupations and character traits that are not visible in a photograph. Emotion recognition datasets ask annotators to judge internal states from external configurations that do not reliably indicate them. When the evidence is insufficient, judgment is completed by expectation, prototype and stereotype. The most effective intervention is often to change the question: to ask what is observable ("does the person appear to be smiling?") rather than what is inferred ("is the person happy?"), or to decline to build the dataset at all. Where the question is genuinely about a contested social property, such as offensiveness or political bias, the design should acknowledge the contest. That may mean asking annotators for their own judgment and, separately, their estimate of how others would judge, a technique that Hube and colleagues found reduced the influence of personal opinion. It may mean asking annotators to identify which features of the item drove their judgment, which makes their reasoning available for later analysis. It may mean defining the category through several narrower questions, such as whether the text contains an insult, whether it targets a group, and whether it uses a slur, rather than one broad one. Narrow questions leave less room for a general impression to substitute for an assessment. Guidelines should be tested before they are used. Survey researchers pilot their questionnaires and run cognitive interviews, in which respondents think aloud while answering, to discover how questions are actually understood. The same practice applied to annotation guidelines reveals ambiguities, unintended framings and prototypes before they reach thousands of labels. Worked examples in guidelines deserve particular scrutiny, because they become the prototypes against which annotators match. They should span the category, including its unusual members, and should include near-misses that show where the boundary lies. Design the interface against the anchor The interface is where anchoring and automation bias operate, and it is the place where the designer has most control. The single most effective technique is to collect an independent judgment before revealing any suggestion. The annotator first labels the item unaided; only then is a model's pre-annotation, if any, shown, and the annotator is allowed to revise. This is the annotation equivalent of linear sequential unmasking in forensic science, in which examiners document their analysis of the evidence before being exposed to potentially biasing context. It costs some of the speed that pre-annotation was meant to provide, but it produces two valuable things: an unanchored label, and a record of how often and in what direction annotators change their minds when shown the model's view. That record is a direct measure of the anchor's strength. Where full independence is too expensive, it can be applied to a random sample. A small fraction of items labelled without pre-annotation, compared with matched items labelled with it, reveals how much the suggestion is shifting the labels. If the shift is large, the pre-annotated data cannot be treated as an independent check on the model that produced the suggestions. A complementary technique is to insert known errors. If some pre-annotations are deliberately wrong, and the rate at which annotators catch them is measured, the team learns whether annotators are actually reviewing or merely accepting. The same principle, sometimes called a catch trial or gold question, is standard in crowdsourcing quality control, but it is usually used to filter out careless workers rather than to measure the effect of the interface. Using it for the latter is a small change with a large payoff. Other interface details matter as well. Context that is not needed for the judgment, such as the source of an item, the identity of its author, prior flags or model confidence scores, should be hidden unless there is a specific reason to show it. Rating scales should not start from a default value that pulls answers in one direction. In paired comparisons, the order of the two items should be randomised and the randomisation checked. The order of items in a queue should be randomised with respect to properties that might correlate with the labels, so that sequential effects become noise rather than bias. Sessions should be limited in length, with breaks, and annotators should receive periodic feedback on their accuracy against carefully adjudicated items, which counters drift and shortcut reliance. Choose the people deliberately Chapter 5 showed that annotators' identities and beliefs shape their judgments of social content. That makes the composition of the annotator pool a design decision with direct consequences for what the model learns. The minimum standard is to know who the annotators are. Many datasets in wide use record nothing about the people who labelled them, which makes it impossible to say whose perspective the labels represent. Collecting basic, consented, privacy-respecting information about annotators' backgrounds, relevant to the task, is the precondition for everything else. For a toxicity dataset, the relevant characteristics might include language, region and membership of groups often targeted by the content in question. For a medical dataset, they might include specialty and years of experience. The second standard is to match the pool to the task. Content written in a dialect should be labelled, at least in part, by people who speak it. Judgments about whether content is harmful to a group should include members of that group. Sap's 2019 finding that dialect priming reduced bias suggests that knowledge is part of the remedy, but there is no substitute for annotators who share the frame of the content's authors. Where the right annotators are hard to recruit, that difficulty should be recorded and reflected in how the data are used. The third standard concerns working conditions. The conditions under which cognitive biases flourish, time pressure, fatigue, repetitive high-volume decisions, low pay that rewards speed over care, exposure to distressing material without support, are in many cases the default conditions of commercial annotation. Improving them is a matter of fairness to workers, and it is also a matter of data quality. A team that pays per item at a rate that only rewards fast clicking has bought a dataset shaped by heuristics. Keep the disagreement The aggregation stage is where the evidence of bias is usually destroyed. The remedy is to stop destroying it. Individual annotations should be stored, linked to annotator identifiers and to whatever information about the annotators has been collected. Datasets should be released, where privacy permits, with the full distribution of labels rather than only the majority. Models can then be trained on soft labels, as in Peterson's CIFAR-10H work, or on multi-annotator objectives that predict each annotator's judgment, as Davani and colleagues proposed. These approaches let the model represent uncertainty where people genuinely disagree, and let analysts examine whether disagreement follows lines of identity or perspective. Where a single label is genuinely needed, for instance because a moderation system must take one action, the choice of how to combine judgments should be explicit and justified. Majority vote is one policy among several. A policy might give extra weight to annotators from the group most affected by the content, or require consensus before a severe action, or route contested items to expert adjudication. The point is not that there is one correct rule but that the rule is a decision about whose judgment counts, and it should be made deliberately rather than by default. When the annotator is a model A growing share of labels is now produced not by people but by language models prompted to act as annotators or judges. The practice is attractive for obvious reasons: model annotators are fast, cheap, tireless and consistent. Fabrizio Gilardi, Meysam Alizadeh and Maël Kubli reported in 2023, in Proceedings of the National Academy of Sciences, that a general-purpose model outperformed crowdworkers on several text annotation tasks, measured against labels from trained research assistants, at a fraction of the cost. Model annotators remove some human biases, such as fatigue and mood, but they do not escape the argument of this booklet. They were trained on human judgments, so they carry the residue of the biases discussed in earlier chapters. And they display systematic tendencies of their own. Lianmin Zheng and colleagues, in a 2023 study of language models used to judge other models' responses, documented a preference for the response shown in a particular position, a preference for longer responses, and a tendency to favour outputs resembling their own. These are recognisably the anchoring, halo and in-group effects of human raters, reproduced in a different medium. The design principles therefore carry over. A model annotator's labels should be validated against an independent human reference on a sample, broken down by subgroup. Position and order should be randomised and the randomisation checked. The model's labels should not be shown to human reviewers before they form their own view, or the human review becomes an anchoring exercise. And the dataset's documentation should record which labels came from models, which from people, and which from people working with models, because these are different measurement processes with different failure modes. Document the process A dataset that does not record how it was made cannot be audited for the biases this booklet describes. Several documentation frameworks now exist to address this. Timnit Gebru and colleagues proposed Datasheets for Datasets in 2018 (published in Communications of the ACM in 2021), a set of questions covering motivation, composition, collection process, preprocessing, uses, distribution and maintenance. Emily Bender and Batya Friedman's 2018 Data Statements for Natural Language Processing focused on the characteristics of speakers and annotators. Margaret Mitchell and colleagues' Model Cards (2019) extended the idea to trained models, including performance broken down by relevant subgroups. For the purposes of tracking cognitive bias, the critical fields concern the judgment process: the exact wording of the task and guidelines, including worked examples; the interface, including whether and when pre-annotations or context were shown; the order in which items were presented; who the annotators were and how they were recruited and paid; how many annotations each item received and how they were combined; and what selection processes determined which items were included. That last item is the survivorship question from Chapter 4: which cases are absent, and what decided their absence. Documenting it requires talking to the people who ran the earlier process, which is why it is so often skipped. Table 5 draws these practices together by stage, showing the bias each primarily targets. Table 5. Design practices for reducing cognitive bias in training data. Stage Practice Primary target Task framing Ask about the observable; pilot guidelines Representativeness, framing Interface Blind first judgment; hide needless context Anchoring, confirmation Interface Known-error checks; randomised order Automation bias, sequence effects Annotator pool Record and match backgrounds; fair pay Shared perspective, fatigue Aggregation Keep individual labels; train on distributions Majority erasure Selection Document what is missing and why Survivorship Deployment Hold-out samples; exploration; monitoring Feedback loops Measure what the labels are doing Good process design reduces bias but cannot guarantee its absence. Measurement is needed to find what remains. Several kinds of measurement are useful. Agreement statistics, broken down by annotator group, reveal whether disagreement follows lines of identity or belief. Comparisons between labels collected under different conditions, with and without pre-annotation, with different framings, in different orders, quantify the influence of the design on the data. Re-annotation of a sample by a different pool, or by experts using a more careful protocol, provides an independent reference. Probing a trained model for shortcut features, by testing whether it relies on dialect markers, length, hospital tokens or other surface cues, shows whether annotator heuristics have been learned. Evaluating models on test sets labelled by a different process from the training data breaks the circularity discussed in Chapter 6. And breaking all of these results down by subgroup is what turns an average into an audit. Tools for detecting label errors, such as the confident learning methods developed by Northcutt and colleagues, help find individual mislabelled items. They are good at finding noise and at finding items where the model and the label disagree. They are less suited to finding bias, because a bias consistent enough to be learned by the model will not produce disagreement between model and label. Detecting systematic bias generally requires an external reference: a different set of judges, a different protocol, or knowledge of the intended concept that the original labelling did not capture. Govern the loop Finally, systems in deployment need monitoring for the feedback loops described in Chapter 7. The practical measures are well understood in principle. Maintain a random sample of decisions made without the model's influence, or with deliberate exploration, to generate data about cases the model would otherwise never see. Track the provenance of training data, distinguishing data generated independently of the model from data generated under its influence, including model-assisted labels and model-generated content. Monitor the distribution of model outputs over time for narrowing or drift. Schedule periodic independent re-annotation of samples from production, using a protocol designed against anchoring. Regulation is beginning to require some of this. The European Union's AI Act, adopted in 2024, requires providers of high-risk AI systems to subject training, validation and testing data to data governance practices that include examination of possible biases, and to take measures to detect, prevent and mitigate them. The Act's high-risk obligations were scheduled to apply from August 2026, and in late 2025 the European Commission proposed delaying parts of that timetable, so organisations should check the current position for their sector. In the United States, the National Institute of Standards and Technology's Special Publication 1270, Towards a Standard for Identifying and Managing Bias in Artificial Intelligence (2022), explicitly identifies human cognitive biases among the categories of AI bias, alongside systemic and statistical biases, and its AI Risk Management Framework (2023) treats bias as a risk to be governed across the system's life cycle. These frameworks are useful as prompts, but they are not specific about annotation processes, and compliance with them can be achieved with documentation that says little about the judgment conditions under which labels were made. The practices in this chapter are what give such documentation content. The cost objection Every practice described here costs something. Blind first judgments are slower than corrections of pre-annotations. Diverse annotator pools are harder to recruit. Storing individual labels requires more infrastructure. Exploration in deployment means making some decisions the model considers suboptimal. Fair pay raises the price of each label. The honest answer is that these costs are real, and that they buy something real. A dataset produced cheaply under conditions that maximise cognitive bias is not cheap; its costs are deferred to the people affected by the model's decisions, to the organisation's reputation when the bias is discovered, and to the engineers who spend months trying to diagnose a model's behaviour without the information that would explain it. Organisations routinely spend large sums on compute and model architecture to obtain improvements that are small compared with what better-designed human judgment could provide. The judgment process is the cheapest place in the pipeline to fix the problem, and the most expensive place to ignore it. Conclusion The story of biased AI is often told as a story about machines: opaque algorithms, inscrutable weights, systems that have somehow absorbed prejudice from the internet. This booklet has told it as a story about people. Not about bad people, but about ordinary minds doing ordinary cognitive work under conditions that make that work systematically unreliable, and about a technology that converts the systematic part of that unreliability into rules applied at scale. The argument can now be stated more sharply than it could at the start. Machine learning systems are trained to reproduce their labels, and labels are compressed records of human judgments. Those judgments are shaped by the expectations annotators bring and are given, by the suggestions placed in front of them, by the selection processes that determine which cases reach them, and by a set of further tendencies, availability, representativeness, halo, framing, sequence and perspective, that psychology has documented for decades. Each of these produces regularities in the data. A learning algorithm cannot distinguish a regularity of the world from a regularity of the labeller's mind, and when the latter is easier to compute, it learns that one first. Aggregation removes the evidence of disagreement, evaluation on similarly labelled data certifies the result, and feedback loops return the model's outputs to the next round of human judgment. What began as a tilt in a few thousand clicks ends as a pattern in millions of decisions. What follows Several conclusions follow that would not have been obvious without tracing the whole path. The first is that representation is necessary but not sufficient. Much effort in responsible AI has gone into ensuring that datasets include enough examples of different groups. That effort matters, but a dataset can be balanced in who appears and still systematically wrong in what is said about them. The dialect bias in toxicity datasets was not a failure to include Black users' tweets; it was a failure in how those tweets were judged. Fixing it required changing the judgment process. The second is that agreement is not evidence of truth. The industry's reliance on inter-annotator agreement as a measure of label quality is a reliance on a tool that detects noise and misses bias. When annotators share expectations, frames, prototypes and anchors, they agree with one another, and they are wrong together. A high agreement score should prompt the question of who was in the pool and what they had in common. The third is that model-assisted annotation is a double-edged tool that is rapidly becoming the default. As more training data is produced by humans correcting, ranking and approving machine outputs, anchoring and automation bias become central to what the next generation of models learns. Without blind first judgments, random independent samples and measurement of how much suggestions shift labels, the industry risks building models whose training data are largely their predecessors' views with a human signature attached. The fourth is that the right unit of design is the loop, not the dataset. A model that shapes its own future data cannot be audited once and declared fair. It must be monitored, its data provenance tracked, and independent human judgment deliberately injected to prevent its errors and its labellers' biases from compounding. What remains unsettled Several important questions remain open, and readers should treat them as such. The most fundamental is the question of whose judgment should count. Once one accepts that many labelling tasks have no single correct answer, the choice of annotators, aggregation rules and adjudication procedures becomes a choice about values, not only about accuracy. Preserving disagreement and modelling individual perspectives are technically promising, but a deployed system usually has to act, and an action implies a decision about which perspective prevails. There is no technical resolution to that problem. It requires the kind of deliberation societies use for other contested questions, and the institutions for conducting it in the context of AI systems are still immature. A second open question concerns the growing role of AI systems as judges. Language models are now widely used to label data, evaluate other models and provide feedback in training. They are cheap, fast and consistent. They also have systematic tendencies of their own, including preferences for their own outputs, for longer responses and for particular positions in a list, which researchers have documented in studies of models used as evaluators. Whether replacing human judges with model judges reduces cognitive bias, by removing fatigue and mood, or entrenches it, by fixing one set of learned human tendencies in place and removing the variability that might have corrected them, is not yet clear. The likely answer is that it depends heavily on design, and the principles in this booklet apply: independent judgment, measured disagreement, known-error checks and documented provenance. A third concerns measurement itself. Detecting systematic bias requires an external reference, and for many tasks no definitive reference exists. Experts disagree, outcomes are selectively observed, and the concepts themselves are contested. Progress will depend on better methods for triangulating between imperfect references, and on honesty about where the uncertainty is irreducible. What to do differently For people who build machine learning systems, the practical message is to treat the production of labels with the same seriousness as the training of models. Ask what question annotators were asked and whether it could be answered from what they saw. Find out who the annotators were. Look at the interface. Check whether a model's suggestion was on screen before the human decided. Keep individual annotations. Ask what cases are missing and why. Evaluate on data produced by a different process. Monitor the loop. For people who buy, regulate or are affected by such systems, the message is to ask the same questions of those who build them, and to regard an inability to answer them as a material finding. A vendor that cannot say how its training labels were produced cannot say what its system has learned. For everyone, the message is more general. The habits of mind that psychologists have catalogued, our tendency to see what we expect, to start from what we are given, to reason from what survived, to judge by what comes easily to mind, were once limitations of individual judgment, important but bounded by the reach of one person's decisions. Machine learning removes that bound. It takes the systematic part of our judgment, strips away the context and the doubt, and applies it with confidence to everyone. That is a reason for humility about our own judgments, and a reason to design the processes that capture them with far more care than we have so far. The machines are learning from us. It would be wise to make sure that what they learn is our considered judgment, and not merely our habits. Further Reading The psychology of judgment Daniel Kahneman, Thinking, Fast and Slow (2011).** The most accessible synthesis of the heuristics-and-biases research programme, including anchoring, availability, representativeness and framing, by one of its founders. Read alongside the caveats that later replication work raised about some of the priming studies it discusses. Daniel Kahneman, Olivier Sibony and Cass R. Sunstein, Noise: A Flaw in Human Judgment (2021).** The clearest treatment of the difference between bias and noise in professional judgment, with practical advice on "decision hygiene" that translates directly to annotation design. Raymond S. Nickerson, "Confirmation Bias: A Ubiquitous Phenomenon in Many Guises", Review of General Psychology (1998).** The standard review of confirmation bias, still the best single source on its many forms. Amos Tversky and Daniel Kahneman, "Judgment under Uncertainty: Heuristics and Biases", Science (1974).** The short paper that introduced anchoring, availability and representativeness to a wide audience. Remarkably readable. Raja Parasuraman and Dietrich H. Manzey, "Complacency and Bias in Human Use of Automation: An Attentional Integration", Human Factors (2010).** The key review of automation bias, essential for anyone designing model-assisted annotation. Human judgment in machine learning data Mary L. Gray and Siddharth Suri, Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass (2019).** An ethnographic account of the people who label data and the conditions under which they work. Lora Aroyo and Chris Welty, "Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation", AI Magazine (2015).** The founding argument that annotator disagreement is signal rather than noise. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi and Noah A. Smith, "The Risk of Racial Bias in Hate Speech Detection", Proceedings of ACL (2019), and Maarten Sap and colleagues, "Annotators with Attitudes: How Annotator Beliefs and Identities Bias Toxic Language Detection", Proceedings of NAACL (2022).** Together, the clearest demonstrations of how annotator expectations and beliefs enter toxicity data. Suchin Gururangan and colleagues, "Annotation Artifacts in Natural Language Inference Data", Proceedings of NAACL (2018).** A compact, influential study of how crowdworkers' strategies became shortcuts that models learned. Barbara Plank, "The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation", Proceedings of EMNLP (2022).** A survey of why human labels vary and what to do about it. Kate Crawford, Atlas of AI (2021).** Includes a critical account of the construction of classification schemes and image datasets, including the ImageNet person categories. Selection, feedback and consequences Himabindu Lakkaraju and colleagues, "The Selective Labels Problem: Evaluating Algorithmic Predictions in the Presence of Unobservables", Proceedings of KDD (2017).** The formal statement of survivorship bias in decision data. Ziad Obermeyer, Brian Powers, Christine Vogeli and Sendhil Mullainathan, "Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations", Science (2019).** A model case study of how a proxy label carries a history of human decisions. *Danielle Ensign and colleagues, "Runaway Feedback Loops in Predictive Policing", Proceedings of FAT\ (2018). A short, clear formalisation of how a model's decisions shape its own future data. Ilia Shumailov and colleagues, "AI Models Collapse When Trained on Recursively Generated Data", Nature (2024).** The key study of what happens when models learn from the outputs of models. Mrinank Sharma and colleagues, "Towards Understanding Sycophancy in Language Models" (2023; published at ICLR 2024). How human preference judgments train AI assistants to tell people what they want to hear. Practice and governance Timnit Gebru and colleagues, "Datasheets for Datasets", Communications of the ACM (2021).** The most widely adopted framework for documenting how a dataset was made. Reva Schwartz and colleagues, Towards a Standard for Identifying and Managing Bias in Artificial Intelligence, NIST Special Publication 1270 (2022). A government framework that explicitly treats human cognitive bias as a category of AI bias. Robert Geirhos and colleagues, "Shortcut Learning in Deep Neural Networks", Nature Machine Intelligence (2020).** An accessible account of why models learn the easiest regularity in the data, including those produced by annotators. Hashtags: #AIAndCognitiveBiases #CognitiveBiases #ArtificialIntelligence #MachineLearning #HumanCognition #HumanJudgment #HumanBias #AlgorithmicBias #AIBias #BiasInMachineLearning #BiasedTrainingData #DataBias #DatasetBias #HumanFlawsInAI #MachineLogic #CognitivePsychology #PsychologyOfJudgment #DecisionMaking #BehavioralScience #HumanCenteredAI #ResponsibleAI #EthicalAI #AIEthics #AlgorithmicFairness #AIAccountability #AITransparency #AITrustworthiness #ConfirmationBias #AnchoringBias #SurvivorshipBias #AvailabilityHeuristic #RepresentativenessHeuristic #HaloEffect #FramingEffect #AutomationBias #SelectionBias #SamplingBias #StereotypeBias #ImplicitBias #JudgmentBias #AnnotationBias #HumanAnnotation #DataAnnotation #DataLabeling #TrainingDataQuality #LabelQuality #LabelNoise #SystematicBias #GroundTruth #AnnotatorDisagreement #InterAnnotatorAgreement #CrowdsourcedAnnotation #HumanInTheLoop #ModelAssistedLabeling #PreAnnotation #BiasAmplification #FeedbackLoops #AlgorithmicFeedbackLoops #SelectiveLabels #DatasetCuration #DataSelection #DataProvenance #DatasetDocumentation #MachineLearningDatasets #AlgorithmicDecisionMaking #BiasDetection #BiasMitigation #DebiasingStrategies #FairMachineLearning #AIModelEvaluation #BenchmarkBias #NaturalLanguageProcessing #ToxicLanguageDetection #ContentModeration #PredictivePolicing #HiringAlgorithms #CreditScoring #HealthcareAlgorithms #MedicalAIBias #RecommenderSystems #LargeLanguageModels #HumanFeedback #RLHF #PreferenceModeling #AIAlignment #AIGovernance #ResponsibleDataPractices #AIQualityAssurance
Latest Book Releases:










































