Download the Book (PDF): Introduction A trained musician who hears thirty seconds of an unfamiliar recording can usually say a surprising amount about it. Roughly how fast it goes and where the downbeats fall. Whether it is in a major or minor key, and when it modulates. Which instruments are playing, and which of them carries the tune. Whether the singer is pushing against the beat or sitting behind it. Whether this is the verse or the chorus, and how many times the chorus is likely to come round again. Often, what tradition the piece belongs to and roughly when it was made. None of this feels like effort. It feels like simply listening. Music information retrieval, usually shortened to MIR, is the field that tries to make a computer do the same thing, and then to use what it can do. The name is a historical accident. The field coalesced around 2000, when the first International Symposium on Music Information Retrieval met in Plymouth, Massachusetts, and its founders were largely librarians, musicologists and signal-processing engineers interested in a concrete problem: how to search large collections of music by its content rather than by the words someone had typed about it. "Retrieval" stuck, though the field long ago outgrew it. Today MIR covers transcribing audio into notes, tracking beats and chords, identifying recordings, recommending songs, pulling a mixed recording apart into its instruments, and, increasingly, generating new music altogether. This booklet is about that whole arc, from analysis to generation, and it argues one thing throughout. Every system that analyses or generates music embodies a theory of what music is, and the quality of the system is bounded by the quality of that theory. The theory is rarely stated. It hides in the choice of input representation, in the labels of the training dataset, in the metric used to declare success, and in the vocabulary of tokens a generator is allowed to emit. When a genre classifier reaches high accuracy by learning the frequency response of the studios a dataset happened to come from, its implicit theory was that genre lives in the spectrum, and that theory was wrong. When a transcription model writes a perfectly accurate list of note onsets that no pianist could read, its implicit theory was that a score is a list of notes, and that theory is incomplete. When a generator produces something fluent in a style but quietly reproduces a recognisable passage from its training data, its implicit theory of style has collapsed into a theory of memory. The practical consequence is that the musicological questions never went away when deep learning arrived. They moved. A decade ago a researcher designing a chord-recognition system had to decide by hand how to fold the spectrum into twelve pitch classes and how to smooth chord labels over time. Today a neural network learns those mappings, but someone still decides what counts as a chord, whose annotations are correct when two experts disagree, whether "C major with an added ninth" is a different answer from "C major", and how wrong a system should be considered when it says A minor instead of C major. These are music-theoretic decisions with direct engineering effects. The best work in the field has always been done by people who could see both sides. What this booklet covers The book moves in a deliberate order. It begins with representation, because everything downstream depends on it: what a waveform, a spectrogram, a chromagram and a piano-roll each preserve and discard, and why the move from audio to symbols is the hardest single step in the field. It then turns to ground truth, the datasets and annotations that every learned system is shaped by, and to the evaluation habits that decide what counts as progress. These two chapters are the foundation for everything after them, and readers who skip them will find the later chapters less persuasive. The middle of the book covers the major analytical tasks. Automatic music transcription, the attempt to turn sound into notes, gets a chapter of its own because it is the problem on which much of MIR depends and the one where progress has been most dramatic for some instruments and most stubborn for others. A second analytical chapter covers time and harmony together: beat and tempo tracking, chord and key recognition, and the analysis of musical form, all of which are attempts to recover the structures a listener perceives above the level of individual notes. Genre classification follows, as the field's most famous and most instructive case study in how a model can appear to succeed while learning nothing musical. Source separation, the unmixing of a stereo recording into vocals, drums, bass and the rest, closes the analytical half. It is the task where deep learning most clearly changed what working musicians and engineers can do, and its story illustrates how a shared benchmark can drive a decade of rapid progress. The final third turns to generation. One chapter traces symbolic composition, from rule-based systems of the 1950s through statistical models to transformers that generate sequences of notes. A second covers the newer and more consequential development: models that generate audio directly, using neural codecs, language-model architectures and diffusion. The last chapter addresses the problem in the subtitle of the field's commercial present, composing within stylistic constraints, which turns out to be inseparable from the questions of control, evaluation, memorisation and provenance that now dominate public argument about AI music. Scope and assumptions The reader is assumed to be curious and literate in either music or computing but not necessarily both. Musical terms are explained where they first matter, and so are technical ones. No mathematics beyond the idea of a frequency, a probability and an average is needed to follow the argument, though some passages describe mechanisms in enough detail that a technical reader could reconstruct them. The field is overwhelmingly built on Western popular and classical music, and in particular on music that uses twelve-tone equal temperament, a regular metre and a vocabulary of functional chords. That bias shapes the datasets, the label vocabularies and the assumptions baked into common tools, and it surfaces repeatedly in the chapters that follow. Where the book discusses it, the aim is not ritual acknowledgement but accuracy: a beat tracker built for four-on-the-floor dance music can fail badly on Carnatic tala or on the rubato of a solo piano nocturne, and knowing why is part of understanding what the tracker actually learned. Specific songs and pieces are named where they illustrate a point, but the book never reproduces lyrics or writes out melodies note by note. Musical material is described rather than quoted, which is in any case closer to how a computer encounters it: as structure, not as text. Finally, a note on currency. The generative side of the field has moved very fast. As of October 2026, several commercial music generators produce complete songs with vocals from a short text prompt; major record labels have both sued and licensed some of those companies; a German court has ruled against one of them over training on protected works; and streaming services report that a large share of the tracks uploaded to them each day are machine-generated. Some of these facts will date quickly. The underlying questions they raise, about what a model learned, whose work it learned from and what it is actually doing when it produces music "in a style", will not. Those questions are the subject of this book, and they turn out to be the same questions MIR has been asking since the beginning. Chapter 1: What a Machine Hears Before a model can classify, transcribe, separate or generate music, someone has to decide what the model will be given to look at. That decision is easy to treat as plumbing. It is not. Every representation of music keeps some information and throws the rest away, and what it keeps quietly defines what the model can ever learn. A system that sees only pitch classes cannot tell a chord in close position from the same chord spread across four octaves. A system that sees a piano-roll cannot hear that a violinist leaned into a note. A system that sees raw audio sees everything, which is to say it must learn for itself which of those everythings matter. This chapter walks through the main representations in MIR, what each one assumes about music, and why the gap between the audio representations and the symbolic ones is the central difficulty of the field. Sound as a signal A recording is a sequence of numbers. A microphone converts fluctuations in air pressure into voltage, and an analogue-to-digital converter samples that voltage at a fixed rate and rounds each sample to a fixed precision. Compact disc audio, still the reference format for much music research, samples 44,100 times per second at 16 bits per sample, in two channels. Three minutes of stereo CD audio is therefore about sixteen million numbers. The sampling rate sets the highest representable frequency at half its value, 22,050 hertz for CD audio, which comfortably exceeds the upper limit of adult hearing. The waveform is complete in the sense that it can be played back. It is also nearly opaque. Looking at sixteen million amplitude values, a person cannot tell what notes are being played, and for most of the history of MIR neither could a model. The reason is that the musically meaningful content of a waveform lives in its periodicities. A violin playing the A above middle C produces a pressure wave that repeats about 440 times per second, plus components at whole-number multiples of that rate, the harmonics, whose relative strengths give the violin its colour. A single waveform sample says nothing about any of this. Periodicity only appears when you look at many samples together. So the first transformation almost every MIR system applies, or in modern end-to-end systems learns an equivalent of, is a move from time to time-frequency. The short-time Fourier transform cuts the waveform into short overlapping windows, typically between about 20 and 100 milliseconds long, and measures how much energy each window contains at each of a set of evenly spaced frequencies. Stack the results side by side and you have a spectrogram: a grid with time running one way, frequency the other, and energy as the value at each cell. Each window also yields a phase value for each frequency, which records where in its cycle each component is; most analysis systems discard phase and keep only magnitude, a choice that becomes important again when we reach source separation and generation. The spectrogram embodies its own compromise. A longer window resolves frequency more finely but smears events in time, so a drum hit appears to last longer than it does. A shorter window catches onsets precisely but blurs neighbouring pitches together. This is not an engineering limitation that better hardware will remove; it is a property of the transform, a version of the uncertainty relation between time and frequency. A low E on a bass guitar is about 41 hertz, and the next semitone up is about 44 hertz, so distinguishing them requires a window long enough to see several cycles of each, on the order of a few hundred milliseconds. A hi-hat pattern at a fast tempo needs resolution of a few tens of milliseconds. A single spectrogram cannot serve both well. Bending the frequency axis toward music The linear frequency axis of the plain spectrogram is musically awkward. Pitch perception is roughly logarithmic: each octave doubles the frequency, so the octave from 110 to 220 hertz spans 110 hertz while the octave from 1,760 to 3,520 hertz spans 1,760. A linear grid devotes most of its bins to the upper octaves, where there is less fundamental pitch information, and starves the lower ones. Two families of fix dominate. The mel spectrogram warps frequency according to a perceptual scale derived from listening experiments, roughly linear below about 1,000 hertz and logarithmic above it, and pools the linear bins into a smaller number of mel bands, often 64 or 128. It came to MIR from speech recognition and is the default input for a great many audio neural networks, including many genre classifiers and tagging systems. Its weakness for music is that mel bands are not aligned with musical pitches, so two notes a semitone apart may land in the same band. The constant-Q transform, introduced to music analysis by Judith Brown in 1991, takes the musical route directly. It spaces its frequency bins geometrically so that each octave gets the same number of bins, typically twelve or a multiple of twelve, aligning bins with the semitones of equal temperament. Its windows are long at low frequencies and short at high ones, which mirrors the way musical events behave: bass notes are sustained and need frequency precision, while upper partials and percussive transients need time precision. Many transcription and chord recognition systems use a constant-Q input, or a learned variant of it, for exactly this reason. The constant-Q transform carries a theory with it. It assumes that the semitone is the relevant unit of pitch and that octaves are equivalent in structure. For most Western music that is a good assumption. For Turkish makam music, whose theory divides the whole tone into nine commas, or for the microtonal inflections of Hindustani raga performance, a twelve-bins-per-octave grid literally cannot see the distinctions that matter. Researchers working on those traditions use finer grids, thirty-six or sixty bins per octave and more, or work from continuous pitch tracks rather than bins. The point is general: choosing the grid is choosing which differences count as musical. Folding pitch into harmony One step further from the waveform lies the chromagram, also called a pitch-class profile, first proposed in this form by Takuya Fujishima in 1999 for chord recognition. Take a pitch-aligned spectrogram and add together all the energy belonging to each of the twelve pitch classes, regardless of octave: every C, from the lowest on a piano to the highest, contributes to the single C value. The result is a twelve-dimensional vector per time frame that summarises which pitch classes are sounding. The chromagram is a deliberately lossy representation with a sharp theory behind it. It asserts that for harmony, octave does not matter, and that timbre should be ignored. Both assertions are roughly true for identifying chords and keys and roughly false for almost everything else. A C major triad voiced as a low C, a middle E and a high G produces nearly the same chroma vector as the same three notes stacked in close position, which is what you want for chord labelling. But a C major chord with E in the bass, a first inversion, has a different harmonic function in tonal music, and the chromagram cannot tell it apart either. Researchers who wanted inversions back had to compute a separate bass chromagram restricted to low frequencies, which is to say they had to restore some of the information the theory had thrown away. Chroma features also illustrate how representations entangle with the physics of instruments. A real piano note does not put its energy only on its fundamental pitch class. Its third harmonic sits an octave and a fifth above the fundamental, so a C produces significant energy in the G pitch class; its fifth harmonic, two octaves and a major third up, adds energy to E. A single sustained C therefore produces a chroma vector that looks a little like a C major chord. Early chord recognisers spent considerable effort on harmonic suppression to counteract this, and much of what neural networks learned when they replaced those hand-built pipelines was, in effect, a better model of how harmonics fold into pitch classes. The symbolic side At the other end of the spectrum from the waveform lies symbolic music: representations that describe notes rather than sounds. The most widely used is MIDI, a protocol defined in 1983 to let synthesisers and sequencers talk to each other. A MIDI file is a list of timed events, chiefly note-on and note-off messages that each carry a pitch number from 0 to 127 and a velocity indicating how hard the note was struck, plus control messages for things like the sustain pedal. MIDI was designed to drive instruments, not to describe music, and it shows. It has no notion of whether a pitch is a C sharp or a D flat, which is the same key on a piano but a different note in the harmonic logic of a score. It has no concept of a beam, a slur, a voice or a phrase. Its timing can be expressed in musical beats or in absolute time, and files drawn from real performances usually encode absolute time with no reliable beat grid at all. Notation formats sit further up. MusicXML, MEI (the Music Encoding Initiative format favoured by musicologists and digital editions) and the older Humdrum kern format all represent what a score represents: pitch spelling, metre, measures, voices, articulations, dynamics, lyrics. ABC notation, a compact text format developed for folk tunes, encodes a melody line in a way that reads almost like a shorthand transcription. Each of these formats makes it possible to ask questions MIDI cannot answer, such as which voice a note belongs to or whether a passage is notated in triple metre. The piano-roll is the symbolic representation most loved by machine learning, because it turns notes into an image. Time is quantised into frames, pitch into 88 or 128 rows, and each cell is on or off (or carries a velocity). It is simple, fixed-size and perfectly suited to convolutional networks. It also discards a great deal: the distinction between one long note and two repeated notes of the same pitch, unless onsets are encoded separately; every aspect of notation; and the voice structure that tells a performer which notes belong to which melodic line. The gap between sound and symbol The various representations differ along a few consistent axes, which Table 1 summarises: how much they preserve, what they assume about music, and what kinds of task they suit. Table 1. Principal music representations and their built-in assumptions. Representation Keeps Discards Typical uses Waveform Everything audible Nothing, but structure is implicit End-to-end separation, generation Linear spectrogram Energy by time and frequency Phase (usually) Separation masks, analysis Mel spectrogram Perceptually weighted energy Phase, fine pitch alignment Tagging, genre, general audio Constant-Q transform Energy on a semitone grid Phase, microtonal detail at 12 bins Transcription, chords Chromagram Pitch-class energy Octave, timbre, voicing Chords, key, cover-song matching MIDI / piano-roll Pitches, onsets, velocities Timbre, spelling, notation, voices Symbolic generation, transcription targets Score formats Notation, metre, voices, spelling Performance timing and sound Musicology, notation-level generation Read the table from top to bottom and you are moving from what a microphone captures toward what a composer writes. The top rows are complete but uninterpretable; the bottom rows are interpretable but incomplete in the opposite direction. Nothing in the middle provides a lossless bridge between them. Going from a recording to a score requires inferring things that are not literally in the signal, such as which notes belong to the melody, where the barlines fall and how a pitch should be spelled. Going from a score to a recording requires inventing things the score never specified, such as exact timing, timbre, balance and expression. This gap explains a great deal about the shape of the field. Transcription, the subject of a later chapter, is the attempt to cross it from the audio side. Most of the analysis tasks that follow it, from beat tracking to chord recognition, are partial crossings that recover one layer of symbolic structure without attempting the whole. Symbolic music generation stays entirely on the right-hand side and leaves the rendering to a synthesiser or a human. Audio generation, the newest development, sidesteps the gap by generating sound directly, which buys realism at the cost of the explicit structure that made symbolic systems controllable. Learned representations The deep-learning era added a third kind of representation: the learned embedding. Instead of designing a transform, train a neural network on some task and take its internal activations as the representation. A network trained to predict tags such as "female vocal", "distorted guitar" or "jazz" from millions of songs develops internal features that are useful for many other tasks, and those features can be extracted as compact vectors and reused. This approach, known as transfer learning, became standard in music tagging and recommendation in the late 2010s. Self-supervised learning pushed the idea further. A model such as MERT, published in 2023 by a group of academic researchers, is trained on large amounts of unlabelled music to predict masked portions of its own input, guided by acoustic and pitch-related targets. Contrastive models such as CLAP learn to place audio clips and text descriptions near each other in a shared embedding space, so that the vector for a recording of a solo cello sits close to the vector for the phrase "solo cello". Neural audio codecs, which compress audio into sequences of discrete tokens, are a different kind of learned representation again, and they have become the foundation of modern audio generation. Learned representations have a peculiar epistemic status. They often outperform hand-designed features on benchmarks, sometimes by wide margins. But the theory of music they embody is not stated anywhere; it is whatever regularities in the training data were useful for the training objective. A tagging network trained on a commercial catalogue may encode production era and mastering loudness as strongly as harmony or rhythm, because those features genuinely predict tags in that catalogue. Whether they predict anything musical in a new context is an empirical question, and one that has to be answered by careful testing rather than by inspecting the representation. Representation as theory It is worth stating the lesson of this chapter plainly, because the rest of the book leans on it. A representation is not a neutral container. The chromagram is a theory that harmony is octave-invariant and timbre-free. The constant-Q transform is a theory that the semitone is the unit of pitch. MIDI is a theory that music is a set of keyboard events. A mel spectrogram is a theory, borrowed from speech, that perceptual loudness in roughly logarithmic bands is what matters. Even the raw waveform carries a theory, namely that the recording, as mixed and mastered, is the object of interest, rather than the performance, the composition or the cultural practice behind it. None of these theories is wrong in general. Each is right for some questions and wrong for others, and much of the craft of MIR lies in matching one to the other. When deep learning made it possible to learn representations end to end, it did not dissolve this problem; it displaced it into the choice of training data, labels and objectives. That is where the next chapter goes. Chapter 2: Ground Truth and Its Discontents Every learned MIR system is a compressed summary of the data it was trained on and the labels attached to that data. If the representation is the first place a theory of music enters a system, the dataset is the second, and usually the more consequential. Labels decide what the system is trying to predict. Their vocabulary decides what distinctions it can make. Their errors become its errors, and their biases become its biases. Evaluation then closes the loop: the metric decides which systems count as better, and therefore which ideas the field pursues. This chapter is about those three things, datasets, annotations and metrics, and why the most important progress in MIR has often come from improving them rather than from improving models. Why music data is hard to get Most machine-learning fields are limited by labelled data, but music has an unusually awkward combination of constraints. The first is copyright. Commercial recordings are the music most people listen to and most researchers want to study, and they are owned by record labels and publishers who have not, historically, been keen to see them redistributed as research datasets. For two decades this pushed MIR into a series of compromises: releasing only 30-second excerpts, releasing precomputed features rather than audio, or building datasets from music licensed under Creative Commons, which skews heavily toward independent and amateur producers. The Million Song Dataset, released in 2011 by Thierry Bertin-Mahieux, Daniel Ellis, Brian Whitman and Paul Lamere, illustrates the feature-only compromise. It provided metadata and precomputed audio descriptors for a million commercial tracks, but not the audio itself, which meant that any research needing a different representation was stuck. The Free Music Archive dataset of 2017, assembled by Michaël Defferrard and colleagues, took the licensing route instead, offering more than 100,000 freely licensed tracks with full audio. It was invaluable, and also unrepresentative of mainstream commercial music in production style, genre balance and audio quality. The second constraint is annotation cost. Labelling a song's genre takes seconds. Labelling its beats takes minutes, because an annotator must tap along and then correct the taps. Labelling its chords takes a trained musician perhaps an hour or more per song. Labelling every note of a recording, with exact onset and offset times, is so slow and error-prone that it was barely attempted by hand at scale. The most detailed labels are exactly the ones that are hardest to produce, so the tasks that need them have historically had the smallest datasets. The third constraint is that musical truth is often plural. Two expert listeners will disagree about chord labels, about where a section boundary falls, about whether a song is at 70 or 140 beats per minute, and even, more often than one might expect, about where the downbeat is. These are not annotation errors to be cleaned away. They reflect real ambiguity in the music and real differences in how trained listeners hear it. Clever ways around the bottleneck Several of the field's best datasets were built by finding a way to get labels for free, or nearly free, from some physical or social process. The most influential example is MAESTRO, released by Curtis Hawthorne and colleagues at Google's Magenta team in 2019. The International Piano-e-Competition has for years been held on Yamaha Disklavier pianos, acoustic grand pianos fitted with sensors that record every key and pedal movement with high precision. Each performance therefore produced both a high-quality audio recording and a precisely aligned record of what the pianist did. MAESTRO packaged roughly two hundred hours of these paired recordings, with note timings accurate to a few milliseconds. It turned piano transcription from a data-starved problem into a data-rich one almost overnight, and, as the next chapter shows, the results followed. A second trick is synthesis. Slakh2100, released in 2019 by Ethan Manilow and colleagues, took two thousand one hundred MIDI files from the Lakh MIDI dataset and rendered each instrument separately using professional sample-based virtual instruments. The result was a set of multitrack recordings in which the notes and the isolated stems were known exactly, ideal for training multi-instrument transcription and separation. The cost is realism: synthesised instruments lack much of the variety of real performance, and models trained on them have to bridge a gap when they meet real recordings. A third route is the multitrack session. Recordings are made instrument by instrument in the studio, and if the separate tracks are available, they provide ground truth for separation and, with additional processing, for pitch and activity labelling. MedleyDB, released in 2014 by Rachel Bittner and colleagues, assembled more than a hundred such multitracks with melody annotations. MUSDB18, assembled by Zafar Rafii and colleagues for the 2018 Signal Separation Evaluation Campaign, collected 150 full-length tracks, about ten hours in total, each split into vocals, drums, bass and an "other" stem. Its smallness is striking given its influence: for years, the entire field of music source separation benchmarked itself on fifty test songs. A fourth source is the wisdom of crowds and enthusiasts. MagnaTagATune, from 2009, gathered tags through an online game in which pairs of players described clips to each other. Large amounts of chord information exist on guitar tablature websites, written by hobbyists for other hobbyists; it is noisy, inconsistently formatted and partly wrong, but it covers hundreds of thousands of songs, and researchers have used it with alignment techniques to build weakly labelled chord datasets far larger than anything an expert team could annotate. Borrowing labels by alignment A fifth strategy deserves separate treatment because it underpins so much of the field: alignment. For a great deal of music, a symbolic version already exists somewhere, a published score, a MIDI file made by an enthusiast, a lead sheet, and a recording exists too. Neither alone is a labelled dataset. Together, if the score can be lined up against the recording moment by moment, they become one. The workhorse technique is dynamic time warping, an algorithm originally developed for speech recognition in the 1970s. Convert both the recording and a synthesised rendering of the score into a comparable representation, typically chroma features, and compute a grid of distances between every frame of one and every frame of the other. Dynamic time warping then finds the cheapest path through that grid from the beginning of both pieces to the end, under the constraint that time only moves forward. The path says, for every moment of the score, which moment of the recording corresponds to it, and so it transfers the score's notes, beats and bar numbers onto the audio. Because chroma is robust to differences in instrumentation and tempo, the method works surprisingly well even when the performance takes liberties with timing. MusicNet, released in 2017 by John Thickstun, Zaid Harchaoui and Sham Kakade, used this approach to label 330 freely licensed classical recordings, about thirty-four hours of chamber and solo music, with more than a million note annotations, by aligning them to scores. The alignment was imperfect, and later researchers found and corrected systematic timing errors in the labels, which is itself instructive: an automatically produced ground truth inherits the limitations of the algorithm that produced it. But MusicNet made multi-instrument classical transcription trainable at a time when no hand-labelled alternative of that size existed. Alignment has a further use that is easy to overlook. Many musicological questions do not need note-perfect transcription; they need to know where in a recording a given bar falls. Aligning dozens of recordings of the same symphony to its score lets a researcher compare how conductors shape tempo across the same passage, how long different pianists hold a fermata, or how performance practice has changed across a century of recordings. Projects on historical recordings of works such as Chopin's mazurkas, at the AHRC Research Centre for the History and Analysis of Recorded Music in London in the 2000s, used exactly this kind of alignment to make performance itself an object of large-scale study. Here MIR serves musicology most directly: not by replacing listening but by making it possible to listen comparatively at a scale no individual could manage. The genre dataset that taught the field humility No dataset illustrates the risks of ground truth better than GTZAN. George Tzanetakis and Perry Cook published "Musical Genre Classification of Audio Signals" in 2002, one of the founding papers of the field, along with a dataset of 1,000 thirty-second excerpts divided evenly among ten genres: blues, classical, country, disco, hip-hop, jazz, metal, pop, reggae and rock. The excerpts had been gathered from Tzanetakis's own CDs, radio recordings and other sources. GTZAN was small, freely circulated and easy to use, and it became the default benchmark for genre classification for more than a decade. Hundreds of papers reported accuracy on it, and those numbers rose steadily toward perfection. Then Bob Sturm looked closely at the data. In a series of papers beginning around 2012 and culminating in a detailed 2013 analysis titled "The GTZAN dataset: Its contents, its faults, their effects on evaluation, and future use", Sturm documented exact repetitions of the same excerpt, multiple excerpts from the same recording, many excerpts by the same artist, recordings with obvious distortion, and excerpts whose genre labels were arguably wrong. Because the standard practice was to split GTZAN randomly into training and test sets, excerpts from the same song or artist routinely ended up on both sides. A classifier could score highly by recognising a particular artist's recording sound rather than anything about genre. Sturm's deeper point was not about one dataset but about the gap between a classification score and a musical capability. A system that reached high accuracy on GTZAN had demonstrated that it could reproduce GTZAN's labels on GTZAN-like excerpts. Whether it had learned anything that a musician would call genre was a separate question, and one that accuracy alone could not answer. He coined the term "horse" for such systems, after Clever Hans, the horse that appeared to do arithmetic but was in fact reading involuntary cues from its questioners. The chapter on genre returns to this idea, because it turned out to apply far beyond genre. Labels as theory Chord annotation shows how deeply theory shapes labels. In 2005 Christopher Harte and colleagues proposed a formal syntax for writing chord labels, which became the standard in MIR. It allowed a root, a quality such as major, minor, diminished or seventh, optional added or omitted intervals, and a bass note. Using that syntax, the Isophonics project at Queen Mary University of London released chord annotations for the complete studio albums of the Beatles, which became the first widely used chord recognition benchmark. Later, John Ashley Burgoyne, Jonathan Wild and Ichiro Fujinaga released the McGill Billboard annotations, covering a random sample of songs from the Billboard Hot 100 chart between 1958 and 1991. These datasets were careful and expert. They also made choices. How should one label a passage where the guitar plays a suspended chord while the bass walks? Is a dominant seventh chord without its third still a dominant seventh? If the vocal melody sings the ninth above a major chord on a strong beat, is the chord a major chord or an added-ninth chord? Different annotators answer differently, and a study by Hendrik Vincent Koops and colleagues published in 2019 collected independent chord annotations from several experienced annotators for the same songs and found substantial disagreement, particularly on chord extensions and inversions. When the "ground truth" is one annotator's interpretation, a model can be penalised for agreeing with a different, equally competent expert. The vocabulary itself is a theory. Most chord recognition evaluation in the 2010s reduced all chords to a vocabulary of 24 major and minor triads plus a "no chord" label. That made the task tractable, and it also defined jazz harmony, with its extensions and alterations, as mostly out of scope. Systems that performed well on pop and rock under that vocabulary were not, in any useful sense, chord recognisers for jazz. The same pattern recurs throughout the field: what counts as success depends on a vocabulary chosen for convenience, and the vocabulary is rarely questioned once it becomes standard. How the field measures itself Evaluation in MIR was formalised early. J. Stephen Downie at the University of Illinois organised the Music Information Retrieval Evaluation eXchange, MIREX, beginning in 2005, on the model of the text retrieval evaluations run by the US National Institute of Standards and Technology. Researchers submitted systems; organisers ran them on held-out data that participants never saw; results were published together at the annual ISMIR conference. Because MIREX kept test data private, it partly protected against the overfitting that public benchmarks invite. It ran for well over a decade and established the standard tasks and metrics that much of the field still uses. As more datasets became public and the culture shifted toward open benchmarks and released code, the metrics themselves were standardised in software. The mir_eval library, published by Colin Raffel and colleagues in 2014, implemented the common metrics for beat tracking, chord recognition, melody extraction, transcription, segmentation and source separation in one place, so that two papers claiming the same score were actually computing the same thing. It sounds like housekeeping. It was a significant correction, because small differences in how onsets were matched or how chord labels were reduced had been producing incomparable numbers. The metrics encode theory in their turn. Transcription is usually scored by matching predicted notes to reference notes and counting a match if the pitch is right and the onset falls within 50 milliseconds of the reference. That tolerance is a claim about musical perception and performance: a note 40 milliseconds early is "correct", and one 60 milliseconds early is wrong. Beat tracking metrics have to decide what to do with a system that taps at double or half the annotated tempo, which a human listener might consider a perfectly reasonable interpretation; some metrics accept these "metrical level" variations, others do not, and the same system can look excellent or mediocre depending on which is reported. Source separation is scored by the signal-to-distortion ratio, a measure of how close the separated waveform is to the true stem, which correlates imperfectly with what listeners notice. Generation has no consensus metric at all, a problem the final chapters return to. What the data shapes The datasets described here share a profile. They are mostly Western, mostly popular or classical, mostly in equal temperament and regular metre, mostly recorded in professional studios or concert halls, and mostly annotated by people trained in Western music theory. Projects such as CompMusic, led by Xavier Serra at Universitat Pompeu Fabra in the early 2010s, made a deliberate effort to build corpora and tools for Hindustani, Carnatic, Turkish makam, Beijing opera and Arab-Andalusian traditions, and showed that many standard techniques needed rethinking to work on them. But the centre of mass of MIR data remains where it was. That matters because a learned system has no other source of musical knowledge. A beat tracker trained on pop is, quite literally, a model of pop rhythm. A chord recogniser trained on Beatles and Billboard songs is a model of the harmonic language of those songs. A music generator trained on a commercial catalogue is a model of that catalogue, including its production conventions, its gender balance among vocalists, and its recurring melodic and harmonic habits. When these systems are deployed to analyse or create music outside their training distribution, they bring their training theory with them, and the results range from subtly wrong to useless. The working lesson is that progress in MIR is at least as often a matter of better ground truth as of better models. The two jumps in capability that the next chapters describe most vividly, in piano transcription and in source separation, followed directly from new data and well-defined benchmarks. And the most important controversies the book ends on, over what generative models learned and from whom, are at bottom arguments about training data. Chapter 3: From Sound to Notes Automatic music transcription is the task of listening to a recording and writing down what was played. For a person, it is a skill that takes years to develop and remains hard even for professionals: transcribing a dense jazz piano solo or an orchestral tutti by ear is slow, and experienced transcribers disagree in the details. For a computer, it has been described for decades as the holy grail of MIR, partly because it would unlock so much else. A reliable transcription turns a recording into data that can be searched, analysed, compared, edited and re-rendered. The story of transcription over the past twenty years is one of uneven but genuine progress: a problem that is now close to solved for solo piano, much improved for many other single instruments, and still far from solved for the general case of a full band or orchestra producing a readable score. What "transcription" actually means The word covers several distinct targets, and much confusion in the field has come from treating them as one. Researchers usually distinguish at least four levels of output, which Table 2 lays out in order of increasing musical structure. Table 2. Levels of automatic music transcription. Level Output Question answered Main difficulty Frame Active pitches per short time frame What is sounding now? Overlapping harmonics Note Notes with pitch, onset, offset What notes were played? Onsets, repeated notes, offsets Stream Notes assigned to instruments or voices Who played each note? Timbre and voice separation Notation A readable score How should it be written? Metre, spelling, quantisation, layout Frame-level transcription is the most basic: for each small slice of time, say which pitches are active. Note-level transcription groups those frames into discrete notes with a start and end time, which requires detecting onsets reliably and handling the case of a note repeated without a gap. Stream-level transcription additionally assigns each note to a source, the flute line rather than the clarinet line, or the alto voice rather than the tenor. Notation-level transcription produces something a musician can read: notes quantised to a metrical grid, barlines placed, pitches spelled correctly as sharps or flats, rhythms written in conventional note values, and voices laid out on staves. Most published systems target the note level, and most reported numbers are note-level scores. A system that achieves excellent note-level accuracy may still produce output that is useless as a score, because the gap between a list of timed notes and a readable piece of notation is itself a hard problem. It requires inferring the metre, deciding that a note held for 0.47 seconds at a given tempo is a dotted eighth rather than an eighth tied to a thirty-second, and choosing between enharmonic spellings using the harmonic context. Notation-level transcription has remained much less studied than the note level, and its difficulty is one reason transcription tools aimed at musicians still frequently require heavy manual cleanup. Why polyphony is hard Transcribing a single melody line, monophonic pitch tracking, has been reasonably tractable for a long time. A sung or played note is periodic, and estimating the period of a periodic signal is a classic signal-processing problem with good solutions. Methods such as YIN, published by Alain de Cheveigné and Hideki Kawahara in 2002, estimate fundamental frequency robustly by looking for the time lag at which the signal best matches a shifted copy of itself. The neural pitch tracker CREPE, released by Jong Wook Kim, Justin Salamon, Peter Li and Juan Pablo Bello in 2018, learned the same mapping directly from waveform and set a new standard for accuracy on singing and solo instruments. Polyphony, several notes at once, breaks the simple approaches for a reason grounded in acoustics. Every pitched note produces harmonics at integer multiples of its fundamental. When notes form consonant intervals, which in tonal music is most of the time, their harmonics coincide. Play a C and the G a fifth above it, and the G's fundamental falls on the C's third harmonic, while every second harmonic of the G coincides with every third harmonic of the C. Play an octave and the upper note's harmonics are a subset of the lower note's. The spectrogram of a consonant chord is therefore not a clean sum of distinct note patterns but a pile of overlapping partials, and the question "is there a G here, or is that just C's third harmonic?" has no answer in the spectrum alone. Musicians resolve such ambiguities through expectation, timbre, onset timing and knowledge of the style. Early transcription systems had none of these. The ambiguity is worst for exactly the intervals music relies on most. Octave errors and errors at the fifth were the characteristic failure of transcription systems for decades, and they are the reason that a transcription score of 80 percent can feel much worse in practice: the mistakes are not random but concentrated on harmonically important notes. The classical approaches Before deep learning, transcription research developed two broad families of method. The first was iterative estimation and cancellation, associated especially with Anssi Klapuri's work in the early 2000s: detect the most prominent pitch in a frame, estimate and subtract its harmonic series from the spectrum, and repeat until nothing significant remains. It worked tolerably on simple textures and degraded as polyphony increased, because each mis-estimated subtraction left residual partials that looked like new notes. The second was spectrogram factorisation, particularly non-negative matrix factorisation as applied to music by Paris Smaragdis and Judith Brown in 2003. The idea is to approximate the magnitude spectrogram as a product of two smaller non-negative matrices: one holding a set of spectral templates, each roughly the spectrum of one note, and the other holding the activation of each template over time. If the templates correspond to piano notes, the activations are a piano-roll. NMF and its probabilistic cousins were elegant, interpretable and could be adapted to particular instruments by learning templates from isolated notes. Their limitation was that real notes do not have fixed spectra; a piano note's spectrum changes as it decays, varies with dynamics, and differs between instruments and rooms. Each refinement to the model, adding time-varying templates or sparsity constraints or instrument-specific priors, encoded a little more acoustic theory by hand. By the early 2010s, a review by Emmanouil Benetos, Simon Dixon, Dimitrios Giannoulis, Holger Kirchhoff and Anssi Klapuri described transcription performance as having reached a "glass ceiling", with incremental methods producing incremental gains. The authors argued that progress would require bringing in musical knowledge and larger data, and in hindsight both turned out to be right. The piano breakthrough The change came with neural networks trained on aligned audio and MIDI, and it came first for piano. In 2018 Curtis Hawthorne and colleagues at Google Magenta published Onsets and Frames. The architecture took a log-mel spectrogram and trained two connected networks: one to detect note onsets, and one to detect which pitches were active in each frame. The crucial design choice was to let the onset detector gate the frame detector, so that a note could only begin where an onset had been predicted. This encoded a simple piece of musical and acoustic knowledge, that a piano note's attack is its most reliable signature and that its sustained portion is ambiguous, directly into the architecture. Note-level accuracy jumped markedly over previous systems. The following year, the same group released MAESTRO, with about two hundred hours of precisely aligned piano performance, and Onsets and Frames retrained on it improved again. Subsequent systems refined the formula: Qiuqiang Kong and colleagues at ByteDance in 2021 regressed precise onset and offset times rather than classifying frames, and explicitly transcribed the sustain pedal, which matters enormously for piano because the pedal decouples when a key is released from when the sound stops. On the MAESTRO test set, the best note-onset scores for solo piano now sit well above 90 percent, high enough that systems of this kind have been used to build large symbolic datasets from piano recordings. Giant-MIDI-Piano, released by Kong and colleagues, applied such a transcriber to roughly ten thousand solo piano works found online, and the Aria-MIDI dataset published in 2025 applied a similar pipeline at a substantially larger scale. Two caveats temper this success. First, it is success at the note level, not the notation level: the outputs are performance MIDI, with expressive timing intact, rather than scores. Second, it is success for one instrument with unusually favourable properties. The piano is percussive, so onsets are sharp; it has a fixed tuning, so pitch is discrete; its notes decay rather than being sustained by bow or breath, so their spectra change in predictable ways; and a single, high-quality, perfectly labelled dataset existed for it. Few other instruments share all of those advantages. Many instruments at once Multi-instrument transcription is harder on every axis. Bowed strings and voices produce continuous pitch with vibrato and glides, so the boundaries between notes are less clear. Wind instruments' timbres shift with dynamics. Drums need a different vocabulary entirely, of hits rather than pitches. And in a full mix, instruments mask each other in frequency as well as overlap harmonically. The most influential response has been to recast transcription as sequence-to-sequence translation. MT3, published by Josh Gardner, Ian Simon, Ethan Manilow, Curtis Hawthorne and Jesse Engel at ICLR in 2022, used a Transformer architecture of the kind developed for machine translation. It took a spectrogram as input and emitted a sequence of MIDI-like tokens as output: tokens for time, for instrument program, for note-on and note-off, and for pitch. Because the output vocabulary included instrument identity, a single model could transcribe piano, guitar, bass, strings, winds and drums together, and it could be trained on a mixture of datasets including MAESTRO, Slakh2100, MusicNet and others. MT3 improved substantially on prior multi-instrument baselines, particularly on low-resource datasets where it benefited from training on everything else at once. The success of MT3 is instructive for the book's argument. Its token vocabulary is a theory of music borrowed largely from MIDI: notes are discrete events from a fixed set of pitches, assigned to a fixed set of General MIDI instrument programs. That theory works well for much Western music and fails gracefully elsewhere. A singer's portamento, a guitarist's bend or a sitar's meend do not fit into note-on/note-off tokens, and the model has no way to express them except as a series of approximate discrete notes. A different design point is represented by Basic Pitch, released by Spotify's audio research team in 2022 (Rachel Bittner and colleagues). It is a deliberately small, instrument-agnostic model that produces note events plus a continuous pitch-bend track, designed to run on a laptop or in a browser. It trades the accuracy of large instrument-specific models for broad applicability, and it is representative of a class of tools aimed at musicians and producers who want a rough MIDI sketch from a recording to start working with. Drums, and what counts as a note Drum transcription is a useful corrective to the idea that transcription is a single problem with a single solution. A drum kit has no pitch in the melodic sense. What matters is which instrument was struck, when, and with what force: kick drum, snare, closed or open hi-hat, toms, ride and crash cymbals. The difficulty is not overlapping harmonics but overlapping noise. A snare hit and a crash cymbal both produce broadband energy, the hi-hat bleeds into every other part of the kit's sound, and in a full mix the kick drum competes with the bass guitar in the same low frequencies. Automatic drum transcription therefore developed its own methods and its own datasets, such as ENST-Drums, recorded in 2006 by Olivier Gillet and Gaël Richard with drummers playing kits instrumented with separate microphones. Early systems used spectrogram factorisation with templates for each drum; later neural systems, surveyed by Chih-Wei Wu and colleagues in 2018, predict onset activations for each drum class in the same way that piano systems predict activations for each key. Most published work restricts itself to three classes, kick, snare and hi-hat, because they carry most of the groove and because finer distinctions are poorly represented in the data. A drummer reading the output would notice immediately what is missing: ghost notes on the snare, the difference between a rimshot and a normal stroke, the open-and-close articulation of the hi-hat that defines so many funk and disco grooves. Those details are exactly what distinguishes one drummer's feel from another's, and they are what a reduced label vocabulary throws away. The lesson generalises. Transcription always means transcription into some vocabulary, and the vocabulary is a decision about which distinctions matter. Western staff notation, MIDI note numbers, drum classes, guitar tablature and the sargam syllables of Indian classical music each make some distinctions and erase others. A system trained to produce one of them inherits its theory. How transcription is scored The standard note-level metric illustrates how evaluation choices shape what systems optimise. A predicted note counts as correct if its pitch matches a reference note to within half a semitone and its onset falls within 50 milliseconds of the reference onset. Precision is the fraction of predicted notes that are correct, recall the fraction of reference notes that were found, and the F-measure combines the two. A stricter variant also requires the offset to be roughly right, typically within 50 milliseconds or a fifth of the note's duration, whichever is larger. Scores with offsets are always noticeably lower, because note endings are acoustically vague: a piano note fades rather than stops, and the moment a listener considers it ended depends on what follows. These conventions are sensible, and they have consequences. A metric that ignores offsets rewards systems that get attacks right and neglects the sustain pedal and legato phrasing, which is why pedal transcription arrived relatively late. A metric that weights every note equally treats a missed bass note in a chord, which may change the harmony entirely, the same as a missed note in an inner voice of a dense texture, which a listener may never notice. Researchers have proposed musically weighted metrics and perceptual evaluations, and studies with listeners have confirmed that standard F-measures correlate only loosely with how good a transcription sounds or reads. The standard metrics persist because they are simple and comparable, and because changing a benchmark's definition makes earlier results incomparable, a recurring trade-off in the field. What transcription still gets wrong Several hard problems remain open as of 2026. Singing voice transcription to notes remains noticeably weaker than piano, because sung pitch is continuous and expressively inflected, and because deciding where one sung note ends and the next begins depends on lyrics and phrasing. Guitar transcription has the specific difficulty that the same pitch can be played on several strings, and guitarists read tablature that specifies which; a system that outputs only pitches has not produced what a guitarist needs. Datasets such as GuitarSet, released in 2018 by Qingyang Xi and colleagues using a hexaphonic pickup that records each string separately, helped, but data remains scarce. Orchestral transcription to a readable score remains far from practical. Above all, the notation-level problem persists. Converting performance MIDI into a score requires beat tracking, metre inference, quantisation, voice separation and pitch spelling, each of which is a research problem in its own right and several of which are covered in the next chapter. End-to-end models that go directly from audio to notation have begun to appear, often by treating a score format as a token sequence, but they remain limited to simpler material. The most reliable commercial transcription tools still produce a draft that a human transcriber edits. This residual gap is not a failure of engineering effort so much as a reflection of what a score is. A score is not a description of a recording. It is a set of instructions written in a notational language with its own conventions, and two competent transcribers can produce different, equally valid scores of the same performance. A system that learns to transcribe is learning not only to hear but to write in that language, and the training data for the second part, recordings paired with expert scores that correspond to them note for note, is still rare. Chapter 4: Beat, Harmony, Form and Identity A list of notes, even a perfect one, is not yet music in the sense a listener experiences it. Listeners hear a pulse and group it into bars. They hear harmony move from tension to resolution and sense a home key. They hear a verse give way to a chorus, a theme return transformed, a bridge that sets up the final repetition. These higher-level structures are what make music comprehensible, and they are the targets of a cluster of MIR tasks: beat and downbeat tracking, tempo estimation, chord recognition, key detection and structural segmentation. Each of these tasks is a partial crossing of the gap between sound and symbol, recovering one layer of musical organisation without attempting a full transcription. Each also reveals with unusual clarity how much the answer depends on a theory of the music being analysed. Finding the pulse Beat tracking asks a system to tap along with the music, outputting the times at which a listener would naturally clap or tap a foot. It is perhaps the clearest case where an MIR task mirrors a universal human ability: most people, without any training, can tap along to familiar music with reasonable consistency. The classical pipeline had two stages. First, compute an onset strength signal, a curve that rises wherever new sounds begin, typically by measuring frame-to-frame increases in spectral energy (a quantity called spectral flux). Second, find a regular sequence of times that lines up with the peaks of that curve. Daniel Ellis's dynamic-programming beat tracker of 2007 is a clean example: estimate a global tempo from the periodicity of the onset curve, then find the sequence of beats that best balances landing on strong onsets against keeping the spacing consistent with that tempo. Masataka Goto's earlier work in the 1990s had added musical knowledge explicitly, using snare and bass drum patterns typical of popular music to infer where beats and bar lines fell. Neural approaches replaced the onset detection stage with learned networks. Sebastian Böck, Florian Krebs and Gerhard Widmer at Johannes Kepler University Linz developed recurrent networks that predicted beat activations directly from spectrograms, followed by a probabilistic model, a dynamic Bayesian network, that inferred the most likely sequence of beat positions given those activations and a prior over tempo and metre. Later work by Matthew Davies and Böck in 2019 used temporal convolutional networks, which process long time spans efficiently, to predict beats and tempo jointly. Their methods, packaged in the open-source madmom library, became standard tools. More recently Transformer-based trackers such as Beat This!, published by Francesco Foscarin, Jan Schlüter and Gerhard Widmer in 2024, have shown that the probabilistic post-processing stage can be dropped entirely without losing accuracy, with the network itself learning the regularity constraints the post-processing used to impose. Downbeat tracking, finding the first beat of each bar, is harder. It requires understanding metre: whether the music groups beats in twos, threes or fours, and which beat is strong. Listeners infer this from harmonic changes, which tend to occur on downbeats; from bass notes; from the drum pattern; and from the phrase structure of the melody. Neural downbeat trackers learn these cues implicitly from data, and they work well on music like their training data and markedly worse elsewhere. Where the pulse becomes ambiguous The theory hidden in a beat tracker becomes visible when it fails. Standard datasets for training and evaluation, such as the Ballroom dataset of dance excerpts or collections of pop songs, are dominated by steady tempos, clear percussion and regular metres in two or four. A tracker trained on them will be confident and accurate on a club track and will flounder on a Chopin nocturne, where the pianist stretches and compresses time expressively (rubato), or on a choral piece with no percussion and soft onsets. Metrical level is the most common source of disagreement, and it is a genuine musical ambiguity rather than a system error. A piece at 70 beats per minute with a busy hi-hat pattern can be felt at 140 just as naturally; a fast waltz may be tapped on every beat or once per bar. Human annotators disagree on this, and evaluation metrics have to decide whether to accept answers at double or half the reference tempo. A tracker can be "wrong" by the strictest metric while being a perfectly reasonable listener. Outside the Western mainstream the ambiguity becomes structural. Carnatic music organises time in talas, cycles such as the eight-beat adi tala or the seven-beat misra chapu, whose internal grouping is defined by convention rather than by the loudness of particular beats; an analysis system must track position within the cycle, which is a different problem from finding a strong-weak alternation. Research groups including the CompMusic project showed that applying pop-trained trackers to such music gave poor results, and that useful systems needed to be designed around the tradition's own metrical theory, including its long cycles and additive groupings. The same holds for much West African and Afro-Cuban music, where interlocking patterns and cross-rhythms mean that the "beat" a listener locks onto depends on cultural training. A beat tracker that outputs one answer has implicitly declared one way of hearing correct. Harmony: chords and keys Chord recognition asks a system to label each stretch of a recording with the chord being played. It is useful to musicians, who want lead sheets of songs they are learning, and to musicologists, who want to study harmonic practice across large corpora. For about fifteen years the standard approach had two parts. Compute a chromagram, the twelve-dimensional pitch-class representation described in Chapter 1. Then compare each frame's chroma vector to a set of chord templates, the ideal pitch-class patterns of each chord, and smooth the results over time with a hidden Markov model so that chord labels did not flicker frame by frame. The hidden Markov model's transition probabilities could encode musical expectations, for instance that chords tend to last for a beat or more and that certain progressions are more common than others. Matthias Mauch's work around 2010 integrated beats, bass notes and keys into a single probabilistic model, and showed that the more musical context the model had, the better it performed. Deep learning replaced the hand-built chroma and templates with networks trained on annotated datasets, first convolutional models and later recurrent and Transformer models that see long stretches of music at once. Filip Korzeniowski and Gerhard Widmer showed in 2016 that a network could learn a chroma-like representation more robust to timbre and harmonics than the hand-designed one. Brian McFee and Juan Pablo Bello in 2017 tackled the vocabulary problem by building a model that predicted chord root, quality and bass jointly, which allowed it to handle a large vocabulary of chord types, including sevenths, suspended chords and inversions, without the rare chord classes being swamped by the common ones. Performance on the reduced vocabulary of major and minor triads rose to roughly the level of agreement between expert annotators on the standard datasets, which is to say that further gains on that metric would mostly mean learning to agree with one particular annotator. On larger vocabularies, with sevenths, extensions and inversions, systems remain noticeably weaker, partly because rare chords are rare in training data, and partly because the labels themselves are less consistent. Jazz, where extended and altered chords are the norm and where the same harmonic function can be voiced in many ways, remains particularly hard. Key estimation, finding the tonal centre of a piece, has the longest roots in music psychology of any of these tasks. In 1982 Carol Krumhansl and Edward Kessler published key profiles derived from listening experiments in which participants rated how well each of the twelve pitch classes fit a key-establishing context. Correlating a piece's pitch-class distribution against those profiles, an approach Krumhansl developed with Mark Schmuckler, gives a simple and surprisingly effective key estimator. Modern systems learn their profiles from data, but the underlying idea, that a key is a characteristic distribution of pitch classes, is the same. Its limitation is also the same: a key is a theory of hierarchical pitch relationships that applies most cleanly to common-practice tonal music, and becomes progressively less apt for modal, blues-based, chromatic or atonal music. Much rock and blues is harmonically centred on a tonic but uses a flattened third or seventh freely, and asking whether it is "in major or minor" may be the wrong question. Form: finding the sections Structural analysis asks where a piece divides into sections, and which sections are repetitions of each other. In popular music this means finding the intro, verses, choruses, bridge and outro; in classical music it might mean the exposition, development and recapitulation of a sonata movement. The central tool here has been the self-similarity matrix, introduced to music analysis by Jonathan Foote in 1999. Compute a feature vector for each moment of a recording, compare every moment to every other, and arrange the comparisons in a square grid. Repeated sections show up as stripes running parallel to the main diagonal, because each moment in the second chorus closely resembles the corresponding moment in the first. Sharp changes in character show up as boundaries between blocks along the diagonal. Foote's novelty detection method slid a checkerboard-shaped kernel along the diagonal to find those boundaries. Later methods, such as the spectral clustering approach of McFee and Ellis in 2014, extracted both the boundaries and the repetition structure together. Which feature you use to build the matrix is, again, a theory. Chroma features make the matrix reveal harmonic repetition, so a verse and chorus over the same chord progression look alike. Timbral features make it reveal instrumentation, so a sparse verse and a loud chorus look different even if they share harmony. Rhythmic features reveal changes in groove. A listener uses all of these at once, and weighs them depending on genre: in much electronic dance music, form is defined by the entry and exit of layers over an unchanging harmony, while in a jazz standard the harmonic form, the chorus of thirty-two bars, repeats while everything else changes. Annotator disagreement is highest of all for structure. The SALAMI dataset, released by Jordan Smith and colleagues in 2011, deliberately collected structural annotations from two independent annotators per piece and at two levels of detail, precisely because experts disagree on both where boundaries are and how coarsely to segment. Two musicologists can describe the same song as having four sections or eleven, and both be right at their own level. Evaluation metrics therefore tend to be lenient, and the ceiling for agreement between machines and humans is set by the agreement between humans. Recognising the same music twice One more family of analytical tasks deserves attention here, because it becomes central in the generative chapters: deciding when two recordings contain the same music. The question has two very different versions. The first is identifying the same recording. This is what audio fingerprinting services do when a phone listens to a few seconds of sound in a noisy café and names the track. The best-known method was described by Avery Wang of Shazam in 2003. Compute a spectrogram, keep only the strongest local peaks, a sparse "constellation" of time-frequency points, and then hash pairs of nearby peaks into compact codes that record their two frequencies and the time between them. Those hashes survive noise, compression and equalisation remarkably well, because strong peaks tend to remain strong whatever else happens to the signal. Matching a query is a matter of looking up its hashes in a database and finding a recording where many of them line up with a consistent time offset. Fingerprinting is designed to be extremely specific: it identifies one particular recording and deliberately does not match a different performance of the same song. Streaming services and video platforms use such systems to identify uploaded copies of commercial recordings and route royalties or enforce takedowns. The second version is identifying the same composition across different recordings: cover song identification, sometimes called version identification. A cover may change key, tempo, instrumentation, genre, structure and singer while remaining recognisably the same song. Fingerprints are useless here, because nothing at the level of spectral peaks survives. What survives is the melody and harmony, abstracted from everything else. Classic approaches, such as the one developed by Joan Serrà, Emilia Gómez and colleagues around 2008 and 2009, compared sequences of chroma features, shifted them through all twelve transpositions to handle key changes, and used alignment methods related to dynamic time warping to tolerate changes in tempo and structure. Neural systems now learn embeddings in which different versions of the same work sit close together and different works sit apart, trained on large collections of known covers. Cover detection embodies a sharp theory of musical identity: that a song is its melody and harmony, independent of timbre, key, tempo and arrangement. That is close to how copyright law in many countries thinks about a musical work as distinct from a sound recording, and also close to how musicians think about a tune. It is a theory with known blind spots. In much hip-hop and electronic music, identity resides as much in a particular beat, sample or sound as in any melody, and a chroma-based system will judge two tracks with the same chord progression to be related even when every listener hears them as unrelated. Melodic similarity, the finer question of how alike two melodies are, has been studied in MIR since its early days for applications such as query by humming, and the field has learned that no single measure matches human judgements across styles. That difficulty will matter a great deal when, later in this book, the question becomes whether a generated song is too close to an existing one. The common thread Beat, harmony and form are all hierarchical. Beats group into bars and bars into phrases. Chords group into progressions and progressions define keys. Sections group into larger forms. Human listeners perceive these levels simultaneously and use each to make sense of the others: a harmonic change helps locate a downbeat, a downbeat helps locate a phrase boundary, a phrase boundary helps locate a section change. Early MIR systems treated each task separately. The strongest modern systems estimate several jointly, beats with downbeats, chords with keys and bass, sections with repetition, and they consistently do better. This is the musicological argument of the book appearing in engineering form. A model that treats music as a flat sequence of frames has to rediscover, from data, the hierarchical organisation that musicians already know is there. Where the data are plentiful and representative, it often can. Where they are scarce, or where the music is organised according to a different theory than the one implicit in the training data, the system's results are only as good as the musical assumptions built into it. The next chapter examines a task where those assumptions are least stable of all. Chapter 5: The Trouble with Genre Genre classification is the task almost everyone imagines first when they hear about computers analysing music. Give a system a clip; it says "jazz" or "metal" or "reggae". It was one of the founding problems of MIR, it is a staple of teaching courses, and it remains commercially relevant, because streaming services organise catalogues, playlists and recommendations partly by genre. It is also the field's most instructive cautionary tale. Systems that reported near-perfect genre accuracy turned out, on close inspection, to have learned remarkably little about music, and the reasons why apply to every learned system discussed in this book, including the generators in its final chapters. Genre is not a property of sound The first difficulty is conceptual. Genre is not a physical attribute of a recording in the way tempo or pitch is. It is a social category, assigned by communities of listeners, musicians, critics, record shops and marketers, and its boundaries shift over time and across places. The musicologist Franco Fabbri, writing in the early 1980s, described a genre as a set of musical events whose course is governed by socially accepted rules, and he listed many kinds of rule: formal and technical conventions, but also semiotic, behavioural, social and economic ones. What makes a record "punk" includes its tempo and guitar sound but also its attitude, its scene, its mode of distribution and the clothing of the band. Some genre markers are reliably audible. Reggae has a characteristic emphasis on the offbeat in the guitar or keyboard, a so-called skank, and a prominent bass line. Drum and bass has a tempo range around 160 to 180 beats per minute and fragmented breakbeat drum patterns. Classical orchestral music has no drum kit and a wide dynamic range. Other markers are not audible at all, or only weakly. Whether a track with acoustic guitar and a plaintive vocal is "country" or "folk" or "Americana" depends on the artist's career, the label's marketing and the radio formats it is pitched to. Whether a dance track is "house" or "tech house" or "deep house" is the subject of passionate disagreement among the people who make and play it. Many genres exist chiefly as ways of describing communities. Genre vocabularies are also neither fixed nor flat. A record shop in the 1990s might have had a dozen sections. The data alchemist Glenn McDonald, working at the music intelligence company The Echo Nest and later at Spotify after it acquired that company in 2014, maintained a public map called Every Noise at Once that by its later years charted roughly six thousand distinct genre labels, many of them highly local or niche, inferred from listening patterns and other data. A genre taxonomy is a theory of how music divides, and different communities hold different theories. A ten-class benchmark encodes one particular, coarse and largely Anglo-American theory. The arc of a benchmark The founding paper, Tzanetakis and Cook's 2002 study introduced in Chapter 2, framed genre classification as a supervised learning problem over hand-designed features. Their features summarised timbre (statistics of the spectrum, including mel-frequency cepstral coefficients borrowed from speech recognition), rhythm (a histogram of the periodicities in the onset signal) and pitch content (a histogram of detected pitches). Fed into standard classifiers, these features achieved around 61 percent accuracy on the ten GTZAN genres, which was well above chance and, the authors noted, broadly comparable with human performance on a related task with short excerpts. Over the next decade accuracy on GTZAN climbed steadily as researchers added richer features and stronger classifiers: sparse representations, auditory models, ensembles. By the early 2010s numbers above 90 percent were being reported. When deep learning arrived, convolutional networks trained on mel spectrograms pushed reported figures higher still. At the same time, larger and more realistic tagging datasets became available, and work such as that of Keunwoo Choi, George Fazekas and Mark Sandler in 2016, and of Jordi Pons and colleagues in 2018, who trained tagging networks on 1.2 million tracks from a commercial catalogue, showed that deep networks could learn useful descriptors at industrial scale. The rising numbers told a story of progress. Sturm's analysis complicated it. Beyond documenting duplicates and mislabelling in GTZAN, he showed that when the dataset was split so that no artist appeared on both sides of the train-test divide, reported accuracies fell substantially. This "artist filter" had in fact been recommended years earlier, by Elias Pampalk and colleagues in 2005 and by Arthur Flexer in 2007, who had demonstrated that genre and similarity results were inflated when the same artist or album appeared in training and test data. Their point had not become standard practice. Many systems, it emerged, had been partly learning to recognise particular recordings and production styles and then reporting that as genre. Clever Hans in the studio Sturm then went further, in a 2014 paper with a title designed to stick: "A simple method to determine if a music information retrieval system is a horse." The method was to apply small, carefully chosen changes to the input audio, changes that a human listener would barely notice or not notice at all, and see whether the system's answer changed. In his experiments, applying subtle filtering, essentially minor equalisation of the kind any mastering engineer might apply, could push a state-of-the-art genre classifier to assign the same excerpt to almost any genre one wished. A blues excerpt could be made to register as classical, or as metal, without a listener perceiving any change in genre. The conclusion was that the systems had not learned what a musician means by genre. They had learned statistical regularities in the spectral envelope that happened to correlate with genre labels in the dataset. In GTZAN, the classical excerpts tended to come from certain recordings with certain frequency characteristics, the metal excerpts from others, and so on. Those correlations were enough to classify the dataset; they were not enough to survive a change in equalisation, let alone a different recording studio or a different decade. This is the pattern that machine-learning researchers now call shortcut learning: a model solving a benchmark by exploiting a cue that is predictive in the data but irrelevant to the intended task. Well-known examples outside music include image classifiers identifying cows by the presence of green grass and medical models detecting a disease from the hospital-specific markers on an X-ray. MIR saw the phenomenon early and documented it carefully, partly because music offers so many tempting shortcuts. Recording era, mastering loudness, sample rate, codec artefacts, the presence or absence of vinyl surface noise, and the characteristic frequency curves of particular studios or record labels all correlate strongly with genre in real catalogues. A network trained on mel spectrograms has full access to every one of them. Later work made the analysis more systematic. Francisco Rodríguez-Algarra, Bob Sturm and Simon Dixon showed in 2019 how deliberate interventions on the input, such as filtering out particular frequency regions, could reveal which parts of the signal a classifier relied on, and found that some systems leaned on frequency content below the range of most musical fundamentals. The general lesson was that high accuracy on a static test set is weak evidence of musical understanding. Stronger evidence comes from testing whether the system's behaviour changes in musically appropriate ways when the music changes, and stays the same when only irrelevant properties change. Beyond ten boxes Part of the field's response to the GTZAN episode was to change the shape of the problem rather than only the evaluation. If genre is plural, hierarchical and contested, then a task that asks for exactly one of ten labels per excerpt is the wrong formulation, however carefully it is scored. One move was from single-label classification to multi-label tagging. Instead of choosing one genre, a system predicts a set of independent tags, genres alongside instruments, moods, vocal characteristics and eras, each with its own probability. A track can then be both "folk" and "electronic", both "melancholic" and "danceable". Datasets built for this include MagnaTagATune, mentioned in the previous chapter, and the MTG-Jamendo dataset released in 2019 by Dmitry Bogdanov and colleagues at the Music Technology Group in Barcelona, which provides tens of thousands of full-length Creative Commons tracks with tags for genre, instrument and mood. Tagging is closer to how listeners and catalogues actually describe music, and it lets models express uncertainty and overlap instead of forcing a choice. A second move was to take disagreement seriously as data. The AcousticBrainz Genre dataset, presented by Bogdanov, Alastair Porter, Hendrik Schreiber, Julián Urbano and Sergio Oramas at ISMIR in 2019, gathered genre annotations for the same recordings from four different sources, including the editorial taxonomy of an online music guide, the user-contributed taxonomy of a record-collecting database and folksonomy tags from a social listening site. The four taxonomies differed in size, in structure and in where they drew boundaries, and the same recording often received labels that did not map cleanly from one to another. Rather than resolving these conflicts, the dataset preserved them, so that researchers could study how well a model trained on one community's theory of genre transferred to another's. The answer, unsurprisingly, was imperfectly. A third move was hierarchy. Genres nest: bebop within jazz, Chicago house within house within electronic dance music. Models that predict at several levels of a hierarchy can be right about the broad family while uncertain about the subgenre, which is more useful than a single wrong fine label. Hierarchies, though, are also theories. Whether trap belongs under hip-hop or under electronic music, or whether Afrobeats is a subgenre of anything in a taxonomy designed in North America and Europe, are not questions with neutral answers. The parallel case of mood Mood and emotion recognition, a task closely related to genre, raises the same issues in sharper form. Systems are trained to predict whether music is happy or sad, calm or energetic, often using the two-dimensional model of affect developed by the psychologist James Russell in 1980, with valence (pleasant to unpleasant) on one axis and arousal (calm to excited) on the other. Arousal turns out to be fairly predictable from audio, because it correlates with tempo, loudness, brightness and rhythmic density. Valence is much harder, because whether music sounds happy depends on harmony and mode, on cultural conventions about which sounds signify what, and on lyrics, and because listeners disagree with each other far more about valence than about energy. Here the distinction between perceived and induced emotion matters too. Most datasets ask annotators what emotion the music expresses, not what they themselves feel, since a listener can recognise a piece as sad while enjoying it. A system trained on such labels is predicting a social convention about musical expression, which is a perfectly reasonable target as long as it is described as such. Like genre, mood is an attribute of music-in-a-community rather than of sound alone, and systems that predict it are learning that community's codes, with all the shortcuts those codes allow. What genre systems are for None of this means that automatic genre and tag prediction is useless. It is widely used, and for good practical reasons. A streaming service receiving tens of thousands of new tracks a day cannot have a human listen to each one. Predicting tags such as genre, mood, instrumentation, vocal presence, energy and danceability from audio lets a new track with no listening history be placed in a recommendation system immediately, which addresses what is known as the cold-start problem. Sander Dieleman, then a doctoral student, described experiments in 2014 using convolutional networks to predict listening-behaviour embeddings from audio, a line of work that made clear how audio content could complement collaborative filtering based on what users play. In that operational setting, shortcuts are not always a problem. If a production style genuinely predicts which listeners will enjoy a track, a system that uses it is doing its job. The difficulties come when the system's outputs are interpreted as musical claims, for instance by a musicologist studying genre history, by an artist trying to understand why their work is classified one way, or by a policy-maker relying on automated classification to apply rules. They come also when the system is applied to music unlike its training data, where the shortcuts no longer correlate with anything. There is also a feedback loop worth naming. Genre labels predicted from audio influence which playlists a track lands in, which listeners hear it, and therefore which listening data accumulate around it. Those data then train the next generation of models. Over time a recommender's implicit theory of genre can become partly self-fulfilling, reinforcing the boundaries it was trained on, which may be one reason why the most commercially successful music tends to be easier to classify than music that crosses boundaries. Doing it better The genre story produced a set of practices that now apply widely in MIR and well beyond it. Test sets should be split by artist, album or recording session, so that a model cannot score by recognising a familiar source. Performance should be checked on data from a different collection than the one used in training, to see whether it generalises across recording conditions. Systems should be probed with interventions, including small perceptually irrelevant changes that should not alter the answer and musically meaningful changes that should. Labels should be treated as one community's interpretation, ideally with multiple annotators and with their disagreement retained rather than averaged away. And claims should be calibrated to evidence: a system that classifies a benchmark accurately should be described as such, not as having learned genre. These practices matter even more for generation than for classification. A music generator that has been trained to produce output "in the style of" some genre or artist is, at bottom, a model of the statistical regularities of its training data, exactly as a classifier is. It may have learned harmony, rhythm, form and instrumentation, or it may have learned production texture and a store of memorised fragments, or more likely some mixture. Telling those possibilities apart requires the same discipline the genre debate developed: asking what the system's success actually consists of, and testing for it directly. The final chapters return to that question. First, though, the book turns to the analytical task where deep learning delivered its most tangible change in what working musicians can do. Chapter 6: Unmixing the Mix For most of recording history, a finished mix was effectively final. Once the vocals, drums, bass, guitars and keyboards had been balanced, processed and summed into two stereo channels, there was no general way to get the individual parts back. If the original multitrack tapes were lost, as they often were for older recordings, the separate performances were gone for good. Music source separation, the task of recovering the individual sources from a mixture, was for decades a research problem whose results were interesting to specialists and unusable to anyone else. In the space of roughly a decade that changed completely. Separation tools built on deep learning are now built into consumer software, used daily by producers, DJs, karaoke services and music teachers, and have been used to restore and remix recordings by some of the most famous artists in history. The story of how this happened is the clearest case in MIR of steady, benchmarked progress, and it also shows what the benchmark could and could not measure. The problem and why it was hard The difficulty of source separation follows from how mixing works. A stereo mix is, to a first approximation, the sum of its individual tracks, each scaled and placed somewhere in the stereo field. Recovering the parts from the sum is an underdetermined problem: there are more unknowns than equations, in the most literal sense. With two channels and four or five sources, infinitely many combinations of source signals could have produced the same mixture. The only way to choose among them is to bring in knowledge about what each kind of source typically sounds like. Early approaches used whatever regularities were available. If the vocals were panned dead centre and the instruments to the sides, subtracting the left channel from the right removed the vocals, an old karaoke trick that worked crudely on some recordings and badly on most. Methods based on independent component analysis assumed sources were statistically independent and needed as many microphones as sources. Non-negative matrix factorisation, introduced in the transcription chapter, could separate a spectrogram into components and group them into sources, but grouping the components correctly was itself the hard part. A method called REPET, developed by Zafar Rafii and Bryan Pardo around 2012, exploited the fact that accompaniment in popular music is often highly repetitive while vocals are not: find the repeating pattern, model it, and what is left over is mostly voice. It was a neat example of musical structure supplying the missing constraint, and like all such methods it worked only where its assumption held. Masks and the deep-learning turn The approach that came to dominate is conceptually simple. Compute a magnitude spectrogram of the mixture. For each source, estimate a mask: a grid of the same shape with values between zero and one, saying what fraction of the energy at each time-frequency point belongs to that source. Multiply the mixture spectrogram by the mask, combine the result with the mixture's phase, and invert back to a waveform. The idea of masking predates deep learning; what changed was how the mask was estimated. Instead of deriving it from an explicit model, train a neural network to predict it, using datasets where the true sources are known. Po-Sen Huang and colleagues showed the promise of this for singing voice separation around 2014. Andreas Jansson and colleagues at Spotify, in 2017, adapted the U-Net architecture from biomedical image segmentation to vocal separation, treating the spectrogram as an image and predicting the vocal mask with an encoder-decoder network whose skip connections preserved fine detail. Results were dramatically better than older methods, particularly in reducing the bleed of instruments into the separated vocal. Two releases in 2019 put the technology into wide circulation. Open-Unmix, from Fabian-Robert Stöter, Stefan Uhlich, Antoine Liutkus and Yuki Mitsufuji, was designed as an open reference implementation for researchers: a recurrent network operating on spectrograms, released with code and trained weights. Spleeter, from the research team at the streaming service Deezer, was released as a ready-to-use tool that separated a song into two, four or five stems faster than real time on ordinary hardware. Spleeter in particular spread far beyond research, into hobbyist remixing, music education and commercial products, because it made a capability that had previously required a lab available to anyone with a laptop. Learning in the waveform Spectrogram masking has a built-in limitation, and it relates directly to the representational choices of Chapter 1. Masking works on magnitudes and reuses the mixture's phase for each separated source. Where two sources overlap at the same time-frequency point, the mixture's phase is a combination of theirs, and the separated outputs inherit errors that sound like smearing or a metallic, watery quality. The ideal mask, computed with knowledge of the true sources, gives an upper limit on how well any masking method can do, and that limit is not perfect. An alternative is to separate directly in the waveform. Wave-U-Net, by Daniel Stoller, Sebastian Ewert and Simon Dixon in 2018, applied the U-Net idea to raw audio. Alexandre Défossez and colleagues at Facebook AI Research introduced Demucs in 2019, a waveform model with a convolutional encoder, a recurrent middle and a convolutional decoder, which matched and then exceeded spectrogram methods on drums and bass in particular, the sources where phase and transients matter most. Défossez's Hybrid Demucs, presented at the Music Demixing Challenge of 2021, combined a waveform branch and a spectrogram branch in one network, letting each domain handle the sources and frequency ranges it suited best, and was among the top systems in that challenge. The next step brought in Transformers. Hybrid Transformer Demucs, published by Simon Rouard, Francisco Massa and Alexandre Défossez in 2022 and presented at ICASSP in 2023, inserted cross-domain Transformer layers between the two branches so that information could flow between the waveform and spectrogram representations. Trained with 800 songs in addition to the MUSDB18 training set, it reported an average signal-to-distortion ratio of 9.20 dB on the MUSDB18 test set, the state of the art at the time. The Band-Split RoPE Transformer, from Wei-Tsung Lu, Ju-Chiang Wang, Qiuqiang Kong and Yun-Ning Hung at ByteDance, published in 2023, took a different route: it split the spectrogram into frequency bands of musically motivated widths, modelled each band with Transformer layers that attend both across time and across bands, and reported 9.80 dB average on MUSDB18-HQ without any extra training data. A version trained with additional songs won the music separation track of the Sound Demixing Challenge 2023. Table 3 places these systems side by side. It is not a leaderboard, since the systems were evaluated under different conditions and with different training data, but it shows how the field's design choices shifted over a few years. "STFT" marks systems that work on a short-time Fourier spectrogram, "Wave" those that work on the raw waveform. Table 3. Selected music source separation systems (descriptions from the cited papers). System Year Input Core architecture Notable for U-Net (Jansson et al.) 2017 STFT Convolutional encoder-decoder Vocal masks via image segmentation Open-Unmix 2019 STFT Recurrent network Open research baseline Spleeter (Deezer) 2019 STFT Convolutional U-Net Mass adoption as a tool Demucs 2019 Wave Convolutional with recurrent core Strong drums and bass Hybrid Transformer Demucs 2022 Both Cross-domain Transformer 9.20 dB SDR with 800 extra songs BS-RoFormer 2023 STFT Band-split Transformer 9.80 dB SDR without extra data What a decibel buys The signal-to-distortion ratio that these systems report measures, in decibels, how much of the separated output's energy matches the true source relative to everything else in it: interference from other sources, artefacts introduced by processing, and distortion. A gain of a few decibels sounds small but is substantial on a logarithmic scale. In listening terms, the progress from early deep models to recent Transformers is the difference between a vocal that clearly contains ghosts of the drums and a phasey quality, and one that a casual listener might mistake for the studio stem. The metric also has known weaknesses, which the field has been candid about. SDR is computed against MUSDB18's test set, which is fifty songs, mostly rock and pop, with a particular set of four stems. It rewards closeness to the reference waveform rather than perceptual quality, so two separations with the same score can sound different, and some artefacts that listeners find objectionable, such as a "musical noise" of isolated chirps, are penalised less than their audibility would suggest. Studies comparing SDR with listening tests have found the correlation real but imperfect. The four-stem target also embodies a narrow theory of the mix: "other" lumps together guitars, keyboards, strings, synthesizers and backing vocals. Separating a piano from a guitar, or a lead vocal from harmony vocals, is a much harder problem that the main benchmark did not measure, and that newer datasets and challenges have only begun to address. There is a deeper limit as well. A mix is not just a sum of tracks but the product of decisions: reverb that blends instruments into a shared space, compression that makes the drums duck when the bass hits, effects that deliberately smear one source into another. Some of what a listener hears as the vocal is room reverberation that also carries the guitar. When a separator returns a "clean" vocal, it has made a judgement about what belongs to the voice, and that judgement is learned from the stems in its training data, which were themselves products of particular engineers' choices. Beyond four stems The research frontier has moved in several directions away from the fixed four-stem setup, each of which revisits the theory of a mix the benchmark embedded. One direction is finer and more flexible targets. Separating many instruments requires training data with many isolated instruments, which synthetic datasets such as Slakh2100 can supply in quantity, at the cost of realism. Another approach is query-based separation: instead of training a separate output for each of a fixed list of sources, train a single model to extract whatever source a query describes. The query might be a short audio example of the target instrument, a class label, or a text description, using the joint audio-text embeddings described in Chapter 1. A system of this kind can, in principle, be asked to pull out "the acoustic guitar" or "the backing vocals" from any mix. In practice, quality for rare or vaguely described sources is well below that for the familiar four, but the shift from fixed classes to open-ended queries is significant, because it moves the definition of a source from the dataset designer to the user. A second direction is separating sources that the four-stem benchmark lumped together. The Sound Demixing Challenge of 2023 included a track on cinematic audio, separating dialogue, music and sound effects in film soundtracks, a task with large practical value for dubbing and remastering old films. Separating a lead vocal from harmony vocals, or one guitar from another in a two-guitar band, is particularly hard, because the sources share timbre and often harmony, and a model can rely only on subtle cues of timing, pitch and spatial placement. These are problems where musical knowledge, for instance of which line is the melody, ought to help, and where some research has begun to combine separation with transcription so that each informs the other. A third direction concerns robustness to the messiness of real training data. Professionally produced stems contain bleed, since a drum microphone picks up some of the bass amplifier, and mislabelled tracks, since a stem labelled "vocals" may contain a tambourine. The 2023 challenge deliberately included tracks with such corruptions in a dedicated track on training with noisy labels, and found that models could be made substantially more tolerant of them. The finding matters because it opens the door to training on much larger, less carefully curated collections of multitrack material, which is where the next gains in quality are likely to come from. Uses, from restoration to remix The practical consequences have been large. Karaoke and sing-along features in streaming and consumer apps now rely on on-the-fly vocal removal rather than specially prepared instrumental versions. Music teachers and learners isolate a bass line or drum part to study it. DJs separate stems live to blend a vocal from one track over the instrumental of another. Producers extract drum breaks or instrumental phrases for sampling. Separation also serves the rest of MIR as a front end, cleaning up input for transcription, lyric alignment, melody extraction and cover detection, so that downstream models face a single instrument rather than a full mix. Restoration is the most publicised use. For Peter Jackson's 2021 documentary series The Beatles: Get Back, his team developed a machine-learning separation system, which they called MAL, to isolate voices and instruments in mono recordings made during the 1969 sessions. The same technology was then used on the band's catalogue, enabling new stereo mixes of the 1966 album Revolver released in 2022, from tapes on which several instruments had been bounced together. In 2023 it made possible "Now and Then", released as a final Beatles single, by separating John Lennon's voice from the piano on a home demo cassette he had recorded in the late 1970s, a task earlier attempts in the 1990s had abandoned because the voice and piano could not be pulled apart. It is hard to imagine a more vivid demonstration that a research problem had crossed into practical use. Separation also creates new questions. The ability to extract a clean vocal from any commercial recording makes it trivial to build training data for voice cloning, and isolated stems can be used to construct remixes and mashups that rights holders have not licensed. Separation does not change the legal status of the underlying recording, but it lowers the technical barrier to reusing it, and that has been part of the context for the music industry's wariness about AI tools generally. The lesson of a good benchmark Source separation is often held up as MIR's success story, and in large part it is. The reasons are worth naming. The task had a clear definition, a public dataset with exact ground truth, a standard metric implemented in shared code, regular open challenges, and a culture of releasing models and code. Those conditions let dozens of groups compete on the same problem and build on each other's work, and the result was a decade of real, cumulative progress. The same story shows the limits of benchmark-driven progress. The benchmark defined separation as four stems on fifty mostly Western rock and pop songs, measured by waveform similarity. Systems became extremely good at that, and somewhat less good at what the benchmark did not test: separating arbitrary instruments, handling genres with very different mixing conventions, or producing output that sounds right rather than measures right. The four-stem theory of a mix became the field's working definition of what a mix contains, because it was the definition the field could measure. That pattern, a well-defined proxy pursued with great success while the underlying question remains partly open, is one worth carrying into the generative chapters, where no comparably clear proxy exists at all. Chapter 7: Composing in Symbols Long before neural networks could generate a convincing second of audio, computers were writing music. They did so in symbols: lists of pitches and durations that a human or a synthesiser then performed. Symbolic generation is the older half of algorithmic composition and, in several respects, the more instructive one. Because its output is notes rather than sound, its successes and failures are visible in a way that audio generation often hides. A model that produces a chord progression no musician would write, or a melody that wanders without any sense of phrase, cannot disguise the fact behind a convincing vocal timbre. And because composers and theorists have been formalising musical rules for centuries, symbolic generation has always been a direct meeting point between musicology and computation. Rules, grammars and chance Composers had used chance and systematic procedures long before computers. The eighteenth-century musical dice games, in which a minuet was assembled by rolling dice to select pre-written bars from a table, are a familiar example. So are the strict rules of species counterpoint, codified by Johann Joseph Fux in 1725 in Gradus ad Parnassum, which governed how independent melodic lines could move against each other and remained a teaching staple for two centuries. Those rules are, in effect, a constraint system, and that is exactly how early computer composition treated them. The first substantial piece composed with a computer, the Illiac Suite for string quartet, was produced in 1956 and 1957 by Lejaren Hiller and Leonard Isaacson at the University of Illinois. Its movements explored different procedures: generating random notes and rejecting those that broke rules of first-species counterpoint, then progressively more complex rule sets, and finally Markov chains in which the choice of each note depended probabilistically on the previous ones. The work was a deliberate experiment in which procedures produced which musical qualities. Iannis Xenakis, around the same time, used stochastic processes and mathematical models in composition, and his notion of stochastic music treated sound masses as statistical phenomena rather than collections of individual notes. Expert systems in the 1980s pushed rule-based composition further. Kemal Ebcioğlu's CHORAL system harmonised chorale melodies in the style of Johann Sebastian Bach using some three hundred and fifty rules, expressed in a logic programming language, covering voice leading, chord choice and melodic shape. Its output was competent and stylistically appropriate, and its rules were readable: anyone could see exactly what theory of Bach chorale harmonisation it embodied. The cost was enormous manual effort, and the rules captured the textbook version of the style rather than its exceptions. David Cope's Experiments in Musical Intelligence, begun in the early 1980s and widely known by its nickname EMI or "Emmy", took a different approach that anticipated modern machine learning in an important way. Instead of encoding rules, Cope's system analysed a corpus of a composer's works, broke them into small fragments, classified those fragments by their harmonic and structural function, and recombined them into new pieces according to patterns derived from the corpus. Cope called the underlying idea recombinancy. The results, in the styles of Bach, Mozart, Chopin and others, were striking enough to provoke public listening tests in which audiences sometimes mistook the output for the real thing, and also fierce debate about whether recombining fragments of a composer's own music counted as composing in that composer's style. That debate, it will turn out, has not gone away. Learning the statistics of style Markov models formalise one simple theory of style: the next event depends only on a fixed number of preceding events. A first-order Markov chain over pitches learns, from a corpus, how often each pitch follows each other pitch; a higher-order model conditions on longer histories. Trained on folk melodies, such models produce output that sounds locally plausible, with typical intervals and scale degrees, and globally aimless, because nothing in the model represents phrases, cadences or repetition at the scale of bars. Increasing the order improves local coherence but quickly leads to copying, since with a long enough context in a modest corpus, there is often only one continuation that has ever been seen. That tension between coherence and copying is the central problem of all data-driven composition, and it is visible here in its simplest form. Constraint-based approaches addressed the problem from the other side. François Pachet, Pierre Roy and colleagues at the Sony Computer Science Laboratory in Paris developed methods in the 2010s that combined Markov models with explicit constraints, for instance requiring a melody to end on the tonic, to fit a given metre, or to avoid reproducing any sequence longer than a set length from the training corpus. The last of these constraints made the copying problem explicit and controllable. Pachet's Flow Machines project used such techniques, together with human musicians, to produce songs in the style of particular artists or eras; a 2016 song released by the project, written with the French composer Benoît Carré and styled after the Beatles, attracted wide attention. In that workflow the machine proposed melodic and harmonic material and human musicians selected, arranged, wrote lyrics and produced the final track, a division of labour that remains common in practical AI-assisted composition. Neural sequence models Neural networks entered symbolic composition early. Peter Todd and Michael Mozer both used recurrent networks for melody generation around 1989 to 1994; Douglas Eck and Jürgen Schmidhuber showed in 2002 that LSTM networks could learn blues chord structures and improvise over them in a way simpler recurrent networks could not, because LSTMs could carry information across longer stretches of time. These were small experiments, limited by data and computing power, but they established the template: represent music as a sequence of tokens and train a network to predict the next one. The modern era of symbolic generation arrived with large corpora and better architectures. Bob Sturm and colleagues trained an LSTM on about 23,000 Irish and related folk tunes written in ABC notation, scraped from an online community archive, producing a system they called folk-rnn in 2016. Its output, generated in a few seconds per tune, was often playable and stylistically recognisable as session music, and the researchers did something unusual: they took the output to traditional musicians, published a collection of generated tunes, and studied how players responded to and adapted them. Those musicians found some tunes good, many forgettable, and a few strikingly strange, and the project made the point that evaluation of a generative system is best done by the community whose style it imitates. Gaëtan Hadjeres, François Pachet and Frank Nielsen's DeepBach, also from 2016, generated four-part chorales in the style of Bach using a model designed to allow constraints: a user could fix any notes, such as the soprano melody or a cadence, and the system would fill in the rest consistently. In a listening test reported by the authors, a substantial proportion of participants judged its chorales to be by Bach. Google's Magenta team, formed in 2016, released a series of symbolic models; the Bach Doodle of March 2019, which let users write a melody and have a model called Coconet harmonise it in Bach's chorale style, was used by millions of people in a few days. The Transformer architecture changed the scale of the problem. Music Transformer, by Cheng-Zhi Anna Huang and colleagues at Magenta in 2018, adapted the Transformer to music by using relative attention, which lets the model attend to musical events according to how far apart they are rather than their absolute positions. This matters musically because repetition and variation are relative: a motif that returns four bars later is related to its first appearance by distance, wherever in the piece it occurs. Trained on piano performances, Music Transformer generated minute-long passages with recurring motifs and a much more convincing sense of development than recurrent models. OpenAI's MuseNet of 2019 scaled the same idea to many instruments and styles, trained on a large collection of MIDI files. Tokenisation is theory Every symbolic language model depends on a vocabulary, and the choice of vocabulary turns out to be one of the most important design decisions. Music Transformer's performance encoding represented music as a stream of events: note-on with pitch, note-off with pitch, a time-shift advancing the clock by a small increment, and a velocity change. This representation captures expressive timing and dynamics exactly, which is why it works well for piano performance. It has no notion of beats or bars. Yu-Siang Huang and Yi-Hsuan Yang's REMI representation, introduced in their Pop Music Transformer in 2020, made a pointed alternative. It added explicit tokens for bar lines and for position within the bar, and tokens for chords and tempo. The model, trained on pop piano covers, produced output with noticeably more consistent rhythm and metre, because the vocabulary itself told it what a bar was. Compound-word tokenisation and later schemes went further, grouping the attributes of a note into a single token to shorten sequences. Each scheme is a theory of what musical events are and how they relate, and models trained on different tokenisations of the same corpus produce music with recognisably different strengths and weaknesses. A vocabulary without bar lines makes it harder for a model to learn metre; a vocabulary without separate voice tokens makes it harder to learn counterpoint. Generation as conditional infilling One development in symbolic generation deserves emphasis because it changed what such systems are useful for. Early models generated left to right: given a beginning, continue. That suits improvisation and suits the training objective, but it does not match how composers work. A composer rarely writes a piece from the first note to the last. They sketch a melody, revise the harmony under it, fill in an inner voice, rewrite a bridge, and move between levels of detail and between sections. Models designed for infilling address this directly. Coconet, developed by Cheng-Zhi Anna Huang, Tim Cooijmans, Adam Roberts, Aaron Courville and Douglas Eck in 2017, was trained by randomly erasing parts of four-part Bach chorales and learning to restore them. At generation time it could start from any partial score and repeatedly rewrite the parts with the lowest confidence, an approach the authors likened to the way a composer drafts and revises. DeepBach's constraint mechanism worked similarly. Later Transformer models trained with masking objectives, and anticipatory models that condition on future events as well as past ones, let a user fix the melody of a chorus and the chord on which a bridge must end and ask the model to write everything in between. This matters for the theory of style that a system embodies. A left-to-right model learns what tends to follow what. An infilling model learns what tends to co-occur with what, across voices and across time, which is closer to how harmony and counterpoint are actually organised. A Bach chorale is not composed as a sequence; it is a set of four lines whose simultaneous and successive relationships are both constrained. A model whose training objective respects that structure tends to produce more convincing results, and it is also more controllable, because constraints can be imposed anywhere rather than only at the start. Expressive performance as its own problem A score is not a performance, and a separate strand of research has addressed the step from one to the other. Given a piece as notated, how should it be played: where should the tempo stretch, which notes should be louder, how much should the pianist overlap one note with the next? Expressive performance rendering has a long history, including rule-based systems from the KTH Royal Institute of Technology in Stockholm, developed by Johan Sundberg and colleagues from the 1980s onward, which encoded principles such as slowing at phrase endings and emphasising melodic peaks, each rule a small theory of musical expression tested by listening experiments. Gerhard Widmer's group at Linz later used machine learning on large corpora of measured performances to discover such regularities from data, and found that some of the learned patterns matched the hand-written rules closely. Performance-trained models such as Music Transformer blur the boundary between composition and performance by learning both at once from performance MIDI. That is one reason their output sounds lively: the expressive timing is in the data. It also means that composition and interpretation are entangled in the model in a way that makes each harder to control separately. A user who wants the same generated passage played with more rubato, or a different passage played the same way, cannot easily ask for it. Systems that separate the two levels, generating a score and then rendering it, give up some naturalness for that control, and the choice between them is another instance of the trade-off between implicit fluency and explicit structure that runs through this book. What symbolic generation reveals Symbolic systems make the strengths and weaknesses of data-driven composition easy to see. They have become very good at local style: plausible chord progressions, idiomatic melodic shapes, convincing voicings and rhythms over a span of a few bars. They are much less good at long-range structure: introducing material, developing it, departing and returning, building toward a climax, and ending in a way that resolves what came before. Most generated pieces of more than a minute or two still lack the sense of an architecture that a listener expects from human composition. Systems that do better usually have structure imposed from outside, for instance by conditioning on a chord chart or a section plan, which is itself a theory of form supplied by the user. They also make memorisation easy to detect. A symbolic output can be compared note by note with every piece in the training set, and researchers routinely measure how often generated sequences reproduce long spans of training data. Such studies have found that copying increases with model size and with how often a passage appears in the training corpus, and that common patterns, scales, cadences, standard progressions, are reproduced constantly, which is unremarkable since all musicians share them, while distinctive passages are reproduced less often but not never. Separating legitimate stylistic commonality from copying of protected expression is, in symbolic music, at least a tractable technical question. Finally, symbolic systems are naturally collaborative. Because their output is editable notes, they fit into composers' existing tools as assistants that suggest continuations, harmonisations, countermelodies or variations. Tools of this kind have been built into music notation and production software and used by composers who want a sparring partner rather than a replacement. That collaborative position is what most people imagined AI composition would look like a decade ago. The next chapter describes why the field moved somewhere else. Chapter 8: Generating Sound Itself Symbolic generation leaves the hardest part of making a record to someone else. A list of notes says nothing about the grain of a voice, the snap of a snare drum in a particular room, or the way a synthesiser pad swells under a chorus. Generating audio directly, sixteen million numbers for three minutes of stereo, was long considered out of reach for a simple reason: the structure of music spans timescales from a fraction of a millisecond to several minutes, and no model could attend to all of them at once. In the space of about five years, a series of technical moves made it possible. Today several commercial systems produce complete songs, with vocals, lyrics, arrangement and production, from a sentence of text. How they work matters for understanding what they can and cannot do, and in particular for understanding what "generating in a style" means for a system that never sees a note. The timescale problem Audio at 44,100 samples per second means that one beat at a moderate tempo spans about twenty-five thousand samples, and a four-bar phrase several hundred thousand. A model that predicts audio one sample at a time, conditioning on everything before, must somehow keep track of the phrase while predicting each individual sample. WaveNet, introduced by Aäron van den Oord and colleagues at DeepMind in 2016, showed that such sample-by-sample prediction could produce remarkably natural speech and piano sounds, using stacks of dilated convolutions to see further back in time. But its piano output, trained on piano recordings, wandered harmonically and had no structure beyond a second or two. The raw audio was simply too long a sequence. The solution that eventually worked was compression into learned tokens. If audio could be converted into a much shorter sequence of discrete symbols, with a decoder that turned those symbols back into high-quality sound, then a sequence model could operate on the symbols, at a manageable length, and leave the fine detail to the decoder. OpenAI's Jukebox, published in 2020 by Prafulla Dhariwal and colleagues, was the first large demonstration. It used a vector-quantised autoencoder at three levels of compression, the coarsest reducing audio to a sequence many times shorter than the original, and trained Transformers to generate tokens at each level, conditioned on artist, genre and lyrics. Trained on 1.2 million songs, Jukebox produced minutes of audio with recognisable singing in the styles of named artists. It was also extremely slow, taking hours of high-end computing to render a minute of music, and its output was noisy and often incoherent. It proved the concept and showed its cost. Neural codecs and language models of sound The key enabling technology turned out to be the neural audio codec. Google's SoundStream, published in 2021 by Neil Zeghidour and colleagues, and Meta's EnCodec, published in 2022 by Alexandre Défossez and colleagues, are neural networks trained to compress audio into a small number of discrete tokens per frame and reconstruct it with high fidelity. They use a technique called residual vector quantisation: the first codebook captures a coarse approximation of each audio frame, the second encodes the error left by the first, the third encodes what remains, and so on. Coarse codebooks hold the broad shape of the sound, while later codebooks hold progressively finer detail. A few seconds of music become a grid of tokens, perhaps fifty frames per second with four or eight codebooks per frame, which is short enough for a Transformer to model over tens of seconds. With codecs available, music generation could borrow the architecture of large language models directly. Google's MusicLM, described in January 2023 by Andrea Agostinelli and colleagues, combined two kinds of token: semantic tokens, derived from a self-supervised audio model and capturing higher-level musical content, and acoustic tokens from SoundStream, capturing fine sound detail. It generated the semantic tokens first, conditioned on a text description, and then the acoustic tokens conditioned on those. To connect text and music, it used MuLan, a model trained to embed music and free-text descriptions in a shared space. The MusicLM team also released MusicCaps, a dataset of about five and a half thousand ten-second clips with descriptions written by professional musicians, which became a standard evaluation set. Meta's MusicGen, released in June 2023 by Jade Copet and colleagues, simplified the design: a single Transformer generating EnCodec tokens directly, with a clever interleaving pattern that let it predict several codebooks per step instead of flattening them into one long sequence. MusicGen was trained on about twenty thousand hours of music that Meta described as licensed, including tracks from commercial stock music libraries. It could be conditioned on text and, in one variant, on a melody extracted from a reference recording as a chromagram, so a user could hum a tune and receive an orchestral or electronic arrangement of it. Meta released the code and model weights, and MusicGen became the standard open baseline for text-to-music research. Diffusion A second family of generators uses diffusion rather than token prediction. A diffusion model is trained to reverse a gradual noising process: given an input that has been corrupted with noise, predict a slightly cleaner version. At generation time it starts from pure noise and repeatedly denoises, guided by a conditioning signal such as a text embedding, until a clean output emerges. Diffusion had transformed image generation in 2021 and 2022 and was adapted to audio in parallel. Riffusion, a hobby project released in late 2022 by Seth Forsgren and Hayk Martiros, made the connection vivid by fine-tuning an image diffusion model to generate spectrogram images, which were then converted to audio. Most serious audio diffusion systems operate in a compressed latent space rather than on raw waveforms or spectrograms. Stability AI's Stable Audio, released in 2023, used a latent diffusion model conditioned on text and on timing information, so that it could generate audio of a specified length, and was trained on music licensed from the stock library AudioSparx. Its open variant, Stable Audio Open, followed in 2024. Diffusion models are well suited to generating coherent textures, loops and sound design, and to editing tasks such as filling in a gap or extending a passage, because they refine a whole stretch of audio at once rather than committing to it left to right. Many current commercial systems are believed to combine a token-based language model for long-range structure with diffusion-based decoders for fidelity, though the leading companies have published few technical details. How good is it, and how would we know? Evaluating audio generators is notoriously difficult, and the difficulty is instructive. For analysis tasks there is a reference answer: the true notes, the true stems, the annotated beats. For generation there is not. A generated piece can be good in countless ways, and bad in ways no single metric captures. The most widely reported automatic measure is the Fréchet Audio Distance, adapted from a similar measure used for generated images. It takes a pretrained audio model, computes embeddings for a large set of generated clips and a large set of real reference clips, fits a simple statistical distribution to each set, and measures the distance between the two distributions. A lower distance means the generated audio, taken as a whole, has embedding statistics more like real music. This captures overall audio quality and plausibility reasonably well, which is why it is useful. It says nothing about whether any individual piece is musically coherent, whether it follows the prompt, or whether it is original, and studies have shown that the result depends heavily on which embedding model is used and which reference set is chosen. A generator can improve its score by producing audio that sounds more like the reference collection in its production characteristics without becoming more musical. Prompt adherence is usually measured with a contrastive audio-text model, scoring how closely the generated audio's embedding matches the text prompt's. Such scores inherit the biases of the model used to compute them: if the audio-text model associates "jazz" mainly with brushed drums and upright bass, a generator that produces those sounds will score well regardless of whether it plays anything resembling jazz harmony. Musically specific checks, for example running a chord recogniser over a generated track to see whether it honoured a requested key, or a beat tracker to see whether it kept the requested tempo, are more targeted and are increasingly used, though they inherit the error rates of the analysis tools described in earlier chapters. Listening tests remain the most trusted method, and they are costly and hard to standardise. Results depend on who listens, how the comparison is framed, and which prompts are chosen. Public preference leaderboards, in which users compare pairs of outputs from different systems and vote, have become common for commercial music generators as they have for language models, and they measure something real, namely what casual listeners prefer in quick comparisons. That is not the same as what a producer, composer or musicologist would value, and it rewards immediate polish over structural interest. The evaluation problem connects directly to the argument of the book: whichever measure is chosen embodies a theory of what makes music good, and systems will be optimised toward it. The song generators The step that brought AI music to public attention was the arrival of systems that generate complete songs with sung lyrics. Suno, a company founded by former machine-learning researchers from the financial data firm Kensho, released its song generator publicly in late 2023. Udio, founded by former Google DeepMind researchers, launched in April 2024. Both produced, from a short text prompt describing style and mood and optionally from user-supplied lyrics, songs of two minutes or more with verses, choruses, vocals, instrumentation and polished production. The results were uneven, with occasional garbled words, odd structural choices and a certain generic sheen, but at their best they were difficult for casual listeners to distinguish from human-made commercial tracks in common genres. Google DeepMind's Lyria models, ElevenLabs Music, and open models such as ACE-Step and YuE followed, and by 2026 the commercial systems generated songs of around four minutes with stems, section-level editing and the ability to extend or replace passages. Neither Suno nor Udio disclosed their training data. In June 2024 the major record labels, Universal Music Group, Sony Music and Warner Music Group, coordinated by the Recording Industry Association of America, sued both companies in United States federal courts, alleging that their models had been trained by copying vast numbers of copyrighted recordings without permission. The labels' complaints included examples of outputs that, given carefully crafted prompts, closely resembled well-known recordings in melody, structure and vocal style. Suno's later court filings acknowledged that its training data included essentially all music files of reasonable quality accessible on the open internet, arguing that such training was fair use under US law. The litigation then moved on several tracks at once. In late 2025, Universal and Warner settled with Udio, and Warner settled with Suno, in each case alongside licensing agreements under which the companies would develop new models trained on licensed material. As of October 2026, Sony's case against Udio and the Sony and Universal case against Suno remained active in US courts, with no final ruling on the central question of whether training on recordings without a licence is fair use. In Germany, the collecting society GEMA, which represents composers and lyricists, won a ruling against Suno from the Munich Regional Court on 31 July 2026. The court found that the model had memorised a small set of songs at issue, including well-known German and international hits, in a form that let simple prompts produce outputs too similar to be coincidental; held that the German text and data mining exception did not cover training where works were retained in reproducible form; and ordered Suno to stop using the works and to pay damages, with the amount yet to be determined. The ruling was not final, and Suno indicated it was considering an appeal. The same court had ruled against OpenAI in November 2025 over the reproduction of song lyrics by its chatbot. Flood and provenance Whatever the courts decide, the volume of generated music has already reshaped the distribution side of the industry. Streaming services accept uploads from distribution platforms that will deliver almost anything for a small fee, and generated tracks can be produced in seconds. The French streaming service Deezer, which developed a detection system for fully AI-generated music and began tagging such tracks, reported in April 2026 that about 44 percent of new tracks delivered to it each day, nearly 75,000, were fully AI-generated, though such tracks accounted for only around 1 to 3 percent of streams, and Deezer said it identified most of the streams on them as fraudulent. By July 2026 Deezer said the share of daily uploads had passed half. Spotify announced in September 2025 that it had removed more than 75 million "spammy" tracks over the preceding year and introduced policies against impersonation and in support of an industry standard for disclosing AI involvement in credits. Detection and provenance have consequently become part of MIR. Detectors exploit artefacts that current generators leave in their output, for instance in the high-frequency detail introduced by neural codec decoders, which is why they work well on known systems and must be retrained as generators change. Watermarking takes the opposite approach, embedding an imperceptible signal at generation time; Google DeepMind's SynthID marks audio from its Lyria models, and Meta released an audio watermarking system called AudioSeal in 2024. Watermarks only help if generators adopt them and if they survive compression, editing and re-recording. Neither detection nor watermarking can answer the deeper question of which training recordings a given output drew on, which is the question at the centre of the legal disputes. What has been learned and what has not Audio generators raise the representational question of Chapter 1 in its starkest form. They do not represent notes, chords, keys or sections explicitly. Their only representation is codec tokens, learned to reconstruct sound. Everything they know about harmony, rhythm and form has been absorbed implicitly from the statistics of those tokens across a very large training set. That they produce songs with coherent verses and choruses at all is a genuine scientific surprise, and evidence that a great deal of musical structure can be learned from audio alone. The same fact explains their characteristic weaknesses. Their control is mostly through text, which is a coarse instrument for musical intent: "upbeat indie pop with a female vocal" leaves almost every musical decision unspecified, and so the model fills those decisions with whatever is most typical of its training data. Their evaluation relies on automatic measures such as the Fréchet Audio Distance, introduced by Kevin Kilgour and colleagues in 2019, which compares the statistics of embeddings of generated and real audio, and on listening tests, which are expensive and vary with listeners and prompts. And their relationship to their training data is opaque. A symbolic generator's output can be compared note by note with its corpus; an audio generator's cannot, except by the indirect methods of melody extraction, cover-song matching and audio fingerprinting, which are themselves MIR techniques and which are now being turned on generated music in court cases and platform audits. The final chapter takes up what that means for the oldest ambition in algorithmic composition: writing new music within a style. Chapter 9: Writing Within a Style "Write me a piece in the style of" is the oldest request in algorithmic composition and the most commercially important one in AI music today. A film editor wants thirty seconds of tense string writing that sounds like the score of a particular kind of thriller. A game studio wants endless variations of a village theme in the idiom of its existing soundtrack. A songwriter wants a demo arrangement in the manner of a current chart hit. A music teacher wants a dozen new chorales to harmonise that follow the conventions of eighteenth-century Lutheran hymnody. Each of these is generation under stylistic constraint: produce something new that obeys a set of conventions, some explicit and some implicit, without simply reproducing an existing piece. This chapter argues that the problem has three parts, each of which tests whether a system's theory of music is adequate: specifying the constraint, enforcing it, and distinguishing style from copying. What a style is Musicologists have long tried to define style precisely, and the most influential definition comes from Leonard Meyer, whose 1989 book Style and Music described style as a replication of patterning that results from a series of choices made within some set of constraints. The constraints come in layers. Some are laws in the sense of acoustics and perception. Some are rules shared by a whole tradition, such as the treatment of dissonance in common-practice harmony or the twelve-bar form of the blues. Some are strategies characteristic of a period, a school or an individual: a composer's habitual way of approaching a cadence, a producer's signature drum sound, a singer's phrasing. Style lives in the pattern of choices, not in any single feature. This layering explains why "in the style of" is so hard to specify. A request for "a piece in the style of Bach" might mean a four-part chorale observing Baroque voice-leading rules; it might mean a keyboard invention with two contrapuntal voices and motivic imitation; it might mean a modern track that borrows the sound of a harpsichord and a walking bass. Each is a different set of constraints at different layers. Humans resolve the ambiguity from context. A system must be told, or must guess, and its guess will be shaped by what its training data associated with the words in the prompt. Specifying the constraint The available ways of telling a system what to produce form a spectrum from explicit and musical to implicit and linguistic. Table 4 summarises the main forms of control in current systems and what each can and cannot pin down. Table 4. Forms of stylistic control in music generation. Control Example Pins down well Leaves open Hard rules Counterpoint constraints Voice leading, range, cadences Expressive character Symbolic conditioning Chord chart, melody, form plan Harmony, melody, structure Sound and production Audio reference Melody hummed or a sample clip Contour, timbre, groove Long-range intent Text prompt "Moody synthwave, female vocal" Genre, mood, instrumentation Nearly every specific note Fine-tuning on a corpus Model adapted to one catalogue Overall idiom of the corpus Which corpus features matter At one end, hard rules state the constraint as music theory: no parallel fifths between outer voices, every phrase ends on a cadence, the melody stays within an octave and a half. Systems such as Ebcioğlu's chorale harmoniser and the constraint-based Markov methods of the Sony CSL group work this way, as do modern techniques that guide a neural model's sampling so that only tokens satisfying the rules can be chosen. Rules give precise control over what they cover and none at all over what they omit. They also require someone to have formalised the style, which is feasible for textbook idioms and far harder for, say, the groove of a particular funk band. Symbolic conditioning supplies part of the music and asks the system to fill in the rest. DeepBach let a user fix any notes and generated the others consistently. Many symbolic Transformers accept a chord progression, a melody, or a structural template, and generate accompaniment, harmonisation or continuation. MusicGen's melody conditioning and similar features in commercial tools let a hummed tune or a reference recording supply the contour while the model supplies the arrangement. Conditioning of this kind narrows the gap between intent and output, and it fits the way musicians actually work, by sketching some elements and elaborating others. Text prompts sit at the opposite end. They are expressive about genre, mood and instrumentation, the categories that people readily put into words and that appear in tags and captions in training data, and nearly silent about the specific musical choices that distinguish one piece from another. The prompt "a melancholy piano ballad in a minor key" is satisfied by an enormous range of pieces, and the model's choice among them reflects the statistical centre of what its training data labelled that way. This is why text-to-music output can sound generic even when it is technically polished: the request specified a region of style space, and the model returned something near its middle. Fine-tuning on a corpus is the most implicit control of all. A model adapted to a composer's catalogue, a game's soundtrack or a library's house sound will reproduce the idiom of that corpus without anyone having stated what the idiom is. That is powerful, since it captures layers of style nobody could formalise, and also risky, for reasons that the last section of this chapter takes up. Enforcing the constraint Even when the constraint is specified, enforcing it is not automatic. Neural sequence models generate by sampling from probability distributions, and a model trained on a style will usually, but not always, produce output that obeys its conventions. In the symbolic domain, researchers have measured how often generated chorales violate voice-leading rules or how often generated pop melodies fall outside the scale implied by the chords, and have found that larger and better-trained models violate them less, without reaching zero. In the audio domain such measurements are harder, because rules must first be checked by an analysis system, a transcriber or chord recogniser of the kind described in earlier chapters, which introduces its own errors. Several techniques tighten control. Constrained decoding forbids tokens that would break a rule. Classifier guidance and its diffusion relative, classifier-free guidance, push generation toward outputs that score highly on a target attribute; the strength of the push trades adherence against diversity and quality, and too strong a push produces caricature. Inpainting and editing tools, now common in commercial generators, let a user regenerate one section while keeping the rest, so that a human enforces the constraint through iteration. And hybrid pipelines split the work: a symbolic model or a human writes the notes, and an audio model renders them in a target sound, which keeps the musically important decisions in an editable and checkable form. The most reliable enforcement still comes from people. Most professional uses of AI generation as of 2026 involve a human selecting among many outputs, editing, rearranging and combining them, in a workflow that resembles curating and producing more than commissioning. That reflects the state of the technology rather than a principle: systems are much better at producing plausible material than at producing material that satisfies a specific musical intention on the first attempt. Three worked cases The abstract framework becomes clearer when applied to concrete commissions. Consider three requests, each typical of real use, and what a system needs to satisfy them. The first is a set of new chorale harmonisations for teaching, in the style of the Baroque Lutheran chorale. The constraint here is unusually explicit. The style has been codified in textbooks for two centuries, the corpus is well defined, the musical surface is symbolic, and the rules, about voice ranges, doubling, parallel motion, cadence types and the resolution of dissonance, can be stated and checked mechanically. A symbolic model trained on the corpus, combined with rule checking, is well matched to this task, and the results can be evaluated by any competent theory teacher. The main risk is subtle: a model can satisfy every textbook rule and still produce harmonisations that feel mechanical, because the corpus contains expressive choices, a chromatic passing chord at a word of particular weight, that the rules do not describe. The theory in the rules is the textbook theory, and the corpus contains more than the textbook. The second is background music for a video game in the idiom of the game's existing soundtrack, varying continuously with the player's situation. Here the style is defined by a small corpus, perhaps an hour of music by one composer, and the constraint is partly musical, matching its harmonic language, instrumentation and pacing, and partly functional, transitioning smoothly when the game's state changes. Game audio has used procedural and adaptive techniques for decades, typically by having composers write layers and segments that a system recombines according to rules. Generative models can extend this by producing new variations, but with only an hour of training material the risk of reproducing the existing score verbatim is high, and the composer's ownership of that material makes memorisation a direct concern. A practical design keeps the composer in charge of the musical vocabulary and uses generation for variation within explicitly defined bounds. The third is a demo track for a pop songwriter, "something like" a named current hit, to pitch to an artist. This is the case commercial text-to-song systems are built for, and it is the hardest to handle well. The style is defined by reference to one specific recording, which is protected, and by a broad genre whose conventions are shared. What the songwriter actually wants is usually the production feel, the energy and the genre conventions, not the melody or lyric of the reference. But a model prompted with a specific track, or with a description that points strongly at it, has every statistical incentive to approach the original. Commercial systems therefore typically refuse prompts naming artists, and those with licensing agreements limit such references to artists who have opted in. The songwriter can still describe the target in musical terms, tempo, instrumentation, vocal register, chord vocabulary, and a system that accepts such descriptions, or accepts a reference track only for analysis of its tempo and instrumentation, can serve the legitimate need without the copying risk. Across all three cases, the systems that work best are those whose constraints are stated at the musical layer that matters for the task, and whose relationship to their training material is understood. That points directly to the question this chapter has been building toward. Style or copy The hardest part of generation in a style is the boundary with reproduction. Every musical style is made of shared materials, and imitation of style has always been legitimate; composition students write pastiche Bach chorales, session musicians are hired to play "like" particular players, and whole genres are defined by shared conventions. Copyright, in most jurisdictions, protects particular expression, a specific melody, lyric or recording, and not style, technique or genre. A generator that writes a new song with the instrumentation, groove and harmonic language of 1980s synth-pop is doing what musicians have always done. A generator that reproduces a recognisable melody from a 1980s hit, or a recording's distinctive vocal performance, is doing something else. The difficulty is that learning-based systems do not maintain this distinction internally. A model trained to predict the next token in a large corpus is rewarded for reproducing whatever is predictable, and for passages that appear many times in the corpus, or that are highly distinctive and seen in full, the most predictable continuation can be the original. Research on memorisation in image and text models has shown that duplicated training examples are much more likely to be reproduced, and that larger models memorise more. Studies of music models have found similar patterns, with the added complication that musical similarity is itself hard to define. Two melodies can share every pitch and differ entirely in rhythm, or share a contour and differ in every interval. Deciding when a generated output is too close to a training piece requires a theory of melodic similarity, which is an old MIR research topic in its own right, and courts in music plagiarism cases have shown how contested such judgements can be even for human-written songs. The Munich ruling against Suno described in the previous chapter turned precisely on this point. The court did not hold that learning a style was unlawful. It held that the specific works at issue had been retained in the model in reproducible form, as shown by outputs whose similarity to the originals was too extensive to be coincidental, and that this retention was a reproduction falling outside the text and data mining exception. Whether courts in other jurisdictions reach similar conclusions remains open, but the reasoning makes memorisation, rather than stylistic imitation, the legally decisive question, and it makes the MIR tools that can detect memorisation, melody extraction, cover-song identification and audio fingerprinting, central to the answer. Developers have several partial defences. They can deduplicate training data so that no work is seen many times. They can train on licensed corpora, which reduces the legal exposure without eliminating memorisation. They can filter outputs, comparing each generated piece against a reference database using the same fingerprinting and similarity techniques that streaming services use to identify recordings, and reject or regenerate outputs that match too closely. And they can limit the use of named artists in prompts, which most commercial services now do, though the licensing deals of 2025 and 2026 have also introduced opt-in features in which artists consent to models trained on, or styled after, their work, in exchange for payment. Each defence reduces the problem; none eliminates it, because a sufficiently close imitation of style can approach a particular work without copying any single passage exactly. The musician in the loop What would a good system for writing within a style look like? The argument of this book suggests an answer. It would represent the style at the layers that matter for the user's purpose, not only as a region of an embedding space but in terms a musician can inspect and adjust: harmony, form, rhythm, orchestration, sound. It would accept constraints in the forms musicians already use, sketches, chord charts, reference tracks, rules, rather than only in prose. It would make its relationship to its training material checkable, so that a user could know whether a passage was new or remembered. And it would be evaluated not only by automatic distances and preference ratings but by the communities whose styles it works in, as the folk-rnn experiment did with traditional musicians. Some current tools approach this, particularly symbolic assistants inside notation and production software and hybrid systems that pair symbolic generation with audio rendering. The large text-to-song systems are further from it, because their power comes from absorbing style implicitly from enormous data, which is exactly what makes their theory of music hard to inspect. The tension between those two approaches, explicit and controllable versus implicit and fluent, is the main open question in music generation, and it is the same tension that runs through every analytical task in the earlier chapters. Conclusion The history traced in this book can be told as a story of steady technical progress, and for good reason. Solo piano transcription went from a problem with a "glass ceiling" to one that is close to solved in about five years. Source separation went from a curiosity to a feature of consumer apps and a tool for restoring some of the most famous recordings ever made. Beat trackers, chord recognisers and tagging systems became reliable enough to run unattended across catalogues of tens of millions of tracks. Generative models went from a few seconds of noisy piano to complete songs that many listeners cannot tell from human work. Each of these advances was real, and each depended on the same ingredients: larger and better-labelled data, architectures able to see more of the music at once, and shared benchmarks that let the field measure itself. Told only that way, though, the story misses what made the difference between systems that worked and systems that merely scored well. The argument of this book has been that every system that analyses or generates music carries a theory of what music is, and that the quality of the system is bounded by that theory. Looking back across the chapters, the theory turns out to live in four places, and each points to something a reader building, buying, evaluating or regulating these systems should do differently. Where the theory lives The first place is the representation. Whether a system sees a chromagram, a constant-Q spectrogram, a piano-roll or codec tokens decides which differences it can perceive. A chord recogniser built on pitch classes cannot hear inversions; a twelve-bin grid cannot hear makam intervals; a MIDI vocabulary cannot express a sitar's glide. The practical question to ask of any system is simply: what can this representation not see, and does the task need it? The second is the ground truth. A learned system knows only what its labels taught it, and labels are interpretations. The Beatles and Billboard chord annotations encode one kind of harmonic listening; the four-stem MUSDB18 split encodes one theory of a mix; GTZAN's ten genres encode one coarse map of popular music. Where the labels are rich, consistent and representative, as with MAESTRO's piano data, systems become excellent. Where they are thin or biased, systems inherit the gaps. Anyone relying on an MIR system should ask what it was trained on and whether that resembles the music they care about, and should be suspicious of any system whose makers cannot say. The third is the metric. Note-onset tolerances, signal-to-distortion ratios, accuracy on a fixed test set and Fréchet distances are all claims about what matters, and all can be optimised without improving the musical capability they stand for. The genre episode showed how a field can climb a benchmark while learning the wrong thing; source separation showed how a good benchmark can drive real progress while quietly narrowing the definition of the task. The habits that came out of those episodes, artist-filtered splits, cross-collection testing, interventions that probe whether a system responds to the right cues, and listening tests by people who know the music, are not academic niceties. They are the only way to know what a system has learned. The fourth is the vocabulary of control. In generation, the theory of music is most visible in what a user is allowed to specify. Hard rules make one theory explicit; chord charts and melodies make another; text prompts make a theory that genre and mood are what matter and that the notes can be left to the model. Systems controlled only through text inevitably return the statistical centre of the region a prompt describes, which is why so much generated music sounds competent and anonymous. What remains unsettled Several questions are open as of late 2026 and will shape the field for years. The first is legal and economic. Whether training on copyrighted recordings without a licence is lawful remains undecided in the United States, where the major outstanding cases had not reached final judgment, while a German court has found that a commercial generator memorised specific works and ordered it to stop using them. In parallel, the largest labels have chosen licensing over litigation with some companies, which points toward a market in which generators are trained on licensed catalogues and pay for the privilege. That outcome would favour well-capitalised companies with access to major-label catalogues, and it leaves open how the composers, session musicians and independent artists whose work makes up most of any catalogue will be compensated. A musicians' union lawsuit filed in 2026, alleging that session players were not paid when their recordings were licensed for AI training, shows that the question is far from settled even within licensed arrangements. The second is scientific. Audio generators learned an impressive amount of musical structure from sound alone, but how much they learned, and how much they memorised, is poorly understood. Measuring that requires analysis tools, transcription, chord recognition, structural segmentation, similarity measures, robust enough to run on generated audio and trustworthy enough to support conclusions in court. In other words, the generative side of the field now depends on the analytical side for its accountability, which gives MIR's older tasks a new urgency. The third is cultural. The data that trains these systems overrepresents some musical traditions and underrepresents most of the world's music. Generators trained on that data will reproduce its centre of gravity, and as generated tracks flood streaming services, that centre may become heavier still. The work of projects that built corpora and tools around other traditions' own theories of pitch, rhythm and form is not a side issue. It is the only route to systems that serve more than one kind of music. What follows for the reader For musicians, the practical conclusion is that these systems are best treated as instruments with a particular built-in theory, not as neutral listeners or composers. A separation tool is superb for isolating a vocal from a pop mix and less reliable on a dense orchestral recording. A transcription tool's piano output may be trustworthy at the note level and still need a human to make a score. A generator will give you the typical; the distinctive still has to come from somewhere, and the most useful systems are the ones that let you supply it. For engineers and researchers, the conclusion is that musicological knowledge remains an engineering input. It moved, when deep learning arrived, from hand-designed features into tokenisations, label vocabularies, dataset construction, evaluation design and control interfaces. The work in those places is less glamorous than a new architecture, and it is where most of the difference between a system that scores and a system that understands is decided. For policy-makers, platforms and rights holders, the conclusion is that the central questions about AI music are technical questions about what models learned and retained, and that MIR already has many of the tools needed to answer them. Provenance, attribution and memorisation testing will not be solved by detection classifiers alone. They need similarity measures with a defensible musical theory behind them, applied transparently. The founders of MIR wanted to search music by its content. A quarter of a century later, the same field is being asked to tell new music from remembered music, to say whose work a model learned from, and to let people make music with machines without surrendering the choices that make music theirs. Those are harder questions than retrieval, and they turn on the same thing retrieval always did: having a good enough theory of what music is to know what you are looking for. Further Reading Meinard Müller, Fundamentals of Music Processing: Using Python and Jupyter Notebooks, 2nd edition (Springer, 2021). The standard textbook for the analytical side of the field, covering representations, alignment, structure, chords, tempo and beat, and retrieval, with exercises. The clearest place to learn how spectrograms, chromagrams and self-similarity matrices actually work. Alexander Lerch, An Introduction to Audio Content Analysis: Music Information Retrieval Tasks and Applications, 2nd edition (Wiley-IEEE Press, 2023). A broad and accessible survey of MIR tasks, from low-level features to genre and mood, with attention to both classical methods and machine learning. Emmanouil Benetos, Simon Dixon, Zhiyao Duan and Sebastian Ewert, "Automatic Music Transcription: An Overview," IEEE Signal Processing Magazine 36, no. 1 (2019).** A compact review of transcription methods and challenges written just as neural methods were taking over. Bob L. Sturm, "A Simple Method to Determine if a Music Information Retrieval System Is a 'Horse'," IEEE Transactions on Multimedia 16, no. 6 (2014).** The paper that made shortcut learning in genre recognition impossible to ignore, and a model of how to probe what a system has really learned. Zafar Rafii, Antoine Liutkus, Fabian-Robert Stöter, Stylianos Ioannis Mimilakis, Derry FitzGerald and Bryan Pardo, "An Overview of Lead and Accompaniment Separation in Music," IEEE/ACM Transactions on Audio, Speech, and Language Processing 26, no. 8 (2018).** A thorough survey of separation from early methods to the first deep-learning systems. Jean-Pierre Briot, Gaëtan Hadjeres and François-David Pachet, Deep Learning Techniques for Music Generation (Springer, 2020).** A systematic analysis of symbolic and audio generation architectures, representations and control strategies, written by researchers responsible for several of the systems it discusses. David Cope, Computer Models of Musical Creativity (MIT Press, 2005).** Cope's own account of recombinancy and his Experiments in Musical Intelligence, useful for understanding the long history of the style-versus-copying debate. Leonard B. Meyer, Style and Music: Theory, History, and Ideology (University of Pennsylvania Press, 1989).** The musicological foundation for thinking about style as a pattern of choices within layered constraints; still the most useful framework for asking what "in the style of" should mean. Cheng-Zhi Anna Huang et al., "Music Transformer: Generating Music with Long-Term Structure," International Conference on Learning Representations (2019).** The paper that brought relative attention to symbolic generation and showed what long-range structure in generated music could look like. Jade Copet et al., "Simple and Controllable Music Generation," Advances in Neural Information Processing Systems 36 (2023).** The MusicGen paper, a clear account of codec-token music generation with text and melody conditioning, with openly released models. Hashtags: #MusicInformationRetrieval #MIR #ArtificialIntelligence #MusicAI #AudioAnalysis #MusicGeneration #AIMusicGeneration #MusicTechnology #AudioSignalProcessing #DigitalSignalProcessing #MusicMachineLearning #DeepLearningForMusic #MusicTranscription #AutomaticMusicTranscription #BeatTracking #TempoEstimation #ChordRecognition #MusicGenreClassification #MusicSourceSeparation #AudioFingerprinting #MusicRecommendation #SpectrogramAnalysis #MusicRepresentation #SymbolicMusicGeneration #MIDIGeneration #NeuralAudioSynthesis #GenerativeMusicModels #MusicTransformers #DiffusionModels #MusicCopyright

Download the Book (PDF): Introduction On a public blockchain, a trade is announced before it happens. When someone asks a decentralized exchange to swap one token for another, the request does not travel privately to a matching engine. It is broadcast to a network of computers, sits in a shared waiting area, and is visible to anyone running software that watches that waiting area. For a few seconds, sometimes less, the world knows what the trader intends to do, roughly how much they are willing to lose to price movement, and how much they are paying to be included. Then someone else decides where in the next block that trade will go. That second fact is the subject of this book. Every block on a blockchain is an ordered list of transactions, and whoever assembles the list chooses the order. Order matters because the transactions interact. A trade that buys a token pushes its price up for the trade that comes after it. A loan that becomes undercollateralized can be liquidated by the first party to call the liquidation function, and only the first. An arbitrage between two exchanges pays whoever executes it before the gap closes. The power to choose the order is therefore the power to choose who profits, and that power has a market value. The profit available to whoever controls ordering, inclusion and exclusion within a block is what the field calls MEV. The term began as miner extractable value, coined in 2019 by Philip Daian and colleagues in a paper titled "Flash Boys 2.0," a deliberate nod to Michael Lewis's account of high-frequency trading on Wall Street. When Ethereum moved from proof-of-work mining to proof-of-stake validation in September 2022, miners ceased to exist on that network, and the term was quietly widened to maximal extractable value. The rename was more honest than it looked. It acknowledged that the value does not belong to any particular role. It belongs to whoever sits at the point where transaction ordering is decided, and the history of MEV is largely a history of that point moving from one set of hands to another. The controlling claim This book argues one thing and tries to argue it carefully. MEV is not a bug that better code will remove, nor a crime wave that enforcement will end. It is a rent created by two design choices that are hard to give up: that transactions are public before they are final, and that some party must decide their order. Given those choices, the rent exists, and the only real questions are who captures it, how much of it leaks out as harm to ordinary users, and how much it distorts the structure of the system that produces it. Every major development in the field, from the bidding wars of 2020 through Flashbots, proposer-builder separation, order flow auctions and encrypted mempools, is best understood as an attempt to move the rent somewhere less damaging. None of them makes it disappear, and the clearest way to judge each one is to ask where it puts the money and who it asks us to trust. That framing excludes a good deal. This is not a trading manual, and it contains no instructions for building bots or running attacks; the mechanics are described at the level needed to understand the economics and the harms, not to reproduce them. It does not survey every chain or catalogue every protocol. It concentrates on Ethereum, because that is where MEV was first named, measured and institutionally addressed, and where the most consequential design decisions are still being made. Other systems appear where they teach something Ethereum does not. Why a reader outside crypto should care It is tempting to treat MEV as an internal matter for a niche industry. Three reasons argue otherwise. The first is that MEV is a clean natural experiment in market microstructure. Traditional finance has spent decades arguing about front-running, payment for order flow, latency arbitrage and the fairness of the queue at the exchange. Those debates are muddied by private data and regulatory opacity. On a public blockchain, almost everything is visible after the fact: every trade, every bid for priority, every payment to the party that built the block. Researchers can watch a market for ordering rights operate in close to full daylight. What they have found sheds light on markets generally, including some uncomfortable lessons about how quickly a competitive free-for-all turns into an oligopoly. The second is that MEV is where the abstract promises of decentralization meet money. Blockchains are sold as neutral infrastructure that no single party controls. MEV is the force that pulls against that neutrality, because sophisticated actors who can extract more value can afford to pay more for the right to build blocks, and their advantage compounds. Whether a public ledger can stay credibly neutral when ordering is worth billions is not a technical footnote. It is the central governance question of the technology. The third is simple consumer protection. A meaningful share of MEV comes directly out of the pockets of ordinary traders, through worse prices that most of them never notice. The tax is invisible because it is folded into the execution price, but it is real, and it falls hardest on the least sophisticated users. Any honest account of decentralized finance has to reckon with it. How the argument unfolds The first chapter lays out the basic machinery: what a mempool is, how a block gets built, and the main forms MEV takes. It separates the forms that perform a useful function, such as arbitrage that keeps prices aligned and liquidations that keep lending markets solvent, from the forms that are pure transfers from users, such as sandwich attacks. The second chapter describes the searchers, the automated systems that hunt for MEV, and the environment that produced them. It covers the early period of open bidding wars, the "dark forest" image that came to define the space, and what those wars cost the network in congestion and failed transactions. The third chapter turns to the users who pay. It works through the economics of the sandwich attack and the subtler, larger cost borne by liquidity providers whose prices go stale every time the outside market moves. The fourth chapter tells the story of Flashbots, the research organization that built a private channel for searchers to bid directly for ordering, ended the public bidding wars, and in doing so made MEV extraction more orderly, more efficient and more institutionalized all at once. The fifth chapter follows MEV through Ethereum's transition to proof-of-stake, where Flashbots' MEV-Boost became the de facto way blocks are built. It examines the separation of proposers from builders, the trusted relays that sit between them, the concentration of block building in very few hands, and the episodes that exposed the system's fragility, from censorship of sanctioned addresses to an exploit of the relay itself. The sixth chapter follows the money downstream to order flow, the stream of user transactions that builders and searchers compete to see first. It asks whether returning MEV to the users who generate it, through auctions and rebates, solves the problem or entrenches a new set of intermediaries. The seventh chapter examines the cryptographic and hardware-based efforts to hide transactions until their order is fixed: threshold-encrypted mempools, trusted execution environments, and batch auctions that make order irrelevant. These are the most ambitious attempts to democratize block building, and their limits are as instructive as their promise. The eighth chapter looks at the protocol-level redesigns now arriving or under debate, including Ethereum's enshrined proposer-builder separation, inclusion lists that constrain censorship, proposals to burn MEV rather than distribute it, and the choices made by rollups and other chains that control their own sequencing. The conclusion draws out what follows from all of this for builders of protocols, for users, and for anyone deciding how much to trust a system that runs on public ordering. A note on numbers MEV is measured, but imperfectly. Arbitrage and liquidation profits leave clear on-chain traces. Sandwich attacks can be detected by pattern. But a large and growing share of MEV happens across venues, between a blockchain exchange and a centralized one, where only one leg is visible on-chain. Private deals between builders and order flow providers leave little trace at all. Every published total is therefore a lower bound, and different dashboards disagree. Where this book cites a figure, it names the source and treats the number as an indication of scale rather than an exact accounting. Where reliable figures do not exist, it says so. The direction and structure of the problem are clear even where the precise size is not. Chapter 1: Who Orders the Transactions A blockchain is often described as a ledger, and the description hides the most important thing about it. A ledger records what happened. A blockchain also decides what happens, because the order in which it records transactions determines their outcomes. Two people trying to buy the last unit of something cannot both succeed; the ledger's order settles who does. Two traders hitting the same pool of liquidity get different prices depending on who goes first. To understand MEV, one has to start with the plain mechanics of how a transaction travels from a user's wallet into a block, and who has a say along the way. The waiting room When a user signs a transaction on Ethereum, their wallet sends it to a node, a computer running Ethereum software. The node checks that the transaction is well formed and that the sender can pay for it, then passes it to its peers, which pass it to theirs. Within a second or two, the transaction has propagated across thousands of nodes. Each of them holds it in a local pool of pending transactions. These local pools, taken together, are what people mean by the mempool. There is no single mempool, only many overlapping copies that are roughly in sync. Pending transactions are public. Anyone can run a node, and a node sees everything its peers send it. The transaction reveals the sender's address, the contract being called, the function and its arguments, the amount of value attached, and the fee offered. For a trade on a decentralized exchange, the arguments typically include the tokens, the amount, and a minimum acceptable output, which tells observers exactly how much adverse price movement the trader will tolerate before the trade reverts. A transaction stays in the mempool until a block producer includes it. On Ethereum today, a new block is produced in each twelve-second slot. In principle, the producer selects transactions from the mempool and arranges them in a block. The producer is paid in two ways: a share of the fees users attach, and whatever additional value the producer can capture from the content and order of the block itself. Since the London upgrade of August 2021, Ethereum's fee mechanism, defined in EIP-1559, splits the fee into a base fee that is burned, and a priority fee, or tip, that goes to the block producer. The base fee rises and falls with demand for block space. The tip is the user's lever for asking to be included sooner. The traditional mental model, inherited from Bitcoin, is that block producers sort transactions by fee and include the highest-paying ones first. That model was always a convention rather than a rule. The protocol does not require any particular order. It requires only that each transaction be valid at the point where it appears. A block producer can put transactions in whatever order it likes, insert its own transactions anywhere, and leave out any transaction it chooses. This freedom is the root of everything that follows. Where the value comes from If ordering were economically neutral, the freedom to order would be worthless. It is not neutral because decentralized finance runs on contracts whose behaviour depends on shared state that every transaction can change. Consider an automated market maker, the design behind Uniswap and its many descendants. Instead of matching buyers with sellers, an automated market maker holds reserves of two tokens in a pool and quotes prices from a formula. In the simplest and most widely used version, the constant-product design, the product of the two reserves must stay constant (before fees) after every trade. If a pool holds 1,000 units of token A and 1,000 units of token B, a trader who wants to buy B must add enough A that the product of the new reserves equals the old product of one million. Buying a small amount moves the price a little. Buying a large amount moves it a lot. Each trade changes the price that the next trader faces. This produces the first source of MEV: arbitrage. Prices for the same asset diverge across venues constantly. A large trade on one exchange pushes its price away from the price on another exchange, or away from the price on a centralized exchange such as Binance or Coinbase. The divergence is an opportunity: buy where it is cheap, sell where it is dear. On a blockchain, an arbitrage between two on-chain pools can be done in a single atomic transaction that either completes entirely or reverts, so the arbitrageur takes no inventory risk if they are included in the right place. The right place is immediately after the trade that created the divergence. Whoever gets that slot captures the spread. The second source is liquidation. Lending protocols such as Aave, Compound and Maker allow users to borrow against collateral. If the collateral's value falls below a threshold, the loan becomes eligible for liquidation, and anyone may repay part of the debt in exchange for the collateral at a discount. The discount is the incentive that keeps the protocol solvent. But only the first liquidator to act gets it. When a price oracle update makes a batch of loans liquidatable, a race begins for the positions immediately following that update. The third source is less benign. Because the mempool shows a pending trade before it executes, an observer can trade ahead of it. If a user is about to buy a large amount of a token, the observer buys first, pushing the price up; the user's purchase then executes at a worse price and pushes it up further; and the observer sells immediately after, at the higher price. This is the sandwich attack, named for the user's trade trapped between the attacker's two. It is a direct transfer from the user to the attacker, limited only by the user's slippage tolerance, the maximum price movement the user agreed to accept. Chapter 3 examines it in detail. Around these three core forms sit many others. Back-running places a transaction directly after a target, as an arbitrageur does after a large swap, without harming the target. Front-running in the broad sense covers any transaction placed ahead of a target in order to exploit it, from copying a profitable transaction outright to beating a user to a scarce resource such as a newly launched token or a mispriced non-fungible token. Just-in-time liquidity adds liquidity to a pool immediately before a large trade and removes it immediately after, collecting the trading fee that would otherwise have gone to passive liquidity providers. Cross-domain MEV links opportunities across different chains, rollups and centralized exchanges, where the block producer of one domain cannot see or control the others. There is also the theoretical extreme. If the value available in a past block exceeds the reward for building the next one, a block producer has an incentive to rewrite history: to fork the chain and reorder the earlier block to capture its MEV. Daian and colleagues called these time-bandit attacks. They have not occurred at scale on Ethereum, partly because proof-of-stake makes reorganizations costly and partly because the stakes have not justified the risk. They matter as a reminder that MEV is not just a tax on users. In the limit it is a threat to the consensus that makes the chain a chain. Useful and harmful It is common to talk about MEV as if it were simply bad. It is more precise to say that MEV is a payment for a service, and that sometimes the service is valuable and sometimes it is theft. Arbitrage performs a function. Automated market makers have no idea what the "true" price of an asset is. They know only their reserves. When the market moves, the pool's price is wrong until someone trades against it, and the arbitrageur who does so is what keeps on-chain prices tethered to the rest of the world. Without arbitrage, decentralized exchanges would quote stale prices indefinitely. Liquidation, similarly, keeps lending protocols solvent. The protocol pays a bounty to whoever repays bad debt, and competition for that bounty ensures debt is repaid fast. But both functions have a cost, and the cost falls on someone. The arbitrageur profits by trading against a stale price, and the counterparty to that trade is the pool's liquidity providers, who sell at the old price to someone who knows the new one. The liquidator profits from a discount that comes out of the borrower's collateral. These are not crimes; they are the designed incentives. Still, their size is a policy choice, and protocols that pay too much for these services are leaking value to whoever controls ordering. The sandwich attack performs no function. It does not move prices toward accuracy, keep anyone solvent, or provide liquidity. The attacker inserts trades whose only purpose is to make the victim's execution worse and capture the difference. It is the canonical harmful MEV, and much of the effort described in later chapters is aimed at eliminating it while preserving the useful forms. Table 1 summarizes the main forms by where the value comes from and who bears the cost. Table 1. Main forms of MEV and who bears the cost. Form Source of profit Who bears the cost Useful function? DEX-to-DEX arbitrage Price gaps between on-chain pools Liquidity providers in the stale pool Yes: aligns prices CEX-DEX arbitrage Gap between on-chain pool and off-chain price Liquidity providers Yes, but at high cost to LPs Liquidation Protocol bounty on undercollateralized loans Borrower's collateral Yes: keeps lending solvent Sandwich attack Victim's slippage tolerance The trader being sandwiched No Just-in-time liquidity Fees on a single large trade Passive liquidity providers Marginal: can improve the trader's price Time-bandit reorg MEV in a past block Every user of the chain No: undermines consensus The table leaves out one important point: these forms are not separable in practice. The same bot may run arbitrage, liquidations and sandwiches. The same bundle of transactions may combine an arbitrage with a liquidation. The infrastructure built to make arbitrage efficient also makes sandwiching efficient. This is why the field's mitigations are rarely surgical. Shutting off public visibility of pending trades, for example, stops sandwiches, but it also affects the information that arbitrageurs and liquidators rely on. The block producer's dilemma If the block producer controls ordering, why does the producer not simply capture all of this value directly? In principle, it could. A miner on proof-of-work Ethereum could have run its own arbitrage and liquidation bots and inserted their transactions at the ideal positions in its own blocks. In practice, finding MEV is hard and specialized. The opportunities are fleeting and diverse. Detecting them requires simulating thousands of pending transactions against the current state of hundreds of contracts, modelling their interactions, and computing optimal trades, all within a fraction of the time between blocks. Mining pools were good at mining. They were not, by and large, quantitative trading firms. The skills that make a good block producer, such as reliable hardware, good network connectivity and capital to stake or mine, are different from the skills that find MEV. This division of labour produced a market. Specialists, called searchers, find MEV opportunities and pay block producers for the ordering that lets them capture it. Block producers sell ordering to the highest bidder. The question that defines the field's history is how that payment is made. In the earliest period, it was made through the public fee mechanism, by bidding up gas prices in the open mempool. That produced chaos, examined in the next chapter. Flashbots replaced it with a private, sealed-bid auction. Proof-of-stake added another layer, separating the party that assembles blocks from the party that proposes them. Each design changed who earned the money and who held power. A last conceptual point clarifies much that follows. MEV is often described as if it were a fixed quantity sitting in each block, waiting to be extracted. That is a useful simplification but a misleading one. The amount of MEV in a block depends on who is looking and what they know. A searcher with a fast connection to a centralized exchange can see price movements there before they show up on-chain, and can therefore capture arbitrage that a slower searcher cannot. A builder that receives private transactions directly from a popular wallet can see trades that never appear in the public mempool. A validator that runs its own trading desk can combine information from its own business with its control over a block. MEV is not a property of the block alone. It is a property of the block together with the information and capabilities of the party building it. This matters because it means MEV rewards informational and infrastructural advantage. Whoever can see more, compute faster and connect to more venues can extract more. That party can then afford to pay more for the right to build blocks. And that, in turn, means block building tends to concentrate in the hands of whoever has the largest information advantage. The researchers who coined the term worried about this from the start: that MEV would push the system toward centralization not because anyone designed it that way, but because the economics reward scale. Later chapters show how that worry played out. It also matters because it changes how one should think about "eliminating" MEV. Some MEV is intrinsic to the existence of on-chain markets that need to be brought into line with the rest of the world; arbitrage against a stale price will exist as long as prices move between blocks. Some MEV is created by design choices, such as public mempools that reveal trade intentions or automated market makers that quote prices without regard to how informed the counterparty is. Mitigations aimed at the first kind can at best redistribute it. Mitigations aimed at the second kind can genuinely shrink it. Keeping the two apart is the beginning of clear thinking about the subject. One further distinction will matter repeatedly. Some MEV is atomic: the entire opportunity can be captured within a single transaction or bundle on a single chain, with no risk beyond the fee. An arbitrage between two Ethereum pools is atomic. The searcher buys in one pool and sells in the other within the same transaction, and if the profit is not there by the time the transaction runs, the whole thing reverts. A sandwich is nearly atomic, since its three transactions execute back to back in one block. Atomic MEV is easy to detect after the fact, because every leg is on-chain, and it is accessible to anyone with good software, because it requires no inventory and no outside position. Other MEV is non-atomic. The most important case is arbitrage between a centralized exchange and an on-chain pool. When the price of ether moves on Binance, an on-chain pool quoting the old price is now mispriced. A trader can buy from the pool at the stale price and sell on the centralized exchange at the new one, but the two legs live in different systems. The on-chain leg settles in a block; the off-chain leg settles on the exchange's internal books. Nothing ties them together, so the trader bears real risk: the off-chain price may move again before the on-chain trade lands, or the on-chain trade may fail. Capturing non-atomic MEV requires capital on both venues, fast access to the centralized exchange's data, and the risk management of a professional trading firm. The distinction shapes who wins. Atomic MEV favours good engineers. Non-atomic MEV favours well-capitalized trading firms with low-latency links to off-chain markets. As decentralized finance matured, and as the easier atomic opportunities were competed down to thin margins, the share of MEV that was non-atomic grew, and with it the advantage of firms that already dominated trading in conventional crypto markets. Measurement suffers too. Because only one leg of a non-atomic trade is visible on-chain, analysts must infer the other, and published totals that count only atomic MEV understate the real figure by a wide margin. The picture assembled so far is static: a mempool full of visible intentions, a block producer free to order them, and a set of contracts whose outcomes depend on order. What turned this into a phenomenon worth naming was competition. As DeFi grew through 2020, the sums at stake rose, and many independent actors began to fight over the same positions in the same blocks. The fighting took place in the open, through the only lever the protocol offered them, the fee. The results were dramatic enough to give the field its defining metaphor. Chapter 2: The Dark Forest In August 2020, Dan Robinson and Georgios Konstantopoulos of the investment firm Paradigm published a short essay that gave MEV its most durable image. Robinson had been approached by someone who had made a mistake: they had sent Uniswap liquidity tokens worth roughly $12,000 directly to a Uniswap pair contract, where they were stuck. On inspection, Robinson realized that the tokens were not stuck at all. Anyone could call the contract's burn function and walk away with them. He also realized that if he simply submitted that call, the transaction would sit in the public mempool where automated bots could see it, copy it, and submit their own version with a higher fee. So he and Konstantopoulos designed a disguised rescue, split across two contracts so that the profitable call would only be revealed at the last moment, in the same block as its setup. The setup and the rescue landed in different blocks. Within seconds of the profitable call becoming visible, a bot executed it first and took the funds. The essay that described the failure was titled "Ethereum is a Dark Forest." The phrase came from Liu Cixin's science fiction novel The Dark Forest, in which the universe is full of silent civilizations that destroy any other civilization that reveals its location. The authors wrote that "if the chain itself is a battleground, the mempool is something worse: a dark forest." Any transaction that exposes a profit, they argued, will be found and taken by predators the sender cannot see. Searchers and what they do The predators in the essay were what the field now calls searchers: automated systems that scan blockchain state and pending transactions for profit opportunities and submit transactions to capture them. The word is neutral by design. A searcher may be running entirely beneficial arbitrage, or it may be sandwiching retail traders. Most large searchers do several things at once. At a high level, every searcher does the same work. It maintains an up-to-date picture of the relevant on-chain state, such as the reserves of every important liquidity pool, the collateral ratios of open loans, and the prices reported by oracles. It watches for events that change that state: new blocks, pending transactions in the mempool, price movements on centralized exchanges. For each event, it simulates what would happen if certain transactions executed in a certain order, looking for sequences that leave it with more assets than it started with. When it finds one, it constructs a transaction and bids for the ordering position that makes the sequence work. Two features of Ethereum make this unusually powerful. The first is atomicity. A searcher can bundle many operations into a single transaction that either succeeds completely or reverts completely. If a three-hop arbitrage across several pools would leave the searcher worse off by the time it executes, the transaction can simply check its own profit at the end and revert. The searcher loses the fee but not the principal. The second is composability. Contracts call other contracts freely, so a single transaction can borrow millions in a flash loan, a loan that must be repaid within the same transaction, use the funds across several protocols, and repay before the transaction ends. Capital is no longer a binding constraint on how large an opportunity a searcher can exploit. Skill and speed are. The essay's most unsettling character was the generalized front-runner. Most searchers look for specific patterns: an arbitrage between named pools, a liquidation on a named protocol. A generalized front-runner looks for anything. It takes each pending transaction, simulates it, and checks whether the simulated execution makes money for the sender. If so, it simulates the same transaction with its own address substituted for the sender's. If that also makes money, it submits the copy with a higher fee. It does not need to understand what the transaction does. It only needs to know that it is profitable and copyable. That was the bot that took Robinson's rescue. The forest's inhabitants were not only predators. Developers and traders adapted, and some of the adaptations turned the bots' own logic against them. Contract authors learned to restrict profitable functions so that only a specified address could call them, which made copies worthless. Others began deliberately baiting bots: publishing tokens or contracts that behaved normally for ordinary users but were rigged so that a bot attempting to sandwich or copy them would lose money, an approach that exploited the fact that a bot acting on simulation can be fooled by code that behaves differently in the conditions of the real attack. These counter-measures made the forest more dangerous for careless bots, but they did not make it safer for ordinary users, who had neither the knowledge to set such traps nor any reason to. A month later, the security researcher samczsun published a sequel titled "Escaping the Dark Forest." In September 2020 he had found a vulnerability in a contract belonging to Lien Finance that would let anyone drain the roughly 25,000 ether it held, then worth about $9.6 million. The obvious rescue, calling the vulnerable function to move the funds to safety, would expose the exploit in the mempool and invite a generalized front-runner to execute it first, keeping the money. The rescue team's solution was to avoid the public mempool entirely. They worked with SparkPool, then one of the largest Ethereum mining pools, which built a private submission path that let them send their transactions directly to the pool for inclusion without broadcasting them. The transactions became visible only when they appeared in a mined block, and the funds were recovered. The two episodes, taken together, describe the problem and foreshadow its resolution. The problem was that the public mempool punished anyone whose transaction revealed value. The resolution, crude in this first instance, was a private channel to the block producer. Within months, the same idea would be generalized and turned into infrastructure. Priority gas auctions Before that infrastructure existed, searchers competed through the only mechanism Ethereum offered: the transaction fee. If two searchers spotted the same arbitrage, both would submit transactions, and the one with the higher gas price would be included first and capture the profit. The other would either fail outright or execute after the opportunity had closed. Daian and colleagues documented this in "Flash Boys 2.0," first released in April 2019 and later published at the IEEE Symposium on Security and Privacy. They called the phenomenon priority gas auctions. Because pending transactions are public, each bot could see its rivals' bids. Ethereum clients allowed a sender to replace a pending transaction with a new version at a higher gas price. So bots engaged in rapid, iterative bidding wars, each raising its gas price slightly over the other's, in a sequence of rounds measured in milliseconds, until the block was produced. The paper's authors observed these auctions on-chain, measured bots' reaction times, and showed that the bidding had the structure of a classic all-pay or first-price auction played in real time. The economics were straightforward. If an opportunity was worth a certain amount, competing bots would bid away most of it in fees, with the winner keeping a sliver. The miner, who collected the fees, captured most of the value. In that sense the market was efficient: the value of ordering flowed to the party that controlled ordering. But the process by which it got there was profoundly wasteful. The first cost was congestion. Priority gas auctions put many transactions into the mempool for every opportunity, and many of those transactions landed on-chain as failures. A losing bot whose transaction was included after the winner's would find the opportunity gone and revert, but it still consumed block space and paid fees. On busy days, a large share of the gas spent in some blocks went to reverted MEV transactions. Every unit of block space occupied by a failed bot transaction was a unit unavailable to ordinary users, who faced higher fees as a result. The second cost was fee inflation for everyone. Gas auctions drove up the gas price needed for prompt inclusion. Users who had no interest in MEV but needed their transactions included had to outbid the bots. The fee market, intended to allocate scarce block space to those who valued it most, was distorted by participants bidding for position rather than for space. The third cost was instability of the consensus layer. In proof-of-work, a miner who found a block containing a large MEV opportunity had an incentive to protect it, and other miners had an incentive to try to orphan it by producing a competing block that captured the opportunity themselves. The Flash Boys 2.0 authors warned that if MEV grew large relative to block rewards, these incentives could destabilize the chain. They described this as a systemic risk, not merely a market inefficiency. The fourth cost was more subtle: the bidding favoured those with the lowest latency and the best network positioning. A bot that could see a rival's bid sooner and respond faster won more often. This pushed searchers toward expensive infrastructure, such as many well-connected nodes around the world, privileged peering arrangements and custom client modifications. A market that was in principle open to anyone was in practice open to those who could afford to compete on speed. Priority gas auctions rewarded more than speed. Because a searcher's profit was what remained after fees, every unit of gas a bot's transaction consumed was a unit of profit lost, or a unit that could have been bid away to win. Searchers therefore optimized their contracts with a care rarely seen in ordinary software. They wrote critical code in low-level assembly rather than in Solidity, Ethereum's main programming language. They chose contract addresses with many leading zero bytes, because zero bytes were cheaper to include in transaction data. They structured calls to avoid touching storage locations that cost more to access for the first time in a transaction. The most striking optimization was the gas token. Ethereum at the time refunded part of the gas cost when a transaction freed up storage, as an incentive to clean up unused state. Developers realized that one could fill storage when gas was cheap and then free it inside an expensive transaction later, collecting the refund. Tokens such as GST2, developed by researchers who included some of the "Flash Boys 2.0" authors, and later 1inch's CHI, packaged this trick into a tradeable form. Bots bought or minted gas tokens during quiet periods and burned them during bidding wars, effectively storing cheap gas for use when it was expensive. The practice bloated Ethereum's state with junk data written purely to be deleted later. It was ended by EIP-3529, included in the London upgrade of August 2021, which sharply reduced the refunds that made gas tokens work. The arms race also extended into the network itself. A transaction's chance of winning depended on how quickly it reached the miners, so searchers ran nodes in many locations, connected directly to the nodes of major mining pools where they could, and modified their software to forward transactions faster than the standard client would. Some measured how quickly their rivals' bids propagated and tuned their responses accordingly. None of this was visible to an ordinary user, and none of it contributed to the network's function. It was the kind of private investment that Budish and his co-authors, writing about stock markets, had called socially wasteful: spending that changed who won without making the market better. Black Thursday The most vivid demonstration of how MEV and congestion could interact came on 12 March 2020, a day that crypto markets remember as Black Thursday. As the global financial panic over the coronavirus pandemic spread, the price of ether fell by roughly half in a little over a day. Every lending protocol saw collateral values collapse, and a wave of positions became eligible for liquidation. MakerDAO, the protocol behind the DAI stablecoin, liquidated undercollateralized positions through auctions in which outside participants, called keepers, bid DAI for the seized ether. The design assumed that competitive keepers would bid close to market value. On Black Thursday, the Ethereum network was so congested, and gas prices spiked so sharply, that many keepers' bids failed to land or were not submitted at all. Some keepers had been configured with gas settings that were too low for the conditions. In that environment, a small number of participants submitted bids of zero DAI and, with no competing bids landing, won. Reports at the time and subsequent analyses put the value of ether acquired this way at around $8.3 million. Maker was left with a shortfall in its system that it had to cover by minting and auctioning governance tokens. Black Thursday was not a sandwich attack or a front-run. It was a failure of a liquidation market under the very congestion that MEV competition helped create. It showed that ordering and inclusion are not just about who profits at the margin. When the mechanism for getting transactions included breaks down under stress, protocols that depend on timely inclusion can fail in ways their designers did not foresee. Later analyses suggested that some participants may have actively worsened congestion to keep competitors out, though the evidence on that point is contested. Either way, the lesson for protocol designers was clear: a mechanism that works only when inclusion is cheap and reliable is not robust. The shape of the problem by 2020 By the end of 2020, the situation looked roughly like this. Decentralized finance had grown from a curiosity into a market holding billions of dollars. MEV was large, contested and visible. Searchers were competing in open bidding wars that clogged the network, raised fees for everyone and enriched miners. Generalized front-runners made it dangerous to broadcast any transaction whose value could be copied. Some mining pools had begun to offer informal private channels, such as SparkPool's, that let selected users bypass the public mempool, but these were ad hoc, opaque and available only to those with the right relationships. There were three obvious ways forward, and each had a constituency. One was to do nothing and accept the dark forest as the price of an open system. A second was to fragment the market into private deals between large searchers and large miners, which was already beginning to happen. That path promised efficiency for insiders and opacity for everyone else, and it pointed toward a world where only the best-connected actors could compete. A third was to build a common, transparent, private channel: a place where any searcher could bid for ordering without exposing its strategy to the public mempool and without bidding through gas price, and where any miner could accept those bids. The third path is the one Flashbots took. Its founders were in several cases the same researchers who had documented the problem. Their argument was pragmatic and, at the time, controversial. MEV could not be wished away, they said. The choice was not between MEV and no MEV, but between MEV extracted chaotically in public, MEV extracted privately by a closed club, and MEV extracted through an open, auditable market that minimized its side effects. They chose the last, knowing that it would make MEV extraction easier, more efficient and more profitable. Whether that was the right call is still debated, and the debate is worth taking seriously. Before turning to how Flashbots worked and what it changed, though, it is necessary to look squarely at who was paying for all of this. The bots' profits, the miners' fees and the failed transactions all came from somewhere. A large share came from traders who never knew they were participants in an auction. Chapter 3: The Hidden Tax MEV is often discussed from the perspective of those who extract it: the searchers, their strategies, their profits. This chapter looks from the other side. Every unit of value a searcher captures comes from someone. In some cases the payer is a protocol that has chosen to pay for a service, as when a lending market pays liquidators a bounty. In others, the payer is a person who never agreed to the transaction and usually never notices it. Two groups bear most of the cost: traders who are sandwiched, and liquidity providers whose prices are picked off by arbitrageurs. Their losses take different forms, but they share a feature that makes MEV politically and ethically difficult. The cost is invisible to the person who pays it. Anatomy of a sandwich A sandwich attack exploits a single parameter that every decentralized exchange trade carries: the minimum amount of output the trader will accept. Because prices on an automated market maker move with every trade, and because a trade may wait in the mempool while other trades execute ahead of it, a trader cannot know the exact price they will receive. So they set a tolerance, called slippage tolerance, expressed as a percentage. A trade with a 1 percent tolerance will execute so long as the output is no more than 1 percent below the quoted amount, and will revert otherwise. The tolerance is a reasonable protection against honest price movement. It is also a precise statement of how much value an attacker can take. A worked example in round numbers shows the mechanics, ignoring trading fees and transaction costs to keep the arithmetic clear. Suppose a constant-product pool holds 1,000 ETH and 2,000,000 USDC, so the quoted price is 2,000 USDC per ETH. A trader submits an order to spend 100,000 USDC on ETH. Because the trade itself moves the price, the trader would receive about 47.62 ETH if nothing else happened, an average price of 2,100 USDC. The trader sets a 1 percent slippage tolerance, so the transaction will accept any output of at least about 47.14 ETH. An attacker who sees this order can compute exactly how far it can push the price before the trader's limit binds. In this example, buying about 5.13 ETH for roughly 10,320 USDC immediately before the trader's order raises the pool's price just enough that the trader's 100,000 USDC now buys only 47.14 ETH, exactly the minimum. The trader's order executes, pushing the price higher still. The attacker then sells its 5.13 ETH back into the pool at the elevated price, receiving about 11,370 USDC. Before fees and costs, the attacker gains about 1,050 USDC. The trader has received about 0.48 ETH less than they would have otherwise, worth roughly the same amount. Three features of the example generalize. First, the attacker's profit is bounded by the victim's tolerance. A trader who sets a 0.1 percent tolerance offers much less room than one who sets 5 percent. Second, the attack requires the attacker's buy to land immediately before the victim and its sell immediately after. It depends entirely on control over ordering, which is why sandwiching became a commodity once ordering could be bought reliably. Third, the trader receives an output within the range they agreed to. From their perspective, nothing went wrong. They asked for at least 47.14 ETH and got it. The loss is the difference between what they received and what they would have received in an honest market, and that counterfactual is not shown anywhere in their wallet. Real sandwiches are messier. Trading fees, typically 0.05 to 1 percent per trade depending on the pool, reduce the attacker's margin, since it pays fees on both legs. The attacker must pay for gas and, increasingly, must pay most of its profit to the block builder for the privilege of the ordering. Competition among sandwich bots for the same victim drives these payments up. Often the attacker's net profit is a small fraction of the victim's loss, with the remainder flowing to builders and validators. But the victim's loss does not depend on how the spoils are divided. It is set by their tolerance and the size of their trade relative to the pool. Sandwiching is most profitable against large trades in thin pools with generous tolerances. In practice that means it falls disproportionately on retail traders using default wallet settings, on traders of newer and less liquid tokens, and on anyone who raises their tolerance after a trade fails. That last group deserves attention. A trader whose order reverts because the price moved is often advised, by interfaces and by forum lore, to raise their slippage tolerance and try again. Raising it makes the trade more likely to execute, and also makes it a more attractive sandwich target. During 2023, a single sandwich bot known by its on-chain name, jaredfromsubway.eth, became famous for the sheer volume of its activity, at times accounting for a large share of Ethereum gas consumption during periods of speculative trading in memecoins. Its prominence made the abstraction concrete. Here was a specific address, visible to all, systematically taking value from thousands of traders in each of whom the loss was small enough to ignore. Nobody had to be deceived in a conventional sense. The bot simply read the public mempool and acted on what it saw. Measuring the aggregate is harder than describing a single case. Dashboards maintained by researchers and analytics firms, such as Flashbots' own MEV-Explore in its early years and later EigenPhi and others, classified on-chain transactions into arbitrage, liquidation and sandwich categories and reported cumulative totals. Those totals reached into the hundreds of millions of dollars for Ethereum alone within a few years of DeFi's growth, but the methods differ, the classification is imperfect, and the figures exclude sandwiches executed through private channels that are harder to detect. Any precise figure for sandwich losses should be treated as an estimate of order of magnitude. What is not in doubt is the direction: the cost was concentrated on the least sophisticated participants, and it was large enough to fund a professional industry. What a single trader can do Before turning to systemic remedies, it is worth asking what an individual trader can do on their own, because the answer reveals the trade-offs that systemic remedies must also confront. The most direct defence is a tighter slippage tolerance. In the worked example, a tolerance of 0.1 percent instead of 1 percent would have cut the room available to an attacker by roughly a factor of ten, and below a certain level the attack stops being worth its fees. The cost is failure. A trade with a tight tolerance will revert whenever honest price movement exceeds it, and in volatile markets that happens often. The trader pays the fee for the failed attempt, then must try again, perhaps at a worse price. Tight tolerances protect against predators at the expense of reliability. A second defence is to split a large trade into smaller ones. Price impact in a constant-product pool grows faster than the trade size, so several small trades, spread across blocks, move the price less in total and offer less room for a sandwich. The cost is time, more fees, and exposure to market movement between the pieces. A third defence is to use a limit order, which specifies an exact price rather than a tolerance. Some decentralized exchanges and aggregators offer limit orders, executed by third parties when the market reaches the specified price. A limit order cannot be sandwiched in the usual way, because it has no tolerance to exploit, but it may never execute. The fourth defence, and the most effective, is not to expose the trade to the public mempool at all, by sending it through a private channel or using an exchange design that does not broadcast it. That option depended on infrastructure that did not exist in 2020 and that later chapters describe. The striking fact about the first three defences is that each one requires the trader to understand the threat, to know the right settings, and to accept a cost in reliability, time or certainty. They are available to sophisticated traders, who are least likely to be sandwiched in the first place. For everyone else, protection had to come from defaults set by others. Front-running and the question of wrongdoing In traditional securities markets, front-running has a specific legal meaning: a broker who learns of a client's order and trades ahead of it for its own account violates its duty to that client. The prohibition rests on the broker's special relationship with the client and on the confidentiality of the information. A broker who trades on a client's order is misusing information entrusted to it. On a public blockchain, the analogy breaks down in revealing ways. The trader has broadcast their order to the world. The searcher has no relationship with the trader and owes them no duty. The information is not confidential; it was published by the trader's own wallet. The block producer, who does have something like a gatekeeper role, has made no promise about order. Much of the crypto community's initial reaction to sandwiching reflected this: it is not front-running in the legal sense, the argument ran, but simply a market participant responding to public information within the rules the protocol allows. If the rules permit it, it is fair game. That argument has never been fully persuasive, and it has weakened as the market has matured. A sandwich is not a response to information about the asset's value. It is a strategy that profits only by worsening the execution of a specific identified order. Its economic substance is close to what securities law treats as manipulation, even if the legal relationships do not fit. And the infrastructure of sandwiching, once it ran through block builders and relays that took payment for ordering, began to look less like an anonymous market and more like a set of intermediaries who knowingly sold the ability to harm users. Courts and regulators have not settled how existing law applies. The criminal case discussed in Chapter 5, in which prosecutors charged two brothers with defrauding sandwich bots themselves, shows how unsettled the questions are. But the ethical question is clearer than the legal one. A system in which ordinary users predictably lose value to an extraction industry, through a mechanism they cannot see, has a design problem regardless of whether anyone is breaking a law. The larger, quieter loss Sandwiches are dramatic, but they are not the largest cost MEV imposes. That distinction belongs to a loss that falls on liquidity providers, the people and institutions who deposit assets into automated market maker pools so that others can trade against them. Liquidity providers earn trading fees. In return, they take on a risk long described as impermanent loss: if the relative price of the pool's two assets changes, the pool's mechanical rebalancing leaves the provider holding more of the asset that fell and less of the asset that rose, compared with simply holding the original assets. The term "impermanent" refers to the fact that the loss disappears if prices return to where they started. For years, this was the standard way to think about the cost of providing liquidity. In 2022, Jason Milionis, Ciamac Moallemi, Tim Roughgarden and Anthony Lee Zhang offered a sharper framing in a paper titled "Automated Market Making and Loss-Versus-Rebalancing." They asked a different question: how much worse does a liquidity provider do than a hypothetical trader who holds the same portfolio as the pool but rebalances it by trading at fair market prices? The answer, which they called loss-versus-rebalancing or LVR, isolates the cost of trading at stale prices. Every time the market price of an asset moves on a centralized exchange, the pool's price is wrong until an arbitrageur corrects it. The arbitrageur buys from or sells to the pool at the stale price, and the difference between that price and the fair price is a loss to the liquidity providers. Unlike impermanent loss, it does not reverse when prices come back. It is a running cost, paid on every price movement. The authors showed that for the standard constant-product design, this cost grows with the square of volatility: a pool in a market twice as volatile loses roughly four times as much to arbitrageurs per unit of time. For liquid, volatile pairs such as ether against a dollar stablecoin, the implied cost can be substantial relative to the fees the pool earns. Subsequent empirical work has found that for many pools and periods, liquidity providers' fee income did not cover their loss to arbitrage, meaning they were paying, on net, for the privilege of providing a market. This loss is MEV in the strict sense. The arbitrageur captures it by being the first to trade against the stale price after each market move, and that position is obtained through the same ordering market that serves sandwich attackers. The largest flow of this kind is arbitrage between centralized exchanges and on-chain pools, which researchers began to recognize in 2023 and 2024 as a very large share of Ethereum MEV, by several estimates the largest single category, even though it is harder to see on-chain because only one leg of each trade happens there. Sophisticated trading firms with low-latency access to centralized exchange prices, and with the capital to hold inventory on both sides, are best placed to capture it. That advantage, as Chapter 5 shows, translated directly into advantage in block building. Both kinds of loss share a property that explains why MEV persisted so long without a consumer backlash. The sandwiched trader sees an execution within their tolerance. The liquidity provider sees a position that rises and falls with the market and earns fees. Neither sees a line item labelled "MEV." To detect the loss, one has to construct a counterfactual: what price would the trader have received without the sandwich, and how would the provider have fared if trades had happened at fair prices? Those counterfactuals are computable, and researchers compute them, but they are not presented to users at the moment of decision. The invisibility shapes the politics. Costs that are diffuse and hidden generate less pressure than costs that are concentrated and obvious. Searchers, builders and validators who receive MEV have a strong, organized interest in the system that delivers it. Users who pay have, individually, a weak and uninformed interest. That asymmetry is familiar from many settings in economics and regulation. It explains why the most effective early responses to sandwiching came not from user demand but from wallets and applications that decided to protect their users, and from researchers who made the losses visible. It also points toward the mitigations. If the problem is that the user's intention is visible before execution, one family of solutions hides it, by sending trades through private channels or encrypting them. If the problem is that the user's tolerance is exploitable, another family reduces the tolerance, by giving better quotes, using tighter defaults, or replacing the trade with a mechanism that guarantees a fair price. If the problem is that liquidity providers sell at stale prices, a third family changes the pool itself, so that it learns the market price or charges arbitrageurs for the privilege of correcting it. Later chapters examine each. The first practical step, however, came from an organization that set out not to stop MEV extraction but to tame its side effects, and that, in doing so, made extraction more efficient than ever. Chapter 4: Bribes and Bundles Flashbots began in 2020 as a research and development organization founded by people who had studied MEV closely, including Philip Daian, the lead author of "Flash Boys 2.0." Its founding argument was blunt. MEV existed, it was large, and it was being extracted in ways that harmed the network: congestion from bidding wars, failed transactions, rising fees, and a drift toward private deals between the biggest searchers and the biggest miners. Pretending it could be eliminated was naive. The responsible course was to build an open, transparent marketplace for ordering that would channel MEV extraction away from its most damaging forms, publish data about it, and in time redistribute its benefits more fairly. The organization summarized its aims under three headings: to illuminate MEV by measuring it, to democratize access to it so that no closed club captured it, and to distribute its proceeds more widely. The bundle The central technical idea was the bundle. A bundle is an ordered list of signed transactions that a searcher wants executed together, in sequence, at a particular point in a block. It might contain a single arbitrage transaction, or a searcher's transaction placed directly after a user's pending trade, or several transactions that together accomplish something none could alone. Crucially, the bundle carries its own payment. Instead of signalling willingness to pay through a high gas price in the public mempool, the searcher includes a direct transfer of ether to the block producer's address, known on Ethereum as the coinbase address, as part of the bundle's execution. If the bundle's transactions execute as intended, the payment executes too. If they do not, the bundle is simply not included. Searchers did not broadcast bundles to the public network. They sent them privately to a Flashbots relay, a server that collected bundles and forwarded them to miners. Miners ran a modified version of the most popular Ethereum client, called MEV-Geth, which accepted bundles from the relay, simulated them, and selected the combination that paid the most, placing them at the top of the block before filling the remainder with ordinary transactions from the public mempool. Flashbots released MEV-Geth as open-source software and opened its auction to miners and searchers in early 2021. The mechanism changed the auction's form in three ways at once. First, bids became sealed. A searcher's bundle was visible only to the relay and the miner, not to rival searchers, so the iterative, real-time bidding wars of the priority gas auction era disappeared. Each searcher submitted its best bid once. Second, failed bids became free. Under the old system, a losing searcher whose transaction landed after the winner's paid fees for a reverted transaction. Under the bundle system, a bundle that would revert, or that lost to a higher-paying bundle competing for the same opportunity, was not included at all. The searcher paid nothing. Third, the payment became conditional on success. Because the payment to the miner was part of the bundle's own execution, the searcher paid only when it captured the opportunity. The effects were fast and measurable. Flashbots published monthly transparency reports in 2021, and by its report covering April of that year, it stated that eight of the top ten mining pool addresses were running MEV-Geth and that pools accounting for more than 80 percent of Ethereum's hashrate were receiving bundles. The number of unique searchers submitting bundles grew rapidly over the same months. Within a short time, the Flashbots auction had become the dominant channel through which MEV was sold. The public bidding wars subsided. Searchers no longer needed to compete by raising gas prices in the mempool, so the bursts of replacement transactions and the floods of reverted bot transactions declined. Block space that had been consumed by failed MEV attempts was freed for ordinary users. In this narrow but important sense, Flashbots did what it set out to do: it reduced the externalities that MEV competition imposed on everyone else. It also changed who captured MEV. In a sealed-bid auction with many competing searchers, each searcher has an incentive to bid close to the full value of the opportunity, because any searcher who shades its bid risks losing to one who does not. As competition intensified, the share of MEV paid to miners rose. Flashbots' reports tracked miners' profits from bundles growing month by month during 2021. Searchers kept a margin that reflected their edge, whether in speed, in strategy or in exclusive information, but the bulk of competitive MEV flowed to the block producer. That outcome was, in economic terms, efficient. The party who controls the scarce resource, ordering, collects most of its value. The case against Flashbots' approach drew sharp criticism from the start, and the criticisms have aged in interesting ways. The first was that Flashbots legitimized MEV extraction, including its harmful forms. By building clean infrastructure for bundles, the organization made sandwiching as easy as arbitrage. A sandwich is just a bundle with three transactions: the searcher's buy, the victim's pending trade, and the searcher's sell. The relay did not distinguish between them. Critics argued that an organization committed to mitigating MEV's harms should not have built a pipeline that delivered sandwich attacks to miners more reliably than before. Flashbots' answer was that sandwiching existed already and would exist without them, and that their infrastructure at least made it measurable. That answer is not wrong, but it does not dispose of the objection. Lowering the cost of an activity tends to increase it. The second criticism was about trust. A searcher sending a bundle to a relay and a miner reveals its strategy to both. A dishonest miner could take a profitable bundle, replace the searcher's transactions with its own, and keep the profit. The system relied on reputation: Flashbots could cut off a miner caught stealing bundles, and losing access to the bundle flow would cost the miner more than any single theft would gain. This was a workable arrangement among a small number of identifiable mining pools. It was not the kind of trustless guarantee that the technology is supposed to provide. It foreshadowed the role of trusted relays in the system that followed. The third criticism was structural. The Flashbots auction created a privileged path to block inclusion. A transaction submitted through it was placed at the top of a block, invisible to the public until included, and protected from failure costs. Transactions submitted through the public mempool were exposed to all the dangers described in the previous chapters. The system therefore created two classes of transaction: those with access to private channels and those without. Flashbots tried to address this by opening its channel to ordinary users as well. In 2021, Flashbots launched Flashbots Protect, a service that let ordinary users send their transactions directly to the Flashbots relay instead of to the public mempool. A user could configure their wallet to use Flashbots Protect as its network endpoint. Transactions sent this way were not visible to sandwich bots scanning the public mempool, so they could not be sandwiched by bots relying on that visibility. If a transaction would revert, it was not included and the user paid no fee for the failure. The service illustrated both the promise and the limits of private channels. For users who knew to use it, it worked: it removed the main exposure that made sandwiching possible. But most users did not know it existed, and defaults matter more than options. A protection that requires users to change their wallet's network settings will be adopted by the sophisticated few. Over time, several wallets and applications began routing their users' transactions through private channels by default. That shift, discussed in Chapter 6, created its own problems. Once private order flow had value, the parties who controlled it could sell it. There was a deeper limit as well. Sending a transaction privately to a relay does not make it invisible; it makes it visible to fewer parties. The relay and the block producer still see it. If either of them is willing to exploit it, the protection fails. Flashbots Protect worked because Flashbots and the miners it worked with committed not to sandwich protected transactions. That was a promise backed by reputation and by the structure of the market, not a cryptographic guarantee. The more the system depended on such promises, the more important it became to ask who was making them and what would enforce them. Roads not taken Flashbots' design was not the only proposal on the table, and the alternatives clarify what it chose. In late 2019, Karl Floersch, then working on the Optimism rollup project, proposed what he called an MEV auction. The idea was to separate the right to order transactions from the right to include them, and to sell the ordering right in an open auction run by the protocol, with the proceeds directed to a public purpose such as funding the development of the system. Under this design, MEV would not be a side income for miners or a profit for searchers but a revenue stream for the community. The concept anticipated proposer-builder separation and, in a different form, the auction-based approaches that several rollups adopted years later. It was not something Ethereum's main chain could adopt quickly, because it required changes to the protocol's rules, and those take years to agree and implement. A second alternative was to change the rules of ordering itself, requiring transactions to be ordered by the time they were first seen, a principle known as fair ordering. Researchers proposed several protocols in which a committee of nodes would agree on the order in which they received transactions and the block producer would be bound to follow it. Fair ordering is attractive in principle because it removes the producer's discretion. In practice, "the order in which transactions were received" is not well defined in a global network, where different nodes see transactions in different orders depending on their location. Every fair-ordering scheme has to resolve those disagreements somehow, and the resolution rules can themselves be gamed by participants who position themselves close to many nodes. Fair ordering tends to convert a bidding war into a latency race, which is not obviously an improvement. A third alternative, discussed seriously at the time, was to accept MEV-Geth as a temporary measure and simultaneously push for a protocol-level redesign. That is roughly what happened, though the timeline proved long. Flashbots itself argued that its infrastructure should be understood as transitional, a way to contain the damage while better designs were built. The question that hangs over its legacy is whether transitional infrastructure, once it becomes profitable and indispensable, ever really becomes temporary. What Flashbots offered that the alternatives did not was speed of deployment. MEV-Geth required no change to Ethereum's consensus rules. It required only that miners run modified software, which they had every financial reason to do, and that searchers use a new submission channel, which they had every reason to adopt since it removed the cost of failed bids. A design that aligned with every major participant's immediate incentive could be deployed in months. Designs that asked participants to give up revenue, or that required the whole community to agree on new protocol rules, could not. That asymmetry, between solutions that ride incentives and solutions that redirect them, runs through the whole history of MEV. Efficiency is not neutrality The most important lesson of the Flashbots period is one that runs through the rest of this book. Making a market more efficient changes who wins it, and efficiency rewards the most capable participants. Before Flashbots, MEV was captured by whoever won messy, public, latency-driven bidding wars, and much of its value was destroyed in failed transactions and congestion. After Flashbots, MEV was captured mostly by block producers through a clean sealed-bid auction, with searchers keeping margins that reflected their edge. Less value was destroyed. More value was transferred. From the perspective of the network as a whole, this was a clear improvement. Fewer failed transactions meant lower fees and less wasted block space. From the perspective of a user being sandwiched, it was a mixed result: the sandwich was now delivered more reliably, but the user could opt out by using a private channel. From the perspective of decentralization, it introduced a new dependency. Miners who did not receive Flashbots bundles earned less than those who did, so nearly all of them adopted MEV-Geth. The Flashbots relay became, in effect, critical infrastructure for the most valuable part of block production on Ethereum. Flashbots was a research organization with good intentions and a public commitment to transparency. But a system that relies on any single organization's good intentions is fragile, and its designers knew it. They also knew that Ethereum was about to change fundamentally. The transition from proof-of-work to proof-of-stake, planned for 2022, would replace a few dozen mining pools with hundreds of thousands of validators, many of them individuals running a single machine at home. Those validators could not be expected to run sophisticated searching or block-building operations. Under proof-of-stake, an ordinary validator that tried to build its own blocks would earn far less than one that outsourced building to a specialist, and the pressure would push validators toward large staking pools that could build blocks well. MEV, in other words, threatened to centralize the validator set itself. The response to that threat was proposer-builder separation, and the infrastructure that implemented it was another Flashbots product: MEV-Boost. Before turning to that transition, it is worth noting what the Flashbots period revealed about Ethereum's economy. A blockchain's security budget, the total reward paid to those who produce blocks, is drawn from two sources: rewards issued by the protocol and fees paid by users. MEV adds a third, a payment for ordering that is not set by the protocol at all but by a market built on top of it. During 2021, MEV payments became a meaningful share of miners' total revenue, and that share fluctuated with market activity, rising sharply during periods of volatile trading. The London upgrade of August 2021 sharpened this point. EIP-1559 began burning the base fee that users paid, so that the bulk of ordinary transaction fees no longer went to miners at all. Only the priority tip remained as fee income. Because MEV payments arrived as tips or as direct transfers within bundles, they were untouched by the burn. The effect was to make MEV a larger share of what block producers actually earned from transactions, and therefore a larger factor in the economics of producing blocks. A design change aimed at making fees more predictable for users had, as a side effect, made the ordering market more important to the people who controlled ordering. This has a double edge. On one hand, MEV supplements the security budget, paying block producers more than protocol rewards and fees alone would. On the other, it introduces a revenue stream that rewards sophistication and scale, and that varies with market conditions in ways the protocol does not control. A validator that captures more MEV earns higher returns, which attracts more stake, which in turn lets the validator propose more blocks and capture more MEV. Any reward that scales better than linearly with the size of the operation is a centralizing force. The challenge for Ethereum's designers, after the Merge, was to find a way to let every validator earn roughly the same MEV regardless of size or sophistication. Their solution worked, in that narrow sense. It did so by moving the concentration to a different part of the system. Chapter 5: After the Merge On 15 September 2022, Ethereum completed the transition known as the Merge, replacing proof-of-work mining with proof-of-stake. Under the new system, anyone who deposits 32 ether can run a validator. Time is divided into twelve-second slots, and in each slot the protocol pseudo-randomly selects one validator to propose a block. Other validators attest to it. The validator set grew to hundreds of thousands of active validators, run by a mixture of individuals at home, professional staking companies, centralized exchanges and liquid staking protocols that pool deposits from many holders. This change transformed the MEV problem. Under proof-of-work, block production was dominated by a few dozen mining pools, each a professional operation that could negotiate with Flashbots and run modified software. Under proof-of-stake, any validator might be selected to propose the next block, including a hobbyist with a single machine. Such a validator has no ability to find MEV, and if it builds its own blocks from the public mempool, it will earn substantially less than a large operator that builds blocks with specialist help. If that gap persists, stakers will move their ether to the operators that earn the most, and the validator set will concentrate. MEV would then be centralizing not just block building but consensus itself. Separating proposers from builders The response was a design principle called proposer-builder separation, or PBS. The idea, developed by Vitalik Buterin and other Ethereum researchers in 2021, is to split block production into two roles. Builders are specialists who assemble full blocks, competing to include the most valuable set of transactions and bundles in the most valuable order. Proposers are the validators selected by the protocol to propose a block in a given slot. Rather than building blocks themselves, proposers simply choose the most valuable block offered to them by builders and sign it. Builders pay proposers for the privilege. If builders compete vigorously, nearly all the value of a block flows to the proposer, regardless of whether that proposer is a hobbyist or a large staking company. Every proposer earns roughly the same MEV per block, so MEV no longer gives larger operators an edge. The long-term plan was to build PBS into the Ethereum protocol itself. That would take years. In the meantime, Flashbots built an out-of-protocol version called MEV-Boost, released in time for the Merge. MEV-Boost is a piece of software that validators run alongside their normal client. When a validator is selected to propose, MEV-Boost asks a set of relays for their best offers. Each relay has been receiving blocks from builders, checking them, and ranking them by the payment they offer the proposer. The relay returns only the header of the best block and the bid, not the block's full contents. The proposer signs the header, committing to propose that block without seeing what is in it. Once the relay receives the signed header, it releases the full block to the network, and the proposer is paid. This design solves one problem and creates another. It solves the problem of builders' strategies leaking to proposers: since the proposer sees only a header, it cannot copy the builder's work and steal the MEV. It creates the problem of trust. Builders must trust that relays will not steal their blocks. Proposers must trust that relays will actually publish the full block after they sign, that the block is valid, and that the bid is honest. The relay is a trusted intermediary at the centre of the most valuable process on the chain. Table 2 summarizes the roles in this supply chain. Table 2. Roles in Ethereum's post-Merge block supply chain. Role What it does How it is paid Main trust it requires User / wallet Creates and signs transactions Pays fees; may receive rebates That its transaction is not exploited Searcher Finds MEV; sends bundles to builders Keeps margin on opportunities That builders do not steal bundles Builder Assembles full blocks, bids to proposer Keeps difference between block value and bid That relays do not steal blocks Relay Validates blocks, escrows bids, releases payload Usually unpaid or subsidized Trusted by both builders and proposers Proposer (validator) Signs the most valuable header Receives builder's bid That the relay publishes a valid block MEV-Boost was optional. Validators could keep building their own blocks. Almost none did. Within months of the Merge, the large majority of Ethereum blocks were being built by external builders through MEV-Boost, and the share has stayed at roughly nine in ten blocks in most periods since, according to public trackers of relay data. The reason is simple economics. A validator that outsources building earns more per block than one that builds locally, often substantially more during periods of active trading. For professional staking operations with fiduciary obligations to their customers, choosing the lower return was hard to justify. For individuals, the extra return was free money. In terms of its main purpose, PBS through MEV-Boost was a success. Small validators earned MEV comparable to large ones. The direct centralizing pressure on the validator set was largely neutralized. But the concentration did not disappear. It moved upstream. The builder oligopoly The builder market began with many participants. Flashbots ran a builder. Several independent teams and trading firms ran others. In principle, the market was open: anyone could build blocks and submit them to relays. In practice, the market concentrated quickly and steadily. The reason lies in what makes a builder win. A builder wins a slot by offering the proposer the highest bid, and it can offer the highest bid only if its block contains more value than its rivals' blocks. Value comes from two places: the transactions and bundles the builder receives, and the builder's own trading. A builder that receives exclusive order flow, meaning transactions or bundles that rivals do not see, can build more valuable blocks. A builder that is integrated with a sophisticated trading operation, especially one that captures arbitrage between centralized exchanges and on-chain pools, can add its own profitable trades to its blocks and bid more aggressively. Both advantages compound. A builder that wins more slots becomes more attractive to searchers and order flow providers who want their transactions included, so it receives more exclusive flow, so it wins more slots. By 2024, a small number of builders produced the large majority of MEV-Boost blocks. A study by Fei Wu, Thomas Thiery, Stefanos Leonardos and Carmine Ventre, titled "From Competition to Centralization: The Oligopoly in Ethereum Block Building Auctions," found that two builders, beaverbuild and Titan, together won over 90 percent of the auctions in the periods they examined, and that the top ten builders built about 97.5 percent of all MEV-Boost blocks. Their analysis attributed the concentration less to latency than to access to private order flow, and found that dominant builders had incentives to bid conservatively, keeping more of each block's value for themselves rather than passing it to proposers. This is the central irony of the post-Merge settlement. PBS was meant to keep MEV from centralizing validators, and it did. But it did so by creating a separate, highly concentrated layer of builders, a market where a handful of firms decide the contents and order of nearly every Ethereum block. Those firms are not protocol participants. They post no stake, face no slashing, and are not chosen by any process the protocol controls. They win because they have the most information and the most capital. Censorship and the sanctioned addresses The fragility of this arrangement became visible within weeks of the Merge. On 8 August 2022, the U.S. Treasury's Office of Foreign Assets Control added Tornado Cash, a smart-contract system that let users break the on-chain link between deposits and withdrawals, to its sanctions list, along with a set of associated Ethereum addresses. The listing raised an immediate question for anyone whose business touched Ethereum transactions: could they legally process transactions involving those addresses? Several relays, including the one operated by Flashbots, responded by filtering out blocks that contained transactions involving sanctioned addresses. Several builders did the same. Because MEV-Boost blocks made up most of the chain, and because the compliant relays and builders were among the largest, a large share of Ethereum blocks in late 2022 excluded sanctioned transactions. Trackers such as MEV Watch reported that, at its peak, a majority of blocks came through relays that filtered. Transactions involving those addresses could still be included, by validators building locally or using non-filtering relays, but they waited longer. This was not censorship by the protocol. Ethereum's rules did not change, and every transaction could still eventually be included. It was censorship by the infrastructure layer that sat between users and the protocol, driven by the legal exposure of a few identifiable companies in a single jurisdiction. It showed that a chain whose blocks are mostly built by a few firms inherits those firms' legal and political constraints. The response from the community was to build alternatives: new relays that did not filter, such as those run by the Ultra Sound and Agnostic teams, and pressure on validators to connect to them. Flashbots open-sourced its relay software so others could run it. The filtering share fell over the following year. The legal picture changed too: in November 2024, the U.S. Court of Appeals for the Fifth Circuit ruled in Van Loon v. Department of the Treasury that Tornado Cash's immutable smart contracts were not property that could be sanctioned, and in March 2025 the Treasury removed Tornado Cash from the sanctions list. But the structural lesson remained. When building concentrates, so does the ability to exclude. Weak points in the machinery A second episode exposed the trust in relays directly. In April 2023, an exploit drained roughly $25 million from a group of MEV bots in about twelve seconds. The bots in question ran sandwich strategies. According to U.S. prosecutors, two brothers, Anton and James Peraire-Bueno, set up validators and waited until one was selected to propose a block. They allegedly broadcast bait transactions designed to attract the bots' sandwich bundles. Then, taking advantage of a flaw in the relay's handling of signed headers, they obtained the contents of the block, including the bots' bundles, before it was final, and replaced the bait transactions with their own, so that the bots' trades executed against altered conditions and lost their funds. Flashbots patched the relay vulnerability shortly afterward. The brothers were indicted in May 2024 on charges including wire fraud and money laundering. The case went to trial in late 2025. The jury could not reach a verdict, and on 7 November 2025 the judge declared a mistrial. The defence had argued, in part, that the brothers had used the system according to its own rules and that the victims were themselves engaged in predatory trading. The prosecution had argued that the scheme relied on deception, a trade that "looked like one thing from the outside" but was secretly something else. Whatever the case's eventual resolution, it raised questions the field has not answered. If MEV extraction is legitimate when bots sandwich users, is it fraud when someone sandwiches the bots? If the rules of a system are its code, what counts as exploiting a flaw, and what counts as using a feature? The case also made the trust model concrete. The relay is supposed to protect builders from proposers and proposers from builders. A bug in the relay turned that protection inside out, and the losses fell on whoever had trusted it. The relay's position raises a question that the MEV-Boost design did not answer: how are relays paid? In the original arrangement, they were not. The relay does real work. It must receive blocks from many builders, simulate each one to check that it is valid and that its payment to the proposer is real, keep the highest bid ready for any proposer who asks, and serve the full block quickly once the header is signed. All of this must happen within the tight timing of a twelve-second slot, with reliability good enough that proposers do not miss blocks. Yet MEV-Boost provided no fee for any of it. The builder paid the proposer; the relay received nothing. Relays were therefore run for other reasons. Some were run by organizations, such as Flashbots, that regarded them as public goods or as part of a broader mission. Some were run by builders, who gained from operating the infrastructure their blocks passed through, an arrangement that raised obvious questions about neutrality. Some were run by community groups funded by grants. And at least one prominent operator, Blocknative, shut down its relay in 2023, a move reported at the time as a blow to the diversity of the relay market. The economic pressure on relays also changed how they worked. Simulating every block before forwarding its bid takes time, and in a market where bids rise through the slot, the time cost is a revenue cost for builders and proposers. Some relays introduced optimistic relaying, in which builders post collateral and the relay forwards their bids without first fully checking the block. If a block later turns out to be invalid, the relay uses the collateral to compensate the proposer for the lost slot. Optimistic relaying cut latency, which builders valued, but it favoured builders with enough capital to post collateral, which once again tilted the field toward the largest participants. It also illustrated the general direction of travel: each refinement of the post-Merge machinery traded a little more trust or capital for a little more speed. A subtler pressure emerged as the market matured. Because builders keep improving their blocks throughout a slot, as new transactions arrive and prices move, the best available bid rises over time. A proposer that waits a little longer before requesting a block header will, on average, receive a higher bid. Researchers documented this behaviour, which they called timing games: some proposers deliberately delay their request by hundreds of milliseconds to capture more MEV. The delay reduces the time available for other validators to receive and attest to the block, which increases the risk that the block is missed or orphaned. Each proposer gains a little by playing the game. The network as a whole bears a small increase in instability. Sophisticated operators can measure the trade-off precisely and push it further than hobbyists can, reintroducing a modest advantage of scale. The same pressure explains why latency matters so much to builders and relays. Being physically close to major validator operators and to centralized exchange data centres shortens the time between a price change and a bid update. Builders, relays and the largest validators have clustered in a small number of data-centre regions. None of this is visible in the protocol's rules. All of it shapes who wins. By 2024, the post-Merge settlement looked like this. Validators were broadly decentralized and earned similar MEV, which was PBS's goal. Building was concentrated in a handful of firms with the best order flow and trading operations. Relays were a small set of trusted intermediaries, some of them run by builders themselves, few of them paid for their service. Users who sent transactions through the public mempool remained exposed to sandwiching, and users who used private channels depended on the promises of whoever received their transactions. Censorship had been demonstrated and partly reversed. A serious exploit had shown how much depended on relay code. Ethereum's researchers regarded this as a temporary arrangement, and the protocol has been moving to absorb parts of it. Before turning to those changes, there is one more piece of the market to examine: the order flow itself, the stream of user transactions that builders compete for and that has become the most valuable input in the system. Chapter 6: Who Gets the Money If MEV is a rent created by control over ordering, its raw material is order flow: the stream of user transactions that create opportunities by moving prices, triggering liquidations and revealing intentions. A sandwich needs a victim's trade. An arbitrage needs the price movement that someone's trade created. A builder's block is valuable because of the transactions in it. Whoever controls the path a transaction takes from a user's wallet to a block therefore controls something worth paying for. Once the public mempool became dangerous, and once private channels became normal, that path itself became a market. This chapter follows the money down that path, and examines the most widely adopted answer to MEV's harms so far: giving some of the value back to the users who create it. The price of a transaction's journey In the early dark-forest period, almost every transaction went to the public mempool. A user's wallet sent it to a node, and the node broadcast it. That made it visible to every searcher, which was the root of front-running, but it also meant that no single party had privileged access. Every searcher saw the same mempool. As private channels spread, this changed. A wallet could send its users' transactions to Flashbots Protect, or to another private endpoint, or directly to a builder. Each of these choices protected the user from public-mempool predators. Each also gave the recipient something valuable: exclusive sight of transactions that rivals could not see. A builder that receives a popular wallet's transactions before any other builder can include them in its own blocks, back-run them for profit, and bid more for the slot. That makes the wallet's order flow worth money, and it gives the wallet a choice: protect its users, sell their flow to the highest bidder, or try to do both. Traditional finance has faced this choice for decades. In the United States, retail brokerages commonly route customer stock orders to wholesale market makers that pay for the privilege, a practice called payment for order flow. The market makers profit because retail orders are, on average, uninformed: the retail trader is unlikely to know something the market maker does not, so trading against them is less risky than trading against professionals. The broker earns revenue that lets it offer commission-free trading. Defenders argue that retail customers receive better prices than they would on public exchanges. Critics argue that the arrangement creates a conflict of interest and hides costs. The debate intensified after the retail trading surge of early 2021 and has not been resolved. Crypto order flow markets reproduce this structure, with a twist. On a blockchain, uninformed retail flow is valuable not only because it is safe to trade against, but because its exact content can be seen before execution and its ordering can be bought. The value of crypto order flow is therefore a mixture of the ordinary market-making value familiar from traditional finance and the MEV value created by visibility and ordering control. The question is who captures that value: the user, the wallet or application, the searcher, the builder, or the validator. The previous chapter described how builders that receive exclusive order flow can win more auctions, and how winning more auctions attracts more flow. Analyses by researchers and analytics firms, including EigenPhi, have attributed a large share of the leading builders' income to private order flow, as opposed to transactions visible in the public mempool. Much of that private flow comes from a small number of sources: popular wallets, trading front-ends, and, notably in 2023 and 2024, trading bots operated through messaging apps such as Telegram, which let users buy newly launched tokens quickly and which sent their users' transactions to particular builders. These arrangements are often opaque. A wallet or bot may route all its users' transactions to one or two builders, in exchange for payment, rebates, or promises of protection. From the builder's perspective, the flow is valuable. From the user's perspective, the arrangement may protect them from sandwiching by public-mempool bots while exposing them to whatever the receiving builder chooses to do. From the network's perspective, every exclusive arrangement strengthens the builders who already have the most flow, reinforcing the oligopoly that the Wu, Thiery, Leonardos and Ventre study documented. Exclusive flow is therefore a centralizing force at least as strong as latency or capital. A new builder entering the market faces a chicken-and-egg problem: it cannot win slots without exclusive flow, and it cannot attract exclusive flow without winning slots. The barrier to entry is not technical skill, which many teams have, but a commercial position that the incumbents already occupy. Giving the value back If order flow is valuable, one response is to make sure the user who generates it gets paid. Several systems launched in 2023 tried to do exactly that, and their common design is now called an order flow auction. Flashbots' version is MEV-Share. A user who submits a transaction through Flashbots Protect has it sent to a MEV-Share node rather than the public mempool. The node does not reveal the full transaction to searchers. Instead, it shares selected hints, such as which pool the transaction touches, according to the user's privacy settings. Searchers who think they can profit from the transaction, typically by back-running it with an arbitrage, submit partial bundles that place their transaction after the user's, along with a bid. The node simulates these bundles and forwards the successful ones to builders, on the condition that the user receives a share of the value. By default, 90 percent of the MEV the user's transaction generates is refunded to the user. In its initial form, MEV-Share permitted only back-runs, which do not harm the user's own execution; front-running and sandwiching were excluded by design. MEV Blocker, launched on 5 April 2023 and developed jointly by CoW Swap, the Agnostic relay team and the builder beaverbuild, with support from a number of other Ethereum projects, took a similar approach. Users who set their wallet's network endpoint to MEV Blocker have their transactions sent to a network of searchers who bid for the right to back-run them, and the system commits to returning at least 90 percent of winning bids to the user, with the remainder going to validators. Like MEV-Share, MEV Blocker is meant to protect users from sandwiching while letting them capture the value of the arbitrage their trades create. These systems turn the economics of the previous chapters around. In the dark forest, a user's trade was a resource that searchers fought over and the user lost. In an order flow auction, the user's trade is still a resource that searchers fight over, but the competition drives payments to the user rather than solely to the block producer. If enough searchers compete, the user receives most of the back-run value, and the sandwich is excluded. Table 3 compares the main routes a transaction can take. Table 3. Main routes from a user's wallet to a block. Route Who sees the transaction first Sandwich exposure Who captures back-run value Main trust assumption Public mempool Everyone High Searchers and builders None, but fully exposed Private RPC (e.g. Flashbots Protect) Operator and its builders Low Builders Operator does not exploit flow Order flow auction (e.g. MEV-Share, MEV Blocker) Operator; searchers see hints Low Mostly the user, via refunds Operator enforces rules honestly Intent-based exchange (e.g. UniswapX, CoW Swap) Solvers or fillers Very low Shared via price improvement Competition among solvers A further step changes what the user submits in the first place. Instead of signing a transaction that specifies exactly what contract to call and how, the user signs an intent: a statement of the outcome they want, such as "sell 10 ETH for at least 20,000 USDC within the next two minutes." Specialists called solvers or fillers compete to fulfil the intent, using whatever combination of on-chain pools, off-chain inventory and other users' orders gives the best result. The winning solver executes the trade and the user receives the agreed outcome. UniswapX, introduced by Uniswap Labs in July 2023, uses a variant of this idea in which an order's price starts high and decays over time, a Dutch auction, until a filler finds it profitable to execute. CoW Swap, discussed further in the next chapter, collects intents into batches and has solvers compete to settle each batch at the best overall prices. In both designs, the user does not broadcast a transaction to the public mempool. The solver bears the execution risk and pays the gas. The user's order cannot be sandwiched in the usual way because the solver, not the user, decides how it reaches the chain, and the competition among solvers is supposed to ensure the user gets a price close to the best available. Intent-based systems shift the problem rather than solving it. The solver now holds the information that searchers used to fight over. If solvers compete vigorously, users benefit. If the solver market concentrates, as builder markets have, a few solvers could extract value from users with much less visibility than a public sandwich. Solvers also tend to be sophisticated trading firms with their own inventory and their own access to centralized exchanges, which gives them structural advantages similar to those of leading builders. Several analyses have observed that the solver and builder markets overlap, with the same firms or their affiliates active in both. One trade, three journeys The differences among these routes are easiest to see by following a single trade along each. Return to the trader from Chapter 3, who wants to spend 100,000 USDC on ether in a pool holding 1,000 ETH and 2,000,000 USDC, with a 1 percent slippage tolerance. On the first journey, the trade goes to the public mempool. A sandwich bot sees it, computes that it can push the price until the trader's tolerance binds, and submits a bundle with its buy before and its sell after. Builders include the bundle because it pays them well. The trader receives about 47.14 ETH instead of about 47.62. Then, because the trade and the sandwich have left the pool's price well above the market, an arbitrageur sells into the pool in the next position and captures the difference. The trader has lost roughly half an ether to the sandwich. The arbitrage value created by their own trade has gone to the arbitrageur and, through bids, to the builder and proposer. The trader receives nothing from either. On the second journey, the trade goes through an order flow auction. It never appears in the public mempool, so no sandwich bot sees it. The auction reveals a hint, perhaps only that a trade is touching this particular pool, and searchers bid to back-run it. The trade executes at about 47.62 ETH, the honest price. The winning searcher's back-run then captures the arbitrage created by the trade's price impact, and most of the searcher's bid is paid to the trader as a rebate. The trader's outcome is better on both counts: no sandwich loss, and a share of the value their trade created. On the third journey, the trader signs an intent on an intent-based exchange: sell up to 100,000 USDC for at least a stated amount of ether. Solvers compete to fill it. One solver might fill part of the order from another user who wants to sell ether at the same moment, avoiding the pool entirely and its price impact. Another might route the rest across several pools to reduce impact, or fill it from its own inventory and hedge on a centralized exchange. The winning solver offers the trader a price that may be better than the single-pool price of the second journey, because it can draw on more liquidity. The trader cannot be sandwiched, because they did not submit a transaction to the chain at all. The value that would have gone to a back-run is competed into the price instead of paid as a rebate. The comparison shows why order flow auctions and intents are not merely defensive. They reverse the direction of the transfer. But the comparison also reveals what each journey assumes. The second journey assumes that the auction operator did not leak the full transaction to a favoured searcher. The third assumes that solvers competed vigorously and that the winning solver did not exploit its knowledge of the order in ways the user cannot see. In the first journey, the user was exploited openly. In the second and third, the user is protected by the honesty and competitiveness of intermediaries whose internal behaviour is much harder to observe. The new intermediaries The order flow auction era illustrates the book's controlling claim with unusual clarity. These systems do not eliminate MEV. A back-run is still a back-run, an arbitrage is still an arbitrage, and someone is still paying for ordering. What changes is who receives the value and who must be trusted to enforce the rules. On distribution, the change is genuinely progressive. Users who route through an order flow auction receive payments that previously went to searchers, builders and validators. Users who use intent-based exchanges receive better prices. Sandwiching through these channels is suppressed. These are real benefits, and their adoption by wallets as defaults has meaningfully reduced the exposure of ordinary users. On trust, the change is more complicated. Each order flow auction is operated by a specific organization that sees users' transactions before anyone else and promises to share them only in particular ways. Each intent system depends on a set of solvers whose behaviour is constrained by competition and by protocol rules, not by cryptography. The user has replaced exposure to anonymous predators with reliance on identifiable intermediaries. That is often a good trade, and in practice those intermediaries have generally behaved as promised. But it means that the safety of an ordinary user's trade now rests on the honesty and competence of a few organizations, on their resistance to legal and commercial pressure, and on the correctness of their software. On structure, the effect is ambiguous. Order flow auctions can reduce builder concentration if they distribute bundles to many builders rather than one, which some are explicitly designed to do. They can also become new chokepoints if a few auctions attract most of the flow. The economics push toward concentration at every layer where scale brings information advantage, and an order flow auction is a place where information aggregates. A final caution concerns what rebates cannot fix. Order flow auctions return back-run value to users, but much of MEV is not back-run value created by a single user's trade. The largest category, arbitrage between centralized exchanges and on-chain pools, is driven by prices moving on off-chain venues, not by any user's transaction. The loss it imposes falls on liquidity providers, who do not submit transactions through order flow auctions at all. Rebating users does nothing for them. Their remedy has to come from the design of the pools themselves: fees that rise with volatility, pools that update prices from external feeds, or mechanisms that auction the right to make the first trade after each block. Some of these designs, made possible by Uniswap's version 4, which launched in early 2025 with programmable "hooks" that let pool creators customize behaviour, are being tried, but none has yet become standard. Rebates also do nothing about the concentration of block building or the trust placed in relays. They operate within the existing supply chain, improving the deal for users while leaving the chain's structure largely as it was. To change that structure, one has to change who can see transactions and when, or change the protocol itself. The next chapter turns to the first of those approaches: the cryptographic and hardware-based attempts to keep transactions hidden from everyone, including the builder, until their order is fixed. Chapter 7: Cryptography Against the Forest Every mitigation described so far has relied on someone's promise. Flashbots Protect relies on Flashbots and its builders not exploiting the transactions they receive. Order flow auctions rely on their operators enforcing the rules. Relays rely on their own honesty and the correctness of their code. The defining aspiration of blockchain technology is to replace promises with mechanisms whose behaviour can be verified, and the most ambitious responses to MEV try to do exactly that. Their shared goal is to make it impossible for anyone, including the party that orders transactions, to see what a transaction does until its position in the block is fixed. If nobody can see a trade before it is ordered, nobody can sandwich it, and the advantage of privileged access to order flow largely disappears. This chapter examines four families of approach: threshold-encrypted mempools, trusted execution environments, batch auctions, and more advanced cryptography still largely at the research stage. Each delivers part of what it promises. Each has limits that say something about the nature of MEV itself. Table 4, near the end of the chapter, compares them. The simplest version of the idea predates blockchains. In a commit-reveal scheme, a participant first publishes a commitment, a cryptographic fingerprint of their intended action that reveals nothing about the action itself, and later publishes the action. Anyone can verify that the action matches the commitment. Because the commitment comes first and is fixed, nobody can react to its contents before it is locked in. Applied to transactions, a user could submit a commitment in one block and the actual trade in a later block, with the protocol enforcing that the trade executes in the order the commitments were received. The scheme defeats front-running in principle, but it is clumsy in practice. It requires two transactions instead of one, doubling cost and latency. It leaves open what happens if a user commits and then fails to reveal, which can be used to grief others or to gain an option on market movements, since a user who sees the market move against them can simply decline to reveal. And because the reveal itself is a public transaction, an observer can still react to it within the same block unless ordering is fixed at commit time. Commit-reveal is used in some narrow applications, such as certain auctions and name registrations, but it has not been a general solution. Threshold-encrypted mempools A more general approach encrypts every transaction so that it cannot be read until after it has been ordered, and arranges for decryption to happen automatically. The difficulty is deciding who holds the decryption key. If one party holds it, that party can decrypt early and exploit what it sees. The answer is threshold cryptography: the decryption key is split among a committee of participants in such a way that any sufficiently large subset of them, the threshold, can decrypt together, but no smaller group can. In a threshold-encrypted mempool, users encrypt their transactions with the committee's public key and submit them. The block producer orders the encrypted transactions without knowing their contents. Once the order is committed, the committee members each release their share of the decryption key, the transactions are decrypted, and they execute in the committed order. If the committee follows the protocol, nobody sees the contents of any transaction until its position is fixed. Shutter Network has built the most prominent production version of this design. Its committee members, called keypers, jointly generate a public encryption key through a distributed key generation process, so that no single keyper ever holds the full private key. Gnosis Chain, an Ethereum-compatible network, deployed Shutter-based encrypted transactions in 2024, offering users an alternative endpoint that encrypts their transactions before submission. The deployment has exposed both the design's value and its costs. Its operators have acknowledged that it is not trustless, because users rely on a permissioned committee whose members could in principle collude to decrypt early, and that inclusion has been much slower than ordinary transactions, partly because only some validators participate. The limitations generalize beyond any single deployment. The first is the committee itself. A threshold scheme is only as secure as the assumption that fewer than the threshold of members will collude. That assumption may be reasonable for a well-chosen committee, but it is a trust assumption, not a mathematical guarantee. Members who collude to decrypt early leave no trace. The second limitation is metadata. Encryption hides a transaction's contents, but not everything about it. The sender's address, the transaction's size, the fee offered and its timing may all be visible. A sophisticated observer who knows that a particular address trades a particular token, and sees an encrypted transaction of the expected size from that address, can make an educated guess about what it contains. Hiding the sender requires additional machinery, which adds cost and complexity. The third limitation is validation. A block producer needs to know that a transaction can pay its fees before including it, or the chain can be flooded with encrypted junk that consumes space and fails on decryption. Checking that an encrypted transaction is valid without decrypting it requires either cryptographic proofs, which are expensive, or rules that charge a fee regardless of whether the decrypted transaction succeeds. The fourth limitation is the most fundamental. Encryption protects a transaction from being exploited before it executes. It does nothing about the value created after it executes. Once the transactions in a block are decrypted and run, the prices they move are visible to everyone, and arbitrageurs can trade against those prices in the next block. Encrypted mempools can eliminate sandwiching, which depends on seeing a trade in advance. They cannot eliminate arbitrage, which depends on prices that differ across venues, or the loss that liquidity providers suffer when their prices are stale. They remove the harmful, avoidable kind of MEV while leaving the intrinsic kind intact. A further subtlety concerns what happens at the moment of decryption. Once the committee releases its shares, every transaction in the batch becomes readable at once, and whoever is positioned to act first on the newly visible information gains an edge. If decryption occurs after a block's order is final, the beneficiary is whoever builds the next block, who can place an arbitrage immediately after the newly executed trades. The value has not been destroyed; it has moved from the party who could see the trade early to the party who can react to it first. Designers try to minimize this by making decryption and execution happen as close together as possible, but the residual advantage of being first to respond is, again, a property of markets, not of mempools. Trusted hardware A different approach replaces trust in a committee with trust in a chip. A trusted execution environment, or TEE, is a protected region of a processor in which code runs in isolation from the rest of the machine, including its operating system and the person who owns it. The chip can produce a cryptographic attestation, a signed statement that a specific piece of code is running inside a genuine protected environment. Intel's Software Guard Extensions (SGX) and its newer Trust Domain Extensions (TDX), and comparable technologies from AMD and others, provide versions of this capability. For MEV, the appeal is direct. If a block builder runs inside a TEE, users and searchers can send transactions to it encrypted so that only the attested code can read them. The builder's operator cannot see the order flow, cannot front-run it, and cannot share it. Searchers' strategies are protected from the builder. Rules about how bundles are combined and how refunds are paid can be written into the attested code and verified by anyone. In principle, many independent operators could run the same attested builder software, sharing order flow among themselves, so that no single firm's commercial position would determine who sees which transactions. Flashbots announced a long-term project in this direction, called SUAVE, in late 2022, describing it as a shared, decentralized platform for expressing and executing preferences about ordering across many chains. The project's scope narrowed over time toward its most practical component. In November 2024, Flashbots, the builder beaverbuild and the client team Nethermind announced BuilderNet, a network of block builders running inside TEEs that share order flow with one another and redistribute MEV according to transparent rules. The stated aims were to provide a neutral alternative to exclusive order flow deals, to improve censorship resistance, and to reduce the concentration of block building. Flashbots and beaverbuild said they would migrate their existing centralized builders into the network. BuilderNet is a significant experiment because it attacks the root of the builder oligopoly identified in Chapter 5: exclusive access to order flow. If flow sent to any BuilderNet operator is shared with all of them, no operator gains an exclusive advantage from attracting it. The advantage that compounded into concentration is, in principle, neutralized. The weaknesses are in the hardware. Trusted execution environments have a long history of vulnerabilities. Researchers have repeatedly demonstrated side-channel attacks against SGX that extract secrets by measuring timing, power consumption or other indirect signals, and manufacturers have repeatedly issued fixes. A TEE's security also depends on the manufacturer: the attestation is ultimately rooted in keys controlled by Intel or AMD, and users must trust that those companies' processes and supply chains are sound. A design that replaces trust in Flashbots with trust in Intel has changed the trusted party, not removed it. For many purposes, that is a reasonable trade, since a chip manufacturer has less incentive and less ability to front-run a particular trade than a builder does. But it is a trade, and it introduces a dependency on a few large hardware vendors whose products were not designed with adversarial financial settings in mind. Making order irrelevant A third approach does not hide transactions at all. Instead, it changes the market so that order within a batch does not matter. The intellectual foundation comes from traditional finance. In 2015, Eric Budish, Peter Cramton and John Shim published "The High-Frequency Trading Arms Race: Frequent Batch Auctions as a Market Design Response" in the Quarterly Journal of Economics. They argued that continuous limit order books, in which orders are processed one at a time in the order they arrive, create a perpetual race to be first. Even tiny speed advantages yield profits, so firms invest enormous sums in faster connections, and the investment is socially wasteful because it changes who wins without improving prices. Their proposed remedy was to process orders in discrete batches, say every tenth of a second, and to clear each batch at a single uniform price. Within a batch, arrival order is irrelevant. Speed advantages shrink to the question of whether an order makes it into a batch at all. The parallel with MEV is exact. Sandwiching and much of front-running depend on precise ordering within a block. If all trades in a batch for a given pair of assets clear at the same price, there is nothing to gain by placing a buy before someone else's buy, because both receive the same price. CoW Protocol, whose trading interface is CoW Swap, applies this logic on Ethereum. Users sign intents rather than transactions. The protocol collects intents into batches and holds an auction among solvers, who compete to propose a settlement for the whole batch. Within a batch, each token pair clears at a uniform price. Where users' orders offset each other, a "coincidence of wants," the protocol can match them directly without touching an external pool, avoiding fees and price impact entirely. Where they do not, solvers route the remainder through on-chain liquidity. The winning solver is the one whose settlement delivers the most value to users, and the settlement is submitted as a single transaction, so its internal ordering cannot be exploited by outsiders. Batch auctions face their own constraints. They introduce latency, since users must wait for a batch to close. They depend on solver competition, and solvers, like builders, are sophisticated trading firms whose market may concentrate. And they protect only the trades inside the batch. Arbitrage between the batch's clearing price and prices elsewhere still exists, and liquidity providers in external pools still trade at stale prices. But for the user's own trade, batch auctions offer something neither private channels nor encryption fully provide: a structural reason that ordering cannot be used against them. The frontier Beyond these deployed approaches lies a body of research that aims to remove trust assumptions altogether. Fully homomorphic encryption allows computation on encrypted data without decrypting it, so in principle a builder could assemble and optimize a block from encrypted transactions without ever seeing them. Secure multiparty computation allows several parties to jointly compute a result without any of them seeing the others' inputs. Delay encryption and related ideas encrypt a transaction so that it can be decrypted by anyone, but only after a predictable amount of computation that cannot be shortcut, removing the need for a committee. These techniques are real and improving, but for general block building they remain far too slow or too costly for production use. The interesting question is not whether they will eventually be practical, which seems likely for narrow uses, but whether they would change the basic picture if they were. They would not. Even perfect encryption until execution leaves intrinsic MEV in place, because prices move between blocks and someone must bring on-chain prices into line with the rest of the world. Cryptography can eliminate the MEV that comes from seeing intentions too early. It cannot eliminate the MEV that comes from the existence of markets. Table 4. Approaches to hiding or neutralizing transaction order. Approach Example What it protects against Main remaining trust or cost Commit-reveal Some auctions and registrations Front-running of committed actions Two transactions; option to not reveal Threshold-encrypted mempool Shutter on Gnosis Chain Sandwiching and front-running Committee honesty; metadata leakage; latency Trusted execution environment BuilderNet Builder misuse of flow; exclusive flow Hardware vendor; side-channel attacks Frequent batch auction CoW Protocol Ordering within the batch Solver competition; batch latency Homomorphic or multiparty computation Research prototypes Nearly all pre-execution exploitation Performance far from practical The phrase "democratizing block building" is often used for these efforts, and it is worth being precise about what it can mean. It cannot mean that every participant earns the same MEV, because some participants have better information about prices and better models of the market, and those advantages persist after execution. It can mean three narrower things. It can mean that no participant gains an advantage merely by seeing other people's transactions first, which encryption and TEEs address. It can mean that no participant gains an advantage from exclusive commercial arrangements over order flow, which shared builder networks address. And it can mean that users are not systematically harmed by the ordering of their own trades, which batch auctions and encryption both address. Those are substantial goals. Achieving them would remove most of the harm described in Chapter 3 and much of the concentration described in Chapter 5. But each approach achieves them by placing trust somewhere: in a committee, in a chip, in a set of solvers. The honest assessment is that cryptography and hardware have narrowed the range of parties that must be trusted, and made their behaviour more verifiable, without eliminating trust. The remaining question is what the base protocol itself should guarantee, and the next chapter turns to the redesigns that are moving parts of the MEV supply chain into Ethereum's rules. Chapter 8: Rewriting the Protocol The arrangements described so far all live outside the core rules of Ethereum. MEV-Boost, relays, builders, order flow auctions and encrypted channels are software and businesses built on top of the protocol, which itself still says only that a proposer may put valid transactions in any order it likes. That gap between what the protocol specifies and how blocks are actually produced has been the source of most of the fragility this book has described: the trusted relays, the censorship episode, the relay exploit, the builder oligopoly. Ethereum's researchers have long argued that some of this machinery should be brought inside the protocol, where its behaviour is guaranteed by consensus rather than by the good conduct of particular companies. In 2026 the first major step in that direction is arriving, and several others are being debated. Other chains, meanwhile, have made their own choices about who orders transactions, and those choices make a useful comparison. Enshrined proposer-builder separation The upgrade known as Glamsterdam, which Ethereum's developers have been preparing through 2026, includes as one of its two headline changes EIP-7732, enshrined proposer-builder separation, usually shortened to ePBS. The Ethereum Foundation announced in September 2026 that Glamsterdam would activate on the Sepolia test network on 6 October 2026, with dates for the remaining test network and for mainnet to be set later by the client teams. As of this writing, ePBS is therefore scheduled but not yet live on Ethereum mainnet. The design moves the core of MEV-Boost's function into the protocol. Builders bid for the right to produce a block's execution payload, and the proposer commits to a builder's bid in the consensus-layer block it proposes. The builder then reveals the full payload. A new committee of validators, the payload timeliness committee, attests to whether the builder revealed its payload on time. The protocol enforces the builder's payment to the proposer even if the builder fails to deliver, so the proposer no longer needs to trust an intermediary to vouch that the payment is real and the block will be published. The change addresses the trust problem that made relays both necessary and dangerous. Under MEV-Boost, the relay guaranteed to proposers that a block was valid and would be published, and guaranteed to builders that their blocks would not be stolen. Under ePBS, the proposer's protection comes from the protocol's enforcement of payment, and the builder's protection comes from the fact that the proposer commits to the bid before seeing the payload. Relays are no longer required for the basic exchange. They may persist for other reasons, such as offering builders the ability to update or cancel bids at the last moment, or providing lower latency, and early analysis suggests some will. But they would no longer be the indispensable trusted parties at the centre of block production. What ePBS does not do is equally important. It does not change who the builders are or why they win. The builder that receives the most exclusive order flow and runs the most profitable trading operation will still offer the highest bids, and the market will still tend to concentrate. Enshrining the auction makes it safer. It does not make it more competitive. Nor does ePBS by itself prevent a dominant builder from excluding transactions it does not wish to include, which brings us to the next change. The censorship episode of 2022 showed that a concentrated builder market can exclude transactions. The proposer, under PBS, has no say over block contents; it signs whatever the winning builder built. If a few builders refuse certain transactions, those transactions wait until a validator that builds its own blocks, or uses a non-filtering builder, is selected. Inclusion lists are the standard proposed remedy. The idea is to let validators, who are numerous and decentralized, specify transactions that must be included, constraining builders who would otherwise be free to leave them out. The leading design, called FOCIL for fork-choice enforced inclusion lists and specified in EIP-7805, assigns a committee of validators in each slot to observe the public mempool and publish lists of transactions they see. A block is considered valid by the fork-choice rule, the process by which validators decide which chain to follow, only if it includes the transactions on those lists, provided there is room. Because the lists come from several validators, a builder cannot evade them by bribing or pressuring a single party. FOCIL was considered for Glamsterdam but, according to public reporting on developers' deliberations, was moved to the following upgrade, known as Hegotá, after concerns that combining it with ePBS in one upgrade would create untested interactions. The sequencing matters. Until inclusion lists are live, ePBS makes the builder auction trustless but leaves builders' power over contents intact. Inclusion lists are the piece that turns PBS from a mechanism for efficient extraction into one that also protects the chain's neutrality. Inclusion lists have a limit relevant to MEV specifically. They guarantee that a transaction is included, not where. A builder required to include a user's trade can still place it wherever it likes in the block, including between two of its own transactions. Inclusion lists defend against censorship. They do not defend against sandwiching. Burning the rent A more radical class of proposals asks what should happen to MEV once it has been captured. Under current arrangements, the builder's payment goes to the proposer. Because the proposer is chosen at random, MEV makes validator income lumpy: a validator lucky enough to propose during a market crash may earn many times its usual reward. That randomness encourages stakers to pool their resources, since pools smooth out the variance, which is a mild centralizing pressure. It also means that the security budget of the chain depends partly on trading activity, which is volatile. MEV burn, proposed by Ethereum researcher Justin Drake and others in 2023, would change the destination. In outline, the protocol would observe the highest builder bids in each slot and require the winning builder to burn an amount close to that figure, rather than paying it to the proposer. The proposer would keep only a small remainder. The value of ordering would then accrue to all holders of ether through reduced supply, rather than to whichever validator happened to be selected. Variance in validator income would fall, removing a reason to pool, and incentives for proposers to play timing games or collude with builders would weaken, since there would be little to gain. MEV burn has not been scheduled for any upgrade. It depends on enshrined PBS being in place, it raises difficult questions about how the protocol can reliably observe the true value of a block when builders might pay proposers through side channels, and it changes the economics of staking in ways that require careful analysis. Related proposals go further, separating the right to propose consensus blocks from the right to produce execution payloads altogether, by selling the latter in advance through mechanisms such as execution tickets or execution auctions. These are research ideas rather than plans. They share a view that MEV is best treated as a protocol-level resource, priced and allocated by the protocol, rather than a side income for whoever is selected to propose. Rollups and other chains Much of Ethereum's trading activity now takes place on rollups, separate chains that execute transactions off the main network and post compressed data and proofs back to it. Most rollups today rely on a single sequencer, typically operated by the team that built the rollup, which receives users' transactions and decides their order. A single sequencer is a centralized point of control, but it also offers an opportunity: the rollup can choose an ordering policy and commit to it, rather than inheriting Ethereum's auction. Different rollups have chosen differently. Arbitrum's sequencer originally processed transactions on a first-come, first-served basis. That eliminated the auction for ordering but created a latency race: searchers competed to get their transactions to the sequencer fastest, investing in network infrastructure and flooding the sequencer with speculative transactions. The policy that was meant to be fair in fact rewarded proximity and speed, much as Budish and his colleagues had warned about continuous markets. In April 2025, Arbitrum One activated Timeboost. Under Timeboost, the rollup holds a regular auction, by default once a minute, for the right to an "express lane." Transactions from the auction's winner are processed immediately, while all other transactions are subject to a short artificial delay, 200 milliseconds by default. The design accepts that someone will gain an ordering advantage and sells that advantage openly, directing the proceeds to the chain's owner, which for Arbitrum One is the Arbitrum DAO. Reports in mid-2025 put Timeboost's revenue in its first three months at around $2 million. The approach does not reduce MEV. It converts a wasteful latency race into a priced auction and redirects part of the value from searchers to the rollup's community. Its critics note that it institutionalizes a privileged lane; its defenders note that the privilege existed already and was simply going to whoever had the fastest connection. Other rollups have ordered transactions by priority fee, reproducing Ethereum's own fee-based competition in a simpler form, or have experimented with trusted hardware for block building, applying ideas from the previous chapter. The variety is itself instructive. A rollup with a single sequencer has the power to adopt almost any ordering rule, and every rule chosen so far either sells ordering, rewards speed, or relies on trust in the sequencer's hardware or honesty. None makes ordering worthless. Solana illustrates the same dynamics in a different architecture. Solana was designed without a public mempool in the Ethereum sense; transactions are forwarded directly to the validators scheduled to produce upcoming blocks. That was expected to limit front-running. In practice, a modified validator client developed by Jito Labs, which lets searchers submit bundles with tips to validators running it, became widely used, and with it an auction for ordering emerged much like Flashbots' on Ethereum. Jito also operated a mempool-like service that let searchers see pending transactions. In March 2024, Jito Labs shut that service down, citing its use for sandwich attacks against Solana users. Closing the visible mempool did not end sandwiching. Private arrangements in which some validators shared pending transactions with sandwich operators emerged in its place. The Solana Foundation responded by removing validators it identified as participating in such arrangements from its stake delegation program, a governance lever that Ethereum, with no comparable central delegator, does not have. The episode replays, in compressed form, the trajectory traced in this book: a public venue for ordering is closed to protect users, the activity moves into private channels, and the remedy turns on which parties can be identified and pressured. Table 5 summarizes how different venues have approached the allocation of ordering rights. Table 5. How different venues allocate transaction ordering. Venue Who orders transactions How ordering rights are priced Status as of October 2026 Ethereum (MEV-Boost) Builders, via relays Sealed-bid auction paid to proposer In use since September 2022 Ethereum (ePBS, EIP-7732) Builders, via protocol In-protocol bids paid to proposer Scheduled with Glamsterdam; testnet from October 2026 Arbitrum One (Timeboost) Single sequencer Per-minute express lane auction Live since April 2025 Solana (Jito client) Leader validator with bundle auction Tips to validators Widely used; public mempool closed March 2024 CoW Protocol Winning solver per batch Solver competition on user surplus Live on Ethereum and other chains Answers at the application layer Protocol changes decide who holds ordering rights and how they are priced. Applications decide how much value there is to fight over in the first place. Some of the most promising recent work on MEV has happened not in Ethereum's core rules but in the design of the lending markets and exchanges whose behaviour creates MEV. Lending protocols offer the clearest case. When a price oracle reports a new price that makes loans eligible for liquidation, the first liquidator to act after the update collects the bonus, and searchers compete for the position immediately behind the oracle transaction. The value of that position comes from the protocol's own liquidation bonus, paid out of borrowers' collateral, and it flows to searchers and block producers. Several projects have proposed that the protocol, or the oracle, should instead auction the right to perform the first liquidation after each update, returning most of the proceeds to the protocol and so to its users. This category has come to be called oracle extractable value, and oracle providers including API3 and Chainlink have built services intended to recapture it for the applications that use their price feeds. The approach does not eliminate the liquidation race. It moves the race from the block builder's auction to the application's auction, where the application decides who gets paid. Exchanges offer a parallel case. The loss liquidity providers suffer to arbitrage, described in Chapter 3, is a transfer from the pool to whoever trades first against a stale price. A pool that knew the market price would not quote a stale one, and a pool that charged the first trader of each block for the privilege could capture part of the arbitrage for its providers. Researchers have proposed designs in both directions. Some pools raise fees when volatility rises, so that arbitrageurs pay more when their profits are largest. In 2024, Austin Adams, Ciamac Moallemi, Sara Reynolds and Dan Robinson proposed an auction-managed automated market maker, in which the right to act as the pool's privileged manager, setting its fee and trading against it without paying that fee, is continuously auctioned, with the rent paid to liquidity providers. Uniswap's version 4, with its programmable hooks, made it far easier for pool creators to deploy designs of this kind without building a new exchange from scratch. These application-level answers share a logic with the protocol-level ones. Both accept that the value of ordering exists and try to direct it deliberately to those who generated it. Their advantage is speed and specificity: an application can redesign its own auction in months, whereas a change to Ethereum's consensus rules takes years. Their limitation is fragmentation. Each application recaptures its own MEV under its own rules, and users must trust each design separately. A chain where every application runs its own ordering auction is safer for users of the applications that do it well, and no safer for users of those that do not. The protocol-level changes now arriving on Ethereum are significant. Enshrined PBS removes the need for trusted relays in the basic exchange between builders and proposers. Inclusion lists, once added, will give the decentralized validator set a guaranteed role in deciding what goes into blocks. If MEV burn or a similar mechanism eventually follows, the value of ordering would flow to the chain's community as a whole, and much of the incentive to manipulate block production would fade. None of these changes touches the source of MEV. Transactions remain public before execution unless users take steps to hide them. Prices remain stale between blocks. Builders with better information remain better builders. The protocol can decide who must be trusted, who gets paid, and who cannot be excluded. It cannot decide that ordering is worthless, because ordering is worth something as long as there are markets on the chain whose outcomes depend on sequence. The remaining choices belong to the people who design those markets and the people who use them, which is where the conclusion turns. Conclusion The history traced in this book can be told as a series of solutions, each answering the failure of the one before. Public bidding wars clogged the network, so Flashbots built a private auction. Proof-of-stake threatened to centralize validators, so MEV-Boost separated proposers from builders. The relays that made that separation work turned out to be trusted chokepoints, so Ethereum is enshrining the auction in the protocol. Builders concentrated around exclusive order flow, so shared builder networks running on trusted hardware try to neutralize the advantage. Users were sandwiched, so order flow auctions and intent-based exchanges route their trades away from predators and pay them rebates. Each step was a real improvement on some dimension. Yet after six years of sustained effort by some of the most capable people in the field, MEV is not smaller in any meaningful sense. It has been relocated, repriced and made safer to handle. It has not gone away. That is not a failure of effort or imagination. It is what the controlling claim of this book predicts. MEV is the rent on a scarce right, the right to decide transaction order, in a system where outcomes depend on order and intentions are visible before they are final. As long as both conditions hold, the right has value, and someone will hold it. The useful questions are the ones each chapter has asked in turn: who captures the rent, how much leaks out as harm to people who never chose to pay it, and what concentrations of power the capture creates. Several conclusions follow from taking those questions seriously. Separate the intrinsic from the avoidable The first conclusion is that the most important distinction in the field is between MEV that comes from the existence of markets and MEV that comes from design choices. Arbitrage between a stale pool and a moving market is intrinsic: wherever prices update between blocks, someone will correct them and be paid for doing so. Sandwiching is avoidable: it exists because user intentions are broadcast before execution and because automated market makers execute against any counterparty at the formula's price. Mitigations should be judged by which kind they address. Encrypted mempools, private channels and batch auctions attack the avoidable kind directly and can, in principle, eliminate it. Nothing that operates on ordering alone can eliminate the intrinsic kind. The appropriate response to intrinsic MEV is not to hide it but to price it, and to decide deliberately who receives the price. Liquidity providers who lose to arbitrage every time the market moves are paying a cost that pool designs can recapture, for instance through fees that rise with volatility or by auctioning the right to trade first against a stale price. The design space opened by programmable pools is the most promising place to address the largest category of MEV, and it receives far less public attention than sandwiching. Every mitigation names a trusted party The second conclusion is that every serious mitigation, examined closely, replaces one trusted party with another. Flashbots Protect asked users to trust Flashbots and its miners. MEV-Boost asked builders and proposers to trust relays. Order flow auctions ask users to trust their operators. Threshold encryption asks users to trust that a committee will not collude. Trusted execution environments ask users to trust chip manufacturers and the absence of side-channel attacks. Even enshrined PBS, which removes the relay from the core exchange, leaves users trusting that a concentrated builder market will not abuse its control over block contents until inclusion lists arrive. This is not an argument against any of these designs. Trust is not binary, and moving it from an anonymous predator to an identifiable intermediary, or from an intermediary with an incentive to cheat to a chip manufacturer without one, is often an improvement. It is an argument for asking the right question when evaluating a proposal. "Does it eliminate MEV?" is the wrong question, because the answer is always no. "Whom does it ask us to trust, and what disciplines them?" is the right one. Competition, reputation, legal liability, cryptographic verification and protocol enforcement are different disciplines with different strengths, and a design that leans on a weak one should be treated with caution however elegant its mathematics. Efficiency concentrates The third conclusion is the most uncomfortable for a technology whose central promise is decentralization. Every time the MEV market was made more efficient, it concentrated. Sealed-bid auctions rewarded searchers with the best strategies. Proposer-builder separation protected validators by pushing concentration into building, where two firms came to win the large majority of auctions. Private order flow, which protects users, became the asset that entrenched those firms. Order flow auctions and intent systems are already showing signs of the same dynamic among their solvers. The mechanism is not conspiracy but compounding. In any market where information yields profit and profit buys more information, the best-informed participant grows. Blockchains do not escape this simply by having open protocols. Openness ensures that anyone may enter; it does not ensure that entrants can compete with incumbents who see more order flow and run larger trading operations. If decentralization in block building is to be more than nominal, it must be designed for directly, through shared order flow, protocol-enforced inclusion, and mechanisms that cap the advantage of scale. Leaving it to competition has been tested, and the result is an oligopoly. Default settings are policy The fourth conclusion concerns users. The single most effective protection against sandwiching has not been a protocol change or a cryptographic breakthrough. It has been the decision by wallets and applications to route users' transactions through protected channels by default, and by trading interfaces to set tighter slippage tolerances and better quotes. A protection that requires users to know about it and switch it on will be used by the sophisticated few, who are least in need of it. A protection that is on by default reaches everyone. This places a responsibility on the businesses that sit between users and the chain, and it creates a conflict. The same wallet that can protect its users by routing their flow privately can also sell that flow to the highest bidder. Traditional finance has wrestled with the equivalent conflict in payment for order flow for decades without resolving it. In crypto, where flows and payments are more visible, there is a better chance of making the terms of these arrangements public and comparable. Users and the communities that govern open protocols should insist on it. A wallet's routing policy is a financial decision made on the user's behalf, and it should be disclosed as one. Neutrality is the stake The final conclusion returns to the reason MEV matters beyond the people who trade on decentralized exchanges. A public blockchain's claim to legitimacy rests on credible neutrality: the assurance that no party can decide whose transactions are included or how they are treated. MEV is the force that tests that claim most directly, because the right to order transactions is worth a great deal and the parties who can extract the most from it can afford to pay the most to hold it. The censorship episode of 2022 showed what happens when a few firms that build most blocks face legal pressure in a single jurisdiction. The relay exploit of 2023 showed what happens when the software at the centre of block production has a flaw. Neither was catastrophic. Both were warnings. The protocol changes arriving now, enshrined separation of proposers and builders, and in due course inclusion lists, are the most important responses yet to those warnings, because they move guarantees from companies into consensus rules. They deserve support and scrutiny in equal measure. But they will not finish the job, because the job cannot be finished. A chain that hosts markets will generate MEV, and the people who capture it will try to capture more. What can be achieved is a system in which the rent is visible, its avoidable forms are designed out, its intrinsic forms are priced and directed deliberately, and no single party's commercial position or legal exposure determines what the chain will accept. That is a less romantic goal than eliminating the dark forest. It has the advantage of being possible. Glossary Automated market maker (AMM). A decentralized exchange design that quotes prices from a formula applied to pooled reserves rather than by matching buyers and sellers. Builder. A specialist that assembles complete blocks from user transactions and searcher bundles, and bids to have its block proposed. Bundle. An ordered set of transactions submitted privately for inclusion together at a specific position in a block, usually carrying a payment to the block producer. Intent. A signed statement of the outcome a user wants, such as a minimum price for a trade, which solvers compete to fulfil, instead of a fully specified transaction. Loss-versus-rebalancing (LVR). The cost to liquidity providers of trading against arbitrageurs at stale prices, measured against a portfolio rebalanced at fair market prices. Mempool. The set of pending transactions that nodes hold and share before the transactions are included in a block. Order flow auction. A mechanism in which searchers bid for the right to act on a user's transaction, typically a back-run, with most of the proceeds returned to the user. Proposer. The validator selected by the protocol to propose the block for a given slot. Proposer-builder separation (PBS). A design that splits block production between builders, who assemble blocks, and proposers, who select and sign them. Relay. A trusted intermediary in MEV-Boost that receives blocks from builders, checks them, and passes headers and bids to proposers. Sandwich attack. Placing one trade immediately before and another immediately after a victim's trade so as to profit from the price movement the victim's trade causes. Searcher. An automated system that looks for MEV opportunities and submits transactions or bundles to capture them. Notes 1. Philip Daian, Steven Goldfeder, Tyler Kell, Yunqi Li, Xueyuan Zhao, Iddo Bentov, Lorenz Breidenbach and Ari Juels, "Flash Boys 2.0: Frontrunning in Decentralized Exchanges, Miner Extractable Value, and Consensus Instability," 2020 IEEE Symposium on Security and Privacy; preprint arXiv:1904.05234 (April 2019). Source for the term MEV, priority gas auctions and time-bandit attacks (Introduction, Chapters 1 and 2). 1. Dan Robinson and Georgios Konstantopoulos, "Ethereum is a Dark Forest," Paradigm, 28 August 2020. Source for the failed rescue and the dark forest metaphor (Chapter 2). 1. samczsun, "Escaping the Dark Forest," 15 September 2020, samczsun.com. Source for the Lien Finance rescue through a private SparkPool submission path (Chapter 2). 1. Contemporary reporting on MakerDAO liquidations of 12 March 2020, including coverage republished by Nasdaq, "Mempool Manipulation Enabled Theft of $8M in MakerDAO Collateral on Black Thursday: Report" (2020). Source for the approximate value of zero-bid auction wins (Chapter 2). 1. Jason Milionis, Ciamac C. Moallemi, Tim Roughgarden and Anthony Lee Zhang, "Automated Market Making and Loss-Versus-Rebalancing," arXiv:2208.06046 (2022). Source for the LVR framework (Chapter 3). 1. Flashbots, "Flashbots Transparency Report — April 2021," writings.flashbots.net, published 13 May 2021. Source for MEV-Geth adoption among mining pools (Chapter 4). 1. Fei Wu, Thomas Thiery, Stefanos Leonardos and Carmine Ventre, "From Competition to Centralization: The Oligopoly in Ethereum Block Building Auctions," arXiv:2412.18074. Source for builder market shares (Chapters 5 and 6). 1. U.S. Department of the Treasury, Office of Foreign Assets Control, designation of Tornado Cash (8 August 2022) and its removal (March 2025); Van Loon v. Department of the Treasury, U.S. Court of Appeals for the Fifth Circuit (November 2024) (Chapter 5). 1. Reporting on United States v. Anton Peraire-Bueno and James Peraire-Bueno, U.S. District Court for the Southern District of New York, including Associated Press coverage of the mistrial declared on 7 November 2025 (Chapter 5). 1. Flashbots documentation, "MEV-Share Overview," docs.flashbots.net. Source for MEV-Share's hint and refund design (Chapter 6). 1. The Block, "Ethereum projects come together on MEV Blocker RPC launch to share profits with users," 5 April 2023 (Chapter 6). 1. Eric Budish, Peter Cramton and John Shim, "The High-Frequency Trading Arms Race: Frequent Batch Auctions as a Market Design Response," Quarterly Journal of Economics 130, no. 4 (2015): 1547–1621 (Chapter 7). 1. Flashbots, "Introducing BuilderNet," collective.flashbots.net, 26 November 2024 (Chapter 7). 1. Cointelegraph Magazine, "Applied MEV protection via Shutter's threshold encryption." Source for the Gnosis Chain deployment's trust model and latency (Chapter 7). 1. Ethereum Foundation, "Glamsterdam Testnet Announcement," blog.ethereum.org, 17 September 2026; EIP-7732 (Enshrined Proposer-Builder Separation); EIP-7805 (Fork-Choice Enforced Inclusion Lists) (Chapter 8). 1. Offchain Labs, "Timeboost for Arbitrum chains," docs.arbitrum.io; The Block, "Arbitrum transaction ordering system Timeboost generates $2 million in fees within three months of launch," 4 July 2025 (Chapter 8). 1. The Block, "Solana client Jito Labs axes mempool function following increase in MEV attacks," March 2024 (Chapter 8). Further Reading Philip Daian et al., "Flash Boys 2.0" (IEEE Symposium on Security and Privacy, 2020). The founding paper. Still the clearest account of why transaction ordering on a public chain creates both a market and a consensus risk. Dan Robinson and Georgios Konstantopoulos, "Ethereum is a Dark Forest" (Paradigm, 2020), and samczsun, "Escaping the Dark Forest" (2020). Two short essays that, read together, show the problem and the first private-channel response through concrete episodes. Jason Milionis, Ciamac Moallemi, Tim Roughgarden and Anthony Lee Zhang, "Automated Market Making and Loss-Versus-Rebalancing" (2022). The essential paper for understanding the cost MEV imposes on liquidity providers, written with unusual clarity for a technical work. Eric Budish, Peter Cramton and John Shim, "The High-Frequency Trading Arms Race" (Quarterly Journal of Economics, 2015). Written about stock markets, not blockchains, but the argument for batch auctions maps directly onto MEV. Michael Lewis, Flash Boys: A Wall Street Revolt (W. W. Norton, 2014).** A narrative account of latency arbitrage in equity markets that supplied MEV's founding paper with its title and much of its framing. Fei Wu, Thomas Thiery, Stefanos Leonardos and Carmine Ventre, "From Competition to Centralization: The Oligopoly in Ethereum Block Building Auctions." A rigorous empirical study of why the builder market concentrated. Flashbots writings and documentation (writings.flashbots.net; docs.flashbots.net). Primary sources on MEV-Geth, MEV-Boost, MEV-Share and BuilderNet, including the organization's own account of its reasoning and its transparency reports. Ethereum research forum (ethresear.ch) and the EIP repository. The working venue for proposals on enshrined PBS, inclusion lists and MEV burn; technical, but the best place to follow the protocol debate as it develops. Hashtags: #MaximalExtractableValue #MEV #MinerExtractableValue #DecentralizedFinance #DeFi #Ethereum #BlockchainTechnology #FrontRunning #SandwichAttacks #BackRunning #Flashbots #MEVBoost #ProposerBuilderSeparation #BlockBuilding #TransactionOrdering #Mempool #PriorityGasAuctions #MEVSearchers #ArbitrageTrading #DeFiLiquidations #AutomatedMarketMakers #FlashLoans #SlippageProtection #PrivateTransactions #OrderFlowAuctions #MEVProtection #BlockBuilderCentralization #EncryptedMempools #EthereumValidators #BlockchainSecurity

Download the Book (PDF): Introduction On March 29, 1960, the New York Times ran a full-page advertisement under the headline "Heed Their Rising Voices." It had been paid for by a committee supporting Martin Luther King Jr., and it described, in general terms, a wave of repression against civil rights protesters in the South. Some of its details were wrong. The police had not "ringed" the Alabama State College campus in Montgomery; the students had sung a different song from the one the advertisement named; King had been arrested four times, not seven. None of the statements named L. B. Sullivan, one of Montgomery's three elected commissioners, whose responsibilities included the police. He sued anyway, arguing that criticism of the police was criticism of him. An Alabama jury awarded him $500,000, the full amount he had asked for, and the Alabama Supreme Court affirmed. The verdict was not an aberration. It was an accurate application of the law of libel as it then stood in Alabama and, in its essentials, across the United States. That law presumed a defamatory statement false, presumed that it had caused harm, and imposed liability without asking whether the publisher had been careless. Other officials had filed similar claims against the Times and against CBS, and the total sought ran into millions of dollars. The point was not really compensation. It was to make national news organizations think twice before sending reporters south. When the Supreme Court reversed in New York Times Co. v. Sullivan in 1964, it did more than rescue a newspaper. It declared that the First Amendment places limits on what a state may do to protect reputation, and it announced a rule that has shaped public discourse in the United States ever since: a public official cannot recover damages for a defamatory falsehood about official conduct without proving, with convincing clarity, that the statement was made "with knowledge that it was false or with reckless disregard of whether it was false or not." That standard, which the Court called "actual malice," has nothing to do with spite. It concerns the publisher's state of mind about the truth. The argument of this book This booklet argues that the legal protection of journalism and public debate in the United States is not a single rule but a system built in layers, and that each layer exists because the one before it proved insufficient on its own. The first layer is substantive. Sullivan and the cases that followed it changed what a plaintiff has to prove, and for whom. Public officials and public figures must show actual malice; private individuals must at least show fault; nobody can recover for a statement of pure opinion or a parody no reasonable reader would take as fact. Yet a demanding standard of proof does not stop anyone from filing a lawsuit. A wealthy plaintiff who expects to lose on the merits can still impose years of discovery and hundreds of thousands of dollars in legal fees on a critic, and the critic may decide that silence is cheaper. The second layer, anti-SLAPP legislation, attacks that problem through procedure: an early motion to dismiss, a pause on expensive discovery, and an award of attorneys' fees against the plaintiff who brought a meritless claim. Such statutes now exist in well over thirty states, and a uniform model act has spread rapidly since 2021. A third layer concerns borders. For decades, claimants who could not win in the United States could sue in England, where the law was far friendlier to them, and then try to enforce the judgment here. The practice acquired a name, libel tourism, and a federal statute, the SPEECH Act of 2010, now bars American courts from enforcing foreign defamation judgments that would not have survived First Amendment scrutiny. Other countries have since moved in the same direction, most notably the United Kingdom with its Defamation Act 2013 and the European Union with its 2024 anti-SLAPP directive. The fourth layer protects the raw material of journalism: sources. Reporting on government and corporate misconduct often depends on people who will speak only if their identities are protected. The Supreme Court refused in 1972 to recognize a constitutional privilege for reporters before grand juries, and the protection that exists today is a patchwork of state shield laws, qualified privileges fashioned by lower federal courts, a federal statute limiting newsroom searches, and Justice Department policies that change with administrations. A related question, increasingly important, is when courts will unmask anonymous speakers online. Seen together, these layers reveal the system's real weak points. The substantive rule is strong but under open attack from members of the Supreme Court and from powerful litigants. The procedural layer is uneven, varying from state to state and, after a unanimous Supreme Court decision in January 2026, facing serious doubt about whether it applies in federal court at all. The cross-border layer protects American defendants from foreign judgments but not from the cost of defending foreign suits. And source protection at the federal level rests largely on executive discretion, which the Justice Department demonstrated in April 2025 when it rescinded the restrictive media-subpoena rules adopted three years earlier. What the chapters do The book proceeds in the order the layers were built. Chapter 1 describes the common law of libel as it stood before 1964 and explains why it threatened public debate, ending with Sullivan itself. Chapter 2 explains what actual malice means in practice, how plaintiffs try to prove it, and why a small number of recent cases, such as Dominion Voting Systems' claims against Fox News, have succeeded. Chapter 3 turns to the question on which most defamation cases actually turn: who counts as a public figure, and what private individuals must prove. Chapter 4 examines the growing challenge to Sullivan from within the Court, from state legislatures, and from high-profile suits against news organizations. Chapter 5 dissects the strategic lawsuit against public participation, or SLAPP, and explains why a strong substantive rule does not deter it. Chapter 6 examines the anti-SLAPP statutes that answer it, from California's pioneering law to the Uniform Public Expression Protection Act, and the unresolved question of their force in federal court. Chapter 7 crosses borders, tracing libel tourism from London's heyday as the world's defamation capital through the SPEECH Act and on to the European and British reforms of the last few years. Chapter 8 addresses the reporter's privilege and the protection of confidential sources. Chapter 9 takes up anonymity online: the tests courts use before unmasking an anonymous critic, and the role of platform immunity. The conclusion draws out what follows for journalists, publishers, lawmakers and readers. The book focuses on the law of the United States, with comparative notes where other systems illuminate the American one or where, as with libel tourism and cross-border SLAPPs, the problem is international by nature. It describes the law as it stands in late 2026. It is written to inform, not to advise; anyone facing a claim or a subpoena should consult a lawyer. A note on stakes It is easy to treat defamation law as a contest between wealthy plaintiffs and large media companies, in which the public has no stake. That picture is badly incomplete. Most defamation defendants are not the New York Times. They are local newspapers operating on thin margins, freelance reporters, academics, advocacy groups, bloggers, people who post reviews of a contractor or a doctor, parents who complain at a school board meeting, and survivors who describe abuse. For them, the difference between a claim dismissed in sixty days with fees awarded and a claim that grinds through three years of discovery is often the difference between speaking and not speaking. Reputation matters too. Defamation law exists because false statements can destroy careers, businesses and lives, and the internet has made it easier than ever to spread them. The layers of protection described in this book are not designed to make lying free. The best of them are designed to separate, early and cheaply, the claims that deserve a trial from the claims whose real purpose is to punish speech. Whether the law does that well is the question the following chapters try to answer. Chapter 1: The Law That Sullivan Replaced To understand why the Supreme Court rewrote American libel law in 1964, it helps to see how the law worked before, and how much of the work it did in the plaintiff's favor. The common law of defamation, inherited from England and developed by state courts for nearly two centuries, was designed around a simple premise: a person's good name is a form of property, and anyone who damages it with words should pay. That premise produced rules that were coherent, even elegant, in private disputes between neighbors. Applied to newspapers covering public controversies, they became something else. The elements of a common-law claim A plaintiff at common law had to establish surprisingly little. The first requirement was publication, which meant only that the statement had been communicated to someone other than the plaintiff. A single letter to a single employer was enough; a newspaper with a circulation of a million was the same claim multiplied. The second was defamatory meaning. A statement was defamatory if it tended to harm the plaintiff's reputation in the eyes of a substantial and respectable minority of the community, to expose the plaintiff to hatred, contempt or ridicule, or to cause others to shun or avoid the plaintiff. Courts read words in their natural sense but also recognized innuendo: a statement innocent on its face could be defamatory to readers who knew extra facts, and the plaintiff could plead those facts to supply the sting. The third was identification. The statement had to be "of and concerning" the plaintiff. A name was not required. It was enough that people who knew the plaintiff would reasonably understand the statement to refer to him or her. This element would turn out to be decisive in Sullivan. That was all. Three things the modern reader might expect a plaintiff to prove were missing entirely. The plaintiff did not have to prove the statement was false. Falsity was presumed. Truth, called justification, was a defense, and the burden of establishing it lay on the defendant. If the evidence was evenly balanced, or if the truth simply could not be established because the witnesses were dead, frightened or unwilling, the defendant lost. The plaintiff did not have to prove the defendant was at fault. Liability was strict. A newspaper that had checked every fact with care, relied on reputable sources and honestly believed what it printed was just as liable as one that invented the story from nothing. Care mattered, if at all, to punitive damages, not to liability. And the plaintiff did not have to prove harm. For libel, meaning defamation in written or other permanent form, most American jurisdictions presumed damages. A jury could award substantial sums on the assumption that a false statement in print must have injured the plaintiff's reputation, without evidence that anyone thought less of him. Slander, spoken defamation, generally required proof of special damages, meaning actual monetary loss, unless the words fell into one of the traditional per se categories: accusing someone of a serious crime, of having a loathsome disease, of misconduct in a trade or profession, or, historically, of unchastity in a woman. The combination was formidable. Once a plaintiff showed that a newspaper had published something damaging about him, the paper had to prove it true or fit the statement into a privilege, and a jury was free to award whatever it thought the insult was worth. The defenses and why they were not enough The common law did offer defenses beyond truth, and some were significant. Absolute privileges protected statements made in judicial proceedings, in legislative debate and by certain executive officials in the course of their duties. Qualified privileges protected statements made to protect a shared interest, such as an employer's reference to a prospective employer, and a fair and accurate report of official proceedings, which allowed newspapers to report what was said in court or at a council meeting even if the underlying statement was false. The defense most relevant to journalism was fair comment. It protected expressions of opinion on matters of public interest, including the conduct of public officials, the quality of public performances and books, and the management of institutions that invited public attention. Its limits were crucial. In most states, fair comment protected only opinion, not statements of fact, and the opinion had to rest on facts that were true or privileged. A critic who wrote that a mayor was corrupt because he had taken a bribe was protected only if the bribe had actually been taken. If the factual premise was wrong, the comment fell with it. A minority of states had gone further. In Coleman v. MacLennan, decided in 1908, the Kansas Supreme Court held that a newspaper discussing the character of a candidate for public office was privileged even as to false statements of fact, so long as it acted in good faith. The Kansas court reasoned that voters needed candid discussion of candidates more than candidates needed protection from honest mistakes. Several states followed this approach, but most did not, and the Supreme Court would later draw on Coleman as evidence that a more protective rule was workable. Retraction statutes, enacted in many states, limited damages if a publisher promptly corrected an error, but they varied widely and often did not apply to presumed or punitive damages. None of these defenses answered the basic problem that strict liability posed for journalism about public affairs. Reporting on government is carried out quickly, often under pressure, frequently on the basis of sources who cannot be named and documents that may not be admissible in court. Errors are inevitable. Under the common-law rules, every error was a potential judgment, and the size of that judgment was up to a jury that might share the plaintiff's politics. Libel outside the Constitution Until 1964 the First Amendment had almost nothing to say about any of this. The Supreme Court had long assumed that libel, like obscenity, belonged to a class of expression outside constitutional protection. In Chaplinsky v. New Hampshire in 1942, the Court listed "the libelous" among the "well-defined and narrowly limited classes of speech, the prevention and punishment of which have never been thought to raise any Constitutional problem." A decade later, in Beauharnais v. Illinois in 1952, it upheld an Illinois criminal law against group libel, reasoning that if libel of an individual could be punished, libel of a racial or religious group could be too. The assumption had a long pedigree but an uncomfortable history. The most notorious American libel law was the Sedition Act of 1798, which made it a crime to publish "false, scandalous and malicious" writing against the government, Congress or the President with intent to bring them into contempt. The Act did allow truth as a defense and required criminal intent, and Federalist prosecutors used it to jail Republican editors and at least one congressman, Matthew Lyon of Vermont. James Madison's Virginia Report of 1800 argued that the Act was unconstitutional precisely because free discussion of public men and measures was the essence of republican government. The Act expired in 1801, Jefferson pardoned those convicted, and Congress later repaid fines. The Supreme Court never ruled on its validity. The episode left an unresolved question: if seditious libel was incompatible with the First Amendment, what about civil libel suits brought by officials, which could achieve the same result with larger penalties and fewer procedural protections? Earlier still, the 1735 trial of John Peter Zenger in colonial New York had made the same point in a different register. Zenger, a printer, was prosecuted for seditious libel against the royal governor. His lawyer, Andrew Hamilton of Philadelphia, conceded that Zenger had printed the offending material but urged the jury to acquit because it was true. Under the law as it then stood, truth was no defense to seditious libel; the jury acquitted anyway. The case became a founding myth of American press freedom, though its legal effect was minimal. Alabama, 1960 The civil rights movement turned this abstract tension into an urgent practical problem. By 1960, national newspapers and television networks were covering the Southern resistance to desegregation, and the coverage was damaging to local officials. Libel suits offered those officials a weapon that did not look like censorship. A jury drawn from the community, applying the ordinary common law, could impose damages large enough to drive a publication out of the state. The advertisement at the center of Sullivan was typical of the material that could be targeted. It was not a news story but a fundraising appeal, prepared by an advertising agency for the Committee to Defend Martin Luther King and the Struggle for Freedom in the South, and published in the Times for a fee. Its account of events in Montgomery contained errors that the paper had not checked against its own news files. The paper's circulation in Alabama was small: the Supreme Court would note that about 394 copies of the relevant issue went to the state, of which some 35 went to Montgomery County. L. B. Sullivan, the Montgomery commissioner responsible for the police department, demanded a retraction. The Times responded with a letter asking how he thought the advertisement referred to him. He sued in state court, and also sued four Black ministers whose names appeared on the advertisement as endorsers: Ralph Abernathy, Fred Shuttlesworth, Joseph Lowery and S. S. Seay Sr. The ministers testified that they had not authorized the use of their names. They were held liable anyway, and property belonging to some of them was seized to satisfy the judgment. The trial judge instructed the jury that the statements were libelous per se, that falsity and malice were presumed, and that general damages need not be proved. The only real questions for the jury were whether the statements were "of and concerning" Sullivan and how much to award. The jury found for Sullivan and awarded $500,000. Around the same time, other Alabama officials, including Governor John Patterson, filed their own suits over the same advertisement, and further suits followed a Times news report by Harrison Salisbury on racial tension in Birmingham. The paper faced claims totaling several million dollars in Alabama courts at a time when it was not especially profitable. Similar suits were brought against CBS. The message to national news organizations was clear. The decision The Supreme Court reversed unanimously on March 9, 1964. Justice William Brennan's opinion for the Court began by rejecting Alabama's argument that the First Amendment did not apply because the case involved a private civil suit rather than a criminal prosecution. The test, Brennan wrote, was not the form in which state power was applied but whether that power had been exercised in a way that abridged freedom of speech and press. A civil libel judgment was state action. Nor did it matter that the statements appeared in a paid advertisement; the advertisement communicated information and sought support for a movement of great public importance. Brennan then placed the case against what he called "a profound national commitment to the principle that debate on public issues should be uninhibited, robust, and wide-open, and that it may well include vehement, caustic, and sometimes unpleasantly sharp attacks on government and public officials." He drew on Madison and the controversy over the Sedition Act, observing that although the Act had never been tested in the Supreme Court, the attack on its validity had "carried the day in the court of history." The central analytical move was this. Erroneous statements are inevitable in free debate, and they must be protected if freedom of expression is to have the "breathing space" it needs to survive, a phrase Brennan borrowed from NAACP v. Button, decided the previous year. A rule that compels critics of official conduct to guarantee the truth of their factual assertions, on pain of libel judgments, leads to self-censorship. Allowing truth as a defense does not cure the problem, because would-be critics may be deterred from voicing criticism they believe to be true, doubting whether they can prove it in court or fearing the expense of having to do so. Such a rule "dampens the vigor and limits the variety of public debate." The Court therefore announced a federal rule. A public official may not recover damages for a defamatory falsehood relating to official conduct unless he proves that the statement was made with actual malice, "that is, with knowledge that it was false or with reckless disregard of whether it was false or not." Actual malice must be shown with "convincing clarity," the standard later described as clear and convincing evidence. And because the constitutional rule would mean little if juries could apply it however they liked, the Court held that appellate courts must independently review the record to ensure that a judgment does not constitute a forbidden intrusion on free expression. Applying that review, the Court found the evidence constitutionally insufficient. The failure of the Times to check the advertisement against its own news files showed, at most, negligence, not knowledge of falsity or reckless disregard. And the evidence did not support a finding that the statements were "of and concerning" Sullivan. An impersonal attack on government operations, the Court reasoned, could not be transformed into a personal attack on the officials responsible for them. To hold otherwise would revive the doctrine of seditious libel under another name. Justices Hugo Black, William O. Douglas and Arthur Goldberg would have gone further. Black, joined by Douglas, argued that the First Amendment gave an absolute, unconditional right to criticize official conduct, and that the actual-malice rule was an inadequate safeguard because malice was an elusive concept that a hostile jury could find almost anywhere. Goldberg, also joined by Douglas, took a similar position. The majority chose the middle course. What changed, and what did not Sullivan altered the architecture of libel law in three ways. It made the plaintiff's burden constitutional, so that states could no longer adjust it downward. It shifted attention from the content of the statement to the defendant's state of mind about its truth. And it made judges, not juries, the ultimate guardians of that inquiry, through independent appellate review. It left a great deal unsettled. It applied only to public officials and only to statements about official conduct. It said nothing about private individuals, about public figures who held no office, or about statements on matters of public concern that did not involve officials at all. It did not define "reckless disregard," or explain how a plaintiff could prove what was in a reporter's mind. It did not address opinion, parody or satire. And it did nothing about the cost of litigation itself. A defendant who would eventually win under Sullivan might still have to pay for years of discovery and trial before winning. The next half-century of doctrine filled in the first set of gaps, and the next two chapters trace how. The last gap, the cost of winning, would require a different kind of solution altogether. Chapter 2: Actual Malice: What It Means and How It Is Proved The phrase "actual malice" is one of the most misleading terms of art in American law. To a lay reader it suggests hatred, spite or an intention to harm. In defamation law it means none of these things. A reporter who despises the subject of a story, and writes it hoping to ruin him, has not acted with actual malice if she believed every word she wrote. A reporter who likes the subject and wishes him well has acted with actual malice if she published something she knew to be false. The standard concerns the publisher's belief about truth, and nothing else. The Supreme Court has regretted the label. In Masson v. New Yorker Magazine in 1991, it observed that the term was confusing and suggested that trial judges instruct juries in terms of knowledge of falsity and reckless disregard rather than using the phrase itself. The phrase survives anyway, and understanding its real content is the starting point for understanding how American defamation cases are won and lost. Knowledge and reckless disregard The first branch of the standard, knowledge of falsity, is straightforward in principle. A publisher who prints something it knows is untrue is liable. The difficulty is proof, since few defendants will admit to lying, and knowledge must almost always be inferred from circumstantial evidence. The second branch, reckless disregard, needed definition, and the Court supplied it in two cases decided within four years of Sullivan. In Garrison v. Louisiana, decided later in 1964, a New Orleans district attorney named Jim Garrison was criminally prosecuted for accusing local judges of laziness and of obstructing his investigations. The Court applied the Sullivan standard to criminal libel and explained that reckless disregard requires a "high degree of awareness of probable falsity." It also rejected the notion that ill will could substitute for this awareness. Even a speaker motivated by hatred is protected if he honestly believes what he says, because debate on public issues would be impoverished if only the dispassionate could engage in it. Then, in St. Amant v. Thompson in 1968, the Court gave the definition that remains the touchstone. A candidate for office in Louisiana had read on television a statement from a union member alleging that a deputy sheriff had taken bribes. The candidate had not verified the allegation and knew nothing of the deputy's reputation. The Louisiana courts found reckless disregard. The Supreme Court reversed, holding that reckless disregard is not measured by whether a reasonably prudent person would have published or would have investigated first. There must be "sufficient evidence to permit the conclusion that the defendant in fact entertained serious doubts as to the truth of his publication." This is a subjective test. It asks what the defendant actually thought, not what a careful journalist should have thought. That means failure to investigate, by itself, does not establish actual malice. A reporter who relies on a single source without checking has been negligent, perhaps grossly so, but she has not acted with reckless disregard unless she actually doubted what she was told. The Court immediately recognized the danger of making the test purely a matter of the defendant's say-so. A defendant cannot, the St. Amant opinion warned, automatically ensure a favorable verdict by testifying that he believed the statement to be true. A jury may disbelieve him. The Court listed situations in which professions of good faith are unlikely to prove persuasive: where the story is fabricated by the defendant, is the product of his imagination, or is based wholly on an unverified anonymous telephone call; where the allegations are so inherently improbable that only a reckless person would have put them in circulation; or where there are obvious reasons to doubt the veracity of the informant or the accuracy of his reports. That list has been the template for proving actual malice ever since. Plaintiffs rarely have a confession. They build a circumstantial case that the defendant must have doubted what it published, and ask the jury to reject the defendant's denials. Circumstantial proof: the cases that succeeded and failed The contrast between two cases decided together in 1967 shows what that circumstantial case looks like. In Curtis Publishing Co. v. Butts, the Saturday Evening Post published a story claiming that Wally Butts, the athletic director at the University of Georgia, had conspired to fix a football game by giving Alabama's coach his team's plays. The story rested on the account of an insurance salesman who said he had accidentally been connected into a telephone call between the two men and had taken notes. The source was on probation for writing bad checks. The magazine did not examine his notes, did not interview a person said to have been with him, did not check game films with an expert to see whether the supposed information had been used, and was not under any real time pressure. Butts won, and the Supreme Court affirmed his judgment. In the companion case, Associated Press v. Walker, a wire service reported that former general Edwin Walker had led a violent charge against federal marshals during the rioting that accompanied James Meredith's enrollment at the University of Mississippi in 1962. The report came from a correspondent on the scene, filed in the middle of a chaotic night. The Court reversed Walker's judgment. The story was hot news, the correspondent was apparently trustworthy, and nothing in the circumstances should have alerted the wire service to a need for further checking. The factors that separated those cases recur in later decisions: the reliability of the source and what the publisher knew about it, the time available for verification, whether the publisher had information in its own possession that contradicted the story, and whether it avoided obvious steps to confirm or refute it. The last factor received sharp treatment in Harte-Hanks Communications, Inc. v. Connaughton in 1989. An Ohio newspaper supporting an incumbent judge published a story, days before an election, reporting a grand-jury witness's claim that the challenger, Daniel Connaughton, had used dirty tricks and offered her and her sister jobs and a trip in exchange for their help. The paper had interviewed Connaughton and five other witnesses, all of whom denied the story or failed to confirm it, but it never interviewed the sister, the one person who could have corroborated or refuted the central claim, and it did not listen to tapes Connaughton had made available. The Supreme Court upheld a verdict for Connaughton, holding that although failure to investigate is not itself enough, "the purposeful avoidance of the truth is in a different category." A publisher who deliberately decides not to acquire knowledge of facts that might confirm the probable falsity of its story can be found to have acted with reckless disregard. The Court has also made clear that the inquiry into the defendant's state of mind can reach deep into the newsroom. In Herbert v. Lando in 1979, a retired Army officer sued CBS over a 60 Minutes segment that questioned his claims about reporting war crimes in Vietnam. He sought to depose the producer about his thought processes, conversations with colleagues and conclusions about sources. CBS argued that the First Amendment created a privilege protecting the editorial process from such inquiry. The Court disagreed. Since Sullivan made the defendant's state of mind the central issue, a plaintiff must be allowed to seek direct evidence of it. The decision is one reason defamation litigation against media defendants is so expensive: the discovery that actual malice requires is exhaustive by nature. The supporting rules Several later decisions reinforced the Sullivan framework. In Bose Corp. v. Consumers Union in 1984, the Court confirmed that appellate judges must exercise independent review of the whole record on the question of actual malice rather than deferring to a trial court's findings. Consumer Reports had written that sound from a loudspeaker system tended to wander "about the room," when the engineer who tested it had heard it wander along the wall. The trial judge found that the engineer's choice of words, and his refusal to admit his mistake, showed actual malice. The Supreme Court held that an imprecise description of an actual perception, adopted in an effort to convey a technical matter to general readers, showed at most a failure to choose the right words and could not sustain a finding of knowing or reckless falsity. In Anderson v. Liberty Lobby, Inc. in 1986, the Court held that when a trial judge rules on a motion for summary judgment in an actual-malice case, the judge must ask whether a reasonable jury could find actual malice by clear and convincing evidence, not merely by a preponderance. The heightened standard therefore operates before trial, not just at it, and it gives media defendants a realistic chance of disposing of weak cases without a jury. In Philadelphia Newspapers, Inc. v. Hepps, also in 1986, the Court reversed the common-law presumption of falsity for speech of public concern. A plaintiff suing over such speech must prove the statement false, even if the plaintiff is a private individual. The Court acknowledged that this would sometimes leave false speech unremedied when the truth could not be established either way, and accepted that cost as necessary to avoid chilling true speech. In Masson v. New Yorker Magazine, the Court addressed a problem specific to journalism: quotations. The psychoanalyst Jeffrey Masson sued the writer Janet Malcolm over a long profile in which she attributed to him statements, including a description of himself as an "intellectual gigolo," that did not appear in that form on her tape recordings. The Court held that a deliberate alteration of the words a speaker used does not equate with knowledge of falsity unless it results in a material change in the meaning conveyed by the statement. Quotation marks signal that the words are the speaker's own, so an invented or altered quotation can be a falsehood in itself, but small changes in grammar or syntax, or edits that preserve the substance, are not actionable. Two further decisions limited the reach of defamation law into satire and opinion. In Hustler Magazine, Inc. v. Falwell in 1988, the Court unanimously held that the evangelist Jerry Falwell could not recover for intentional infliction of emotional distress caused by a crude parody advertisement, because the parody could not reasonably be understood as describing actual facts. Public figures cannot evade the actual-malice rule by relabeling a defamation claim as something else. In Milkovich v. Lorain Journal Co. in 1990, by contrast, the Court declined to create a wholesale exemption for anything labeled opinion. A sports columnist had written that a high school wrestling coach had lied under oath at a hearing about a brawl at a meet. The Court held that the statement implied a provably false assertion of fact, and prefacing it with "in my opinion" would not have changed that. What remains protected is language that cannot reasonably be interpreted as stating actual facts, including loose, figurative or hyperbolic speech, and statements on matters of public concern that are not provably false. Actual malice in the modern courtroom For decades, media lawyers told clients that actual malice was nearly impossible to prove. The statistics supported them: most public-figure plaintiffs lost, many on summary judgment. The last several years have complicated that picture, and the cases that have succeeded show what it takes. The most important is Dominion Voting Systems' suit against Fox News in Delaware Superior Court over broadcasts in the weeks after the 2020 presidential election that repeated claims that Dominion's machines had been rigged to change votes. What made the case unusual was the evidence obtained in discovery. Internal messages showed that hosts, producers and executives had privately expressed disbelief in the fraud claims being aired. In March 2023, Judge Eric Davis granted Dominion partial summary judgment on falsity, ruling that the evidence showed the statements about Dominion were false, leaving actual malice and damages for trial. On the eve of trial in April 2023, Fox settled for $787.5 million, the largest publicly disclosed defamation settlement by an American media company. Newsmax settled Dominion's parallel claim in August 2025 for $67 million. Smartmatic, another voting-technology company, pursued its own multibillion-dollar claim against Fox in New York state court, where it remained pending in 2026. A second example is the suit by Ruby Freeman and Shaye Moss, two Georgia election workers whom Rudolph Giuliani repeatedly accused of manipulating ballots. The federal district court in Washington entered a default judgment on liability as a sanction for Giuliani's failure to preserve and produce evidence, and in December 2023 a jury awarded the two women about $148 million in damages. After a long enforcement battle, the parties settled in January 2025. These cases had features that most defamation cases lack. The falsity of the statements was established by overwhelming public evidence, including recounts, audits and court rulings. The defendants had repeated the claims many times, after being told directly that they were false. And discovery produced documents that let a factfinder infer that the speakers did not believe what they were saying. The cases show that actual malice is a demanding standard, not an impossible one. It is designed to protect honest error, and it does not protect deliberate falsehood. The contrasting case is Sarah Palin's suit against the New York Times. In June 2017, after a gunman shot a Republican congressman and others at a baseball practice, the Times published an editorial that wrongly suggested a direct link between a map circulated by Palin's political action committee and the 2011 shooting of Representative Gabrielle Giffords. The paper corrected the error the next morning. Palin sued, contending that the editorial page editor, who had inserted the language, knew or recklessly disregarded that there was no established link. A jury found for the Times in 2022. The United States Court of Appeals for the Second Circuit ordered a new trial in 2024, citing errors in the first trial, and in April 2025 a second jury again found for the newspaper. Palin's case had a false statement, a prominent publisher and an editor who had not checked the paper's own earlier reporting. What it lacked was persuasive evidence that anyone had doubted the statement's truth when it was published, and a prompt correction pointed the other way. What the standard protects, and what it costs The actual-malice standard reflects a judgment that the risk of error in public debate should fall, within limits, on those who choose to enter it rather than on those who report on them. It tolerates some false statements in exchange for fewer suppressed true ones. The cases above show that the line it draws is intelligible: a publisher that honestly gets something wrong is protected; one that knowingly broadcasts falsehoods, or deliberately shields itself from the truth, is not. The standard has two costs that shape much of the rest of this book. The first is that it depends on who counts as a public official or public figure, a question the Court answered only in stages, and which Chapter 3 examines. The second is that proving or disproving a defendant's state of mind requires the most intrusive discovery the civil system offers: depositions of reporters and editors, production of drafts, emails and text messages, and exploration of editorial deliberations. A defendant confident of winning under Sullivan may still spend years and large sums proving it. That cost is the lever that strategic litigants use, and it is why the substantive protection of Sullivan needed a procedural complement. Chapter 3: Public Figures, Private People and the Gertz Settlement In practice, a great many defamation cases are decided not by asking what the defendant knew but by asking who the plaintiff is. If the plaintiff is a public official or public figure, the actual-malice requirement applies and the case is usually hard to win. If the plaintiff is a private person, the burden drops, sometimes dramatically. The classification question is therefore often the most important ruling in the case, and it is frequently litigated before anything else. The rules that govern it were settled, more or less, in a single decision in 1974, Gertz v. Robert Welch, Inc. But Gertz did not end the debate. It channelled it into a set of categories whose edges remain contested, and the internet has made those edges more contested still. From officials to figures Sullivan concerned a public official, and the Court soon had to say who counted. In Rosenblatt v. Baer in 1966, it held that the designation applies at the very least to those among the hierarchy of government employees who have, or appear to the public to have, substantial responsibility for or control over the conduct of governmental affairs. The supervisor of a county-owned ski area qualified. Lower courts have applied the category to elected officials at every level, to police officers in most jurisdictions, and to many appointed administrators, while dividing over teachers and low-ranking employees. In Monitor Patriot Co. v. Roy in 1971, the Court added that for candidates for office, anything that might touch on fitness for office is relevant, which effectively extends the rule to nearly everything said about a candidate. The next step came in 1967, in the Butts and Walker cases discussed in Chapter 2. Neither Wally Butts, a university athletic director employed by a private athletic association, nor Edwin Walker, a retired general, held public office. A majority of the Court nonetheless agreed that people who are intimately involved in the resolution of important public questions, or who by reason of their fame shape events in areas of concern to society at large, should be subject to a comparable rule. Chief Justice Earl Warren's concurrence explained why: in an increasingly complex society, power was exercised by private individuals and institutions as much as by government, and the public had as great an interest in debating their conduct. In 1971 a plurality of the Court tried to go further. In Rosenbloom v. Metromedia, Inc., a distributor of nudist magazines had been arrested, and a radio station described him in its reports as a smut peddler. The charges were later dismissed. Justice Brennan's plurality opinion reasoned that the constitutional rule should depend on the subject matter, not the plaintiff. If a statement concerned a matter of public or general interest, the actual-malice rule should apply regardless of whether the person described was famous. Rosenbloom, a private businessman caught up in a public event, would have to prove actual malice. The plurality commanded only three votes, and three years later the Court abandoned it. Gertz and its bargain Elmer Gertz was a well-known Chicago lawyer. In 1968 a Chicago police officer named Richard Nuccio shot and killed a young man named Ronald Nelson and was later convicted of murder. Gertz represented Nelson's family in a civil suit against the officer. American Opinion, a monthly magazine published by the John Birch Society, printed an article portraying the prosecution of Nuccio as part of a Communist campaign against the police and describing Gertz as a "Leninist" and a "Communist-fronter." The accusations were false. The magazine's managing editor had made no effort to verify them. Justice Lewis Powell's opinion for the Court in Gertz v. Robert Welch, Inc. rejected the Rosenbloom approach and put the plaintiff back at the center of the analysis. Its reasoning rested on two differences between public and private people. The first was access to the channels of communication. Public officials and public figures usually have a realistic opportunity to counteract false statements through their own speech. They can call a press conference, give an interview, post a rebuttal. Private individuals generally cannot, and so are more vulnerable to injury. The second, which Powell described as more important, was assumption of risk. A person who seeks public office, or who thrusts himself into a public controversy, voluntarily exposes himself to increased risk of injury from defamatory falsehood. A private individual has done nothing of the kind. The state's interest in protecting that person's reputation is therefore stronger. From these premises, Gertz drew three conclusions that still structure the law. First, a public figure can be either of two kinds. Some people achieve such pervasive fame or notoriety that they become public figures for all purposes and in all contexts. More commonly, a person voluntarily injects himself, or is drawn, into a particular public controversy and thereby becomes a public figure for a limited range of issues. The Court also noted the theoretical possibility of becoming a public figure through no purposeful action of one's own, but said such involuntary public figures must be exceedingly rare. Second, for private individuals, the states may define the appropriate standard of liability for themselves, so long as they do not impose liability without fault. Strict liability for defamation of private people, the core of the common law, was therefore abolished for statements of public concern. Most states responded by adopting a negligence standard: the plaintiff must show that the defendant failed to act as a reasonably careful publisher would have. A few chose more protective rules. New York, in Chapadeau v. Utica Observer-Dispatch in 1975, required a private plaintiff suing over a matter of legitimate public concern to show that the publisher acted in a grossly irresponsible manner, without due consideration for the standards of information gathering and dissemination ordinarily followed by responsible parties. Colorado and Indiana, among others, applied actual malice to private plaintiffs on matters of public concern. Third, the Court limited damages. A private plaintiff who proves only negligence may recover only for actual injury, which the Court defined broadly to include impairment of reputation and standing in the community, personal humiliation, and mental anguish and suffering, but which must be supported by competent evidence. Presumed and punitive damages remain available only on a showing of actual malice. This was a significant reform, since presumed and punitive damages had been the source of the largest common-law verdicts. Gertz himself was held to be a private figure. Although he had been active in civic and professional affairs and had published books and articles, he had not achieved general fame in the community, and he had played no part in the criminal prosecution of the officer and had not discussed the case with the press. His role as counsel in a civil case did not make him a public figure for the controversy surrounding it. After a new trial, he ultimately recovered a substantial verdict. The limits of the categories The Court spent the rest of the 1970s policing the line it had drawn, and consistently drew it narrowly. In Time, Inc. v. Firestone in 1976, Time magazine reported that Russell Firestone, heir to the tire fortune, had been granted a divorce on grounds of extreme cruelty and adultery. The divorce decree did not actually rest on adultery. Mary Alice Firestone, his wife, was a prominent member of Palm Beach society and had held press conferences during the divorce proceedings. The Court held that she was nonetheless a private figure. A divorce, even a sensational one among the wealthy, was not the sort of public controversy referred to in Gertz, and resorting to the courts to dissolve a marriage was not a voluntary entry into public debate. In Hutchinson v. Proxmire in 1979, Senator William Proxmire gave his "Golden Fleece of the Month" award, a mock prize for wasteful government spending, to federal agencies that had funded a scientist's research on aggression in monkeys. The scientist, Ronald Hutchinson, sued. The Court held that he was not a public figure. Receiving public grants and publishing in professional journals did not make him one, and his access to the media was limited to responding to the award itself. The Court added an important principle: those charged with defamation cannot, by their own conduct, create their own defense by making the claimant a public figure. The controversy had to exist before the defamation, not be created by it. In Wolston v. Reader's Digest Association in the same year, the plaintiff had failed to appear before a grand jury investigating Soviet espionage in 1958, had been held in contempt, and had received brief press coverage. Sixteen years later, a book published by Reader's Digest listed him among Soviet agents identified in the United States. The Court held that he had not voluntarily thrust himself into the controversy over espionage; he had been dragged into it. It declined to decide whether someone who was once a public figure remains one forever, but its reasoning suggested that the passage of time matters. Six years later, the Court added a final element. In Dun & Bradstreet, Inc. v. Greenmoss Builders, Inc. in 1985, a credit-reporting agency had sent five subscribers a report falsely stating that a construction company had filed for bankruptcy; a teenage employee had confused the company with a former employee who had filed. The plurality held that when defamatory statements involve no issue of public concern, the reasons for the Gertz limitations are weaker, and a state may permit presumed and punitive damages without a showing of actual malice. A confidential credit report sent to a handful of subscribers, concerning a purely private business matter, was not speech of public concern. The resulting framework has five main tiers, and the rules differ at each of them, as Table 1 summarizes. Table 1. The constitutional framework for defamation plaintiffs after Gertz. Plaintiff and subject Leading case Fault needed for liability Presumed or punitive damages Burden on falsity Public official, official conduct New York Times Co. v. Sullivan (1964) Actual malice Actual malice Plaintiff All-purpose public figure Curtis Publishing Co. v. Butts (1967); Gertz (1974) Actual malice Actual malice Plaintiff Limited-purpose public figure, within the controversy Gertz (1974) Actual malice Actual malice Plaintiff Private figure, matter of public concern Gertz (1974); Hepps (1986) At least negligence, set by state Actual malice Plaintiff Private figure, purely private concern Dun & Bradstreet (1985) Set by state Available without actual malice Varies by state How lower courts decide The Supreme Court has not decided a major public-figure case since the 1980s, and lower courts have developed their own tests. The most influential for limited-purpose public figures was set out by the United States Court of Appeals for the District of Columbia Circuit in Waldbaum v. Fairchild Publications, Inc. in 1980. A court first asks whether there was a public controversy, meaning a real dispute whose outcome affects the general public or some segment of it in an appreciable way, not merely a matter that attracted public attention. It then asks whether the plaintiff played a significant role in that controversy, typically by trying to influence its outcome. Finally, it asks whether the alleged defamation was germane to the plaintiff's participation in the controversy. Applied in practice, these tests produce outcomes that are defensible but hard to predict. Corporations that advertise heavily or that aggressively promote a product may be treated as limited-purpose public figures for disputes about that product. Business executives who give interviews, activists who organize campaigns, authors who publish provocative books, and professionals who seek media attention are regularly classified as limited-purpose public figures. People whose only connection to a public matter is that something happened to them usually remain private. The rare involuntary public figure also appears occasionally. In Dameron v. Washington Magazine, Inc. in 1985, the D.C. Circuit held that an air traffic controller on duty at the time of a fatal airliner crash, who was later described as partly responsible for it, had become an involuntary public figure for the limited purpose of discussion of that crash. He had done nothing to seek attention, but he had become a central figure in a public controversy through the circumstances of his job. Public figures in a networked world The Gertz rationale was built for a media environment in which access to mass communication was scarce and unequally distributed. The famous had it; ordinary people did not. That premise has eroded. A private person with a social-media account can reach a large audience; a person who posts a video that is shared millions of times may acquire fame of a kind Gertz never imagined, often involuntarily and for a few days. At the same time, the capacity to respond to falsehood is not the same as the capacity to be heard over it, and a viral accusation can reach far more people than any rebuttal. Courts have not developed a new doctrine for this environment. They continue to apply Gertz and Waldbaum, asking whether the plaintiff voluntarily entered a public controversy and whether he or she has realistic access to the channels of rebuttal. A few recurring patterns are visible. People who build large online followings by commenting on public issues, including influencers, podcasters and self-described citizen journalists, are frequently held to be public figures at least for the topics they address. People who become famous only because of a single incident caught on camera are more often treated as private figures, or as limited-purpose public figures only for the incident itself. The litigation arising from a January 2019 encounter on the National Mall between a high school student, Nicholas Sandmann, and a Native American activist illustrates both the stakes and the uncertainty. Video of the encounter went viral, and national outlets published reports that Sandmann and his lawyers contended falsely portrayed him as the aggressor. He sued several news organizations. CNN and the Washington Post settled in 2020 for undisclosed amounts. Suits against other outlets were dismissed, and in 2023 the United States Court of Appeals for the Sixth Circuit affirmed the dismissal of several remaining claims on the ground that the statements at issue were protected opinion rather than assertions of fact. The case never produced a definitive ruling on whether a teenager who becomes the face of a viral controversy is a public figure, and that question remains open in most jurisdictions. The classification question has also acquired a political dimension. The public-figure doctrine is one of the reasons critics of Sullivan argue that the rule has drifted far from its origins. Sullivan protected criticism of a government official exercising state power. Critics note that the public-figure extension now requires ordinary people who become briefly prominent, and businesses and professionals caught up in local controversies, to satisfy the same demanding standard. Defenders respond that Gertz already solved this problem by keeping private individuals out of the actual-malice regime, and that the fault lines lie in how lower courts apply the categories, not in the categories themselves. That debate is part of the broader challenge to Sullivan taken up in the next chapter. Chapter 4: Sullivan Under Pressure For most of its history, New York Times Co. v. Sullivan was treated as settled law, even by those who disliked its consequences. It had been decided unanimously, reaffirmed repeatedly and extended by justices of every philosophical stripe. The debate was about its reach, not its existence. That changed in 2019. Since then, two sitting justices have called openly for reconsideration, prominent litigants have petitioned the Court to overrule it, a state legislature has considered bills designed to provoke a test case, and a sitting President has brought a series of large defamation suits against major news organizations. The Court has so far declined every invitation. But the pressure has not eased, and the arguments deserve to be understood on their merits. The case against Sullivan from the bench The first sustained judicial critique came from Justice Clarence Thomas in February 2019, in an opinion concurring in the denial of review in McKee v. Cosby. Kathrine McKee had publicly accused the entertainer Bill Cosby of raping her decades earlier. Cosby's lawyer wrote a letter attacking her credibility, and she sued for defamation. The lower courts held that by speaking publicly about her accusation she had become a limited-purpose public figure and dismissed her claim for failure to show actual malice. The Supreme Court declined to hear her appeal. Thomas agreed that the case did not warrant review but wrote separately to question the entire doctrine. Sullivan and the decisions extending it, he argued, were "policy-driven decisions masquerading as constitutional law." The First Amendment as understood when it was ratified in 1791, and the Fourteenth Amendment as understood in 1868, did not displace the common law of libel, which had protected public figures at least as strongly as private individuals and in some respects more so. If the states wished to adopt a more speech-protective rule, they were free to do so, but the Constitution did not require it. He urged the Court to reconsider its precedents in an appropriate case. In July 2021, when the Court declined to review Berisha v. Lawson, Thomas renewed the call, and Justice Neil Gorsuch joined the critique in a separate dissent from the denial. Shkelzen Berisha, the son of a former Albanian prime minister, had sued over a book that he said falsely linked him to an arms-trafficking scheme. The lower courts found him a limited-purpose public figure who could not prove actual malice. Gorsuch's dissent did not rest mainly on original meaning. It argued that the factual premises underlying Sullivan had changed. In 1964 a few major publishers with professional standards and fact-checking departments dominated the news. Today, he observed, anyone can publish anything to a global audience at almost no cost, and the actual-malice standard may create an incentive to avoid investigation, since a publisher who never checks can never be shown to have doubted. He also questioned how far the public-figure category had expanded, noting that it could now encompass people who had done little more than attract momentary attention. The rule, he suggested, may now leave many people who have been falsely maligned without meaningful redress. Thomas made a similar argument in June 2022, dissenting from the denial of review in Coral Ridge Ministries Media, Inc. v. Southern Poverty Law Center, in which a Christian ministry sued over its designation as a hate group. He wrote that the actual-malice standard had allowed the media to cast false aspersions on public figures with near impunity. No other justice has joined these opinions. The clearest test came in March 2025, when the Court declined to hear an appeal by the casino developer Steve Wynn. Wynn had sued the Associated Press over a 2018 article reporting on two police complaints, made decades earlier, that accused him of sexual misconduct. The Nevada courts dismissed his claim under that state's anti-SLAPP statute because he could not show actual malice. His petition to the Supreme Court asked it directly to overrule Sullivan. The Court denied review. A refusal to hear a case is not a ruling on the merits, and a different case with a more sympathetic plaintiff might fare differently. But the denial made clear that there were not four votes, the number required to grant review, to reconsider the rule in a case squarely presenting the question. Pressure from outside the Court The judicial critique has encouraged efforts elsewhere to create the right test case. The most prominent were in Florida. A bill introduced in the Florida legislature in 2023 would have narrowed the definition of a public figure, created a presumption that statements based on anonymous sources were false, and made it easier for plaintiffs to establish actual malice. Its sponsors were explicit that one purpose was to prompt the Supreme Court to revisit Sullivan. The bill drew opposition from both conservative and progressive media organizations and died before the end of the session. Later proposals along similar lines also failed to become law. Because Sullivan is a constitutional rule, a state statute purporting to weaken it would in any event have been vulnerable in the courts; the bills' value to their supporters lay in the litigation they might have generated. A second source of pressure has been litigation by political figures against news organizations, and above all by President Donald Trump. Trump has a long history of suing and threatening to sue over coverage, and before his second term his suits against media defendants generally failed. A suit over a 2005 biography that questioned the size of his fortune, Trump v. O'Brien, was dismissed by a New Jersey court, and the dismissal was affirmed on appeal in 2011. Suits he filed against CNN and the New York Times in the years after his first term were dismissed, and in the Times case a New York court ordered him to pay the newspaper's legal fees under the state's strengthened anti-SLAPP law. The pattern changed after the 2024 election. In December 2024, ABC News agreed to pay $15 million toward Trump's presidential foundation and library, plus $1 million in legal fees, to settle a defamation suit over statements by the anchor George Stephanopoulos that Trump had been found liable for rape in the civil case brought by the writer E. Jean Carroll. The jury in that case had found Trump liable for sexual abuse, not rape as defined under New York's penal law, although the trial judge had written that the jury's finding was consistent with rape as commonly understood. In July 2025, Paramount, the parent company of CBS, agreed to pay $16 million to settle a suit over the editing of a 60 Minutes interview with Vice President Kamala Harris; that suit was framed primarily as a claim under Texas consumer-protection law rather than defamation, and many legal commentators regarded it as weak on the merits. The settlement came while Paramount was seeking federal regulatory approval for a merger. Trump then filed further suits seeking unprecedented sums. In July 2025 he sued the Wall Street Journal and its owners over a report about a birthday letter to Jeffrey Epstein, seeking $10 billion; a federal judge in Florida dismissed the complaint in April 2026 while allowing him to refile. In September 2025 he sued the New York Times and Penguin Random House for $15 billion over articles and a book about his finances; the federal judge in the Middle District of Florida struck the initial complaint as improperly long and argumentative, and the case remained at the pleading stage in 2026. In December 2025 he sued the BBC for $10 billion over a Panorama documentary that spliced together parts of his speech of January 6, 2021, and the BBC moved to dismiss in March 2026. These suits face the actual-malice standard, and those that have been decided on the merits have generally failed. The settlements, however, show why the doctrinal protection of Sullivan is not the whole story. A media company that expects to win in court may still settle if it is part of a larger corporation with interests before government agencies, if the litigation would be long and distracting, or if the cost of defense approaches the cost of settlement. When the plaintiff is also the head of the executive branch, the calculus includes considerations that have nothing to do with the law of defamation. That is precisely the problem that Chapters 5 and 6 address. The arguments on the merits Stripped of their political context, the arguments for and against Sullivan raise real questions. The strongest arguments for reconsideration are three. The first is historical: the framers did not understand the First Amendment to displace the common law of libel, and the Court in 1964 did not seriously claim otherwise. Whatever one thinks of originalism as a method, Sullivan rests on a structural and functional argument about democratic self-government rather than on the original meaning of the text. The second concerns the public-figure category. The rule designed to protect criticism of the police commissioner of Montgomery now applies to a wide range of people who hold no power, from local business owners to people who became briefly famous online. Critics argue that the extension has been driven by lower courts' readiness to find that almost anyone who speaks publicly on a controversy has voluntarily assumed the risk of being defamed. The third concerns the incentive structure. Because actual malice asks what the publisher believed, the publisher who investigates and discovers contrary evidence is more exposed than the one who never checks at all. Gorsuch's point, that the standard can reward ignorance, has force in a media environment where many publishers have no editorial process to speak of. The arguments for retaining the rule are also substantial. The first is precedent. Sullivan has been reaffirmed for sixty years, and a large body of law, and the business practices of every American news organization, has been built upon it. The Court's own criteria for overruling precedent, which include whether a rule has proved unworkable and whether reliance interests are strong, weigh heavily in its favor. The second is that the rule's critics overstate its effect. Recent cases show that plaintiffs can win actual-malice cases when the evidence supports them. The Dominion, Newsmax and Freeman litigation all involved defendants who spread falsehoods knowingly or with clear disregard of the truth, and all produced large recoveries. The rule does not protect deliberate disinformation. What it protects is honest error and aggressive reporting that cannot be fully verified before publication. The third, and most important in practical terms, is what the alternative would look like. A return to the common law would mean presumed falsity, presumed damages and liability without fault for statements about officials and powerful figures. The lesson of Alabama in 1960 is that such rules can be weaponized by officials to punish coverage they dislike. The recent history of suits by political figures against news organizations suggests that the lesson has not become obsolete. Critics who worry about the power of large media companies might reflect that the chief beneficiaries of weakening Sullivan would be those with the resources to sue, and the chief victims those without the resources to defend. Proposals in between Not every reform proposal involves overruling Sullivan outright. In 1988 Judge Pierre Leval, then a federal district judge, proposed what he called the "no-money, no-fault" libel suit, in which a plaintiff could sue for a judicial declaration that a statement was false, without proving fault and without seeking damages. The Annenberg Washington Program published a similar proposal the same year. The idea was to give plaintiffs what many say they want most, a public vindication of the truth, without exposing publishers to damages for honest mistakes. The Uniform Law Commission later promulgated a Uniform Correction or Clarification of Defamation Act designed to encourage prompt corrections by limiting damages when they were made. These approaches have attracted academic support but little legislative adoption, partly because plaintiffs' lawyers are rarely paid for declaratory judgments. New publishers, same questions A final source of pressure comes from technology. When an artificial-intelligence system generates false statements about a real person, who, if anyone, has published them, and what was that publisher's state of mind? The first American case to reach a decision on the merits, Walters v. OpenAI, arose when a journalist using ChatGPT received output falsely stating that a Georgia radio host, Mark Walters, had been accused of embezzlement in a lawsuit. Walters sued in Georgia state court. In May 2025, the Superior Court of Gwinnett County granted summary judgment to OpenAI. The court reasoned that the journalist, who had been warned about the system's limitations and knew the output was false, could not reasonably have understood it as stating actual facts; that Walters, a nationally syndicated host, was a public figure who could not show actual malice; that he could not even show negligence given the company's efforts to reduce errors and warn users; and that he had suffered no damages. The case did not settle the broader questions. A private-figure plaintiff, a widely disseminated false output, or proof that a developer had been told of a specific falsehood and failed to correct it could produce a different result. But Walters showed that the existing framework, with its focus on reasonable understanding, fault and harm, can be applied to new kinds of speakers. Whether it should be, and how the actual-malice inquiry into a publisher's subjective belief translates to a system with no beliefs, is likely to be one of the defining questions of the next decade of defamation law. For now, Sullivan stands. But its survival depends not only on the Court's willingness to reaffirm it but also on whether the protection it offers can be made real for defendants who cannot afford to spend years proving they did not act with actual malice. That is the problem of the SLAPP. Chapter 5: The Anatomy of a SLAPP A defamation claim that has no realistic chance of success at trial can still be an effective weapon. It does not need to win. It needs only to survive long enough, and cost enough, to make the defendant regret speaking, and to make others think twice before doing the same. Lawsuits of this kind have a name, the strategic lawsuit against public participation, or SLAPP. Understanding how they work, and why the substantive protections described in the previous chapters do not stop them, is essential to understanding why more than thirty states have now enacted procedural remedies against them. The discovery of a pattern The term was coined in the 1980s by two professors at the University of Denver, George Pring, a law professor, and Penelope Canan, a sociologist. They had noticed a recurring type of case: ordinary citizens sued for millions of dollars because they had spoken at a zoning hearing, written to a legislator, circulated a petition against a development, reported a violation to a regulator, or complained to a school board. The plaintiffs were usually developers, businesses or public officials. The legal theories were varied: defamation, but also interference with business relationships, conspiracy, nuisance and even antitrust. What united the cases was that the defendants had been sued for trying to influence government. Pring and Canan's research, summarized in their 1996 book SLAPPs: Getting Sued for Speaking Out, found that the great majority of such suits failed in court, often after years of litigation. But the plaintiffs frequently achieved their objectives anyway. Targets withdrew their opposition, abandoned their petitions, or stopped speaking; neighbors who had not been sued took note. The researchers concluded that the lawsuits were effective precisely because litigation itself is the punishment. They described the phenomenon as a transformation of a public political dispute into a private legal one, moving the debate from a forum where the citizen had a voice to one where the better-funded party had the advantage. Pring and Canan's original definition was narrow. It focused on suits against people who had petitioned government, drawing on the First Amendment's Petition Clause. Over time the concept broadened to include suits against journalists, publishers, activists, researchers, consumer reviewers and anyone speaking on a matter of public concern. Modern anti-SLAPP statutes, and the European Union's 2024 directive discussed in Chapter 7, define protected activity broadly in the same way. What a SLAPP looks like No single feature identifies a SLAPP, but most share several of the following characteristics. There is a significant imbalance of resources. The plaintiff is a corporation, a wealthy individual, or a public official with access to public or political resources; the defendant is an individual, a small publication, or a nonprofit. The damages sought are out of proportion to any plausible injury. Claims for tens or hundreds of millions of dollars against individual critics serve less to measure harm than to frighten, to create reportable exposure and to deter insurers and employers. The claims multiply. A single statement becomes the basis for defamation, trade libel, tortious interference, civil conspiracy, intentional infliction of emotional distress and sometimes racketeering claims under the federal Racketeer Influenced and Corrupt Organizations Act. The additional theories are often weaker than the defamation claim, but they complicate an early motion to dismiss and can sometimes be used to try to evade the protections of defamation law. The targets are chosen strategically. Plaintiffs may sue individual reporters or sources rather than the publisher, which has insurance and in-house counsel; they may sue many defendants at once; or they may sue in a distant forum that is expensive to litigate in. Discovery is used as a weapon. Demands for reporters' notes, sources, internal communications, donor lists and membership rosters impose costs and create risks independent of the claim's merits. The goal is not judgment but surrender. Success is measured in retractions, apologies, withdrawn articles, settlement payments, promises not to speak again, and the deterrent effect on others. Why Sullivan does not stop them It might seem that the actual-malice standard would be enough to deter such suits. If the plaintiff must prove by clear and convincing evidence that the defendant knew its statement was false or seriously doubted it, surely weak claims would be quickly dismissed. In practice, several features of American civil procedure blunt the standard's effect. First, the standard applies at trial and at summary judgment, not at the pleading stage. Under the federal rules and their state counterparts, a complaint need only allege facts that make a claim plausible. A plaintiff can allege that the defendant knew its statements were false, cite some circumstantial reasons why, and often survive a motion to dismiss. Since the plausibility standard was tightened in Bell Atlantic Corp. v. Twombly in 2007 and Ashcroft v. Iqbal in 2009, federal courts have dismissed more defamation complaints that rely on conclusory allegations of actual malice, but the bar remains passable for a well-drafted complaint. Second, actual malice turns on state of mind, and state of mind is a fact-intensive question. As Chapter 2 explained, Herbert v. Lando allows plaintiffs to explore editorial processes in discovery. A defendant confident of winning at summary judgment must first produce documents, sit for depositions and make its reporters and editors available for questioning, all of which can take years. Third, the United States follows the American rule on legal fees: each side bears its own costs, win or lose. A defendant who defeats a meritless claim after three years of litigation has won, but has also paid, and the plaintiff who brought the claim bears none of that cost. Sanctions for frivolous litigation under Rule 11 of the Federal Rules of Civil Procedure and state equivalents are rarely imposed, because a claim need only have some colorable basis to avoid them. Fourth, the remedies that do exist for abusive litigation are slow. A defendant who wins can sue the original plaintiff for malicious prosecution or abuse of process, but such claims generally require the original case to have ended in the defendant's favor, require proof that it was brought without probable cause and for an improper purpose, and involve their own years of litigation. Fifth, protection varies with the defendant. Major news organizations carry media liability insurance and employ experienced counsel. Freelancers, independent journalists, academics, activists and ordinary citizens usually have neither, and for them even a few months of litigation may be unaffordable. The constitutional protection for petitioning activity faces the same limitation. Under the Noerr-Pennington doctrine, derived from Eastern Railroad Presidents Conference v. Noerr Motor Freight, Inc. in 1961 and United Mine Workers v. Pennington in 1965, attempts to influence government are generally immune from liability, except where the petitioning is a mere sham. The doctrine began in antitrust law and has been extended by many courts to other claims. It is a strong defense on the merits, but like actual malice, it must be raised and established within ordinary litigation. Examples from the field The following cases illustrate the range of lawsuits that critics have described as SLAPPs. In several of them, the plaintiffs dispute that characterization, and the label should be understood as describing a pattern rather than a legal finding. In 2017, after the comedian John Oliver devoted a segment of his HBO program to the coal industry and criticized the coal executive Robert Murray, Murray and his company sued HBO, Oliver and others for defamation in West Virginia state court. The segment was mocking and sharply critical, but its factual claims were largely drawn from public records. A West Virginia judge dismissed the case in 2018. In 2019, Representative Devin Nunes of California filed a series of defamation suits, including claims against Twitter and against the operators of two parody accounts, one purporting to be his cow and the other his mother, as well as against news organizations including Esquire and McClatchy. Most of the suits were dismissed. The claims against Twitter failed under Section 230 of the Communications Decency Act, which is discussed in Chapter 9. The litigation between the pipeline company Energy Transfer and Greenpeace shows the scale such cases can reach, and the disputes they generate. Energy Transfer first sued Greenpeace organizations in federal court in 2017 under the racketeering statute, seeking hundreds of millions of dollars in connection with protests against the Dakota Access Pipeline at Standing Rock. The federal court dismissed the case in 2019. The company refiled in North Dakota state court, alleging defamation, trespass, conspiracy and other claims. In March 2025, a jury in Morton County returned verdicts against Greenpeace entities totaling roughly $660 million, and in February 2026 the trial court entered a final judgment of about $345 million. Energy Transfer maintains that the case concerned unlawful conduct, not protected speech. Greenpeace and many press-freedom organizations call it a textbook SLAPP. Greenpeace International, based in the Netherlands, responded with its own lawsuit in Amsterdam seeking to recover its damages and costs, and in June 2026 the Amsterdam District Court held that it had jurisdiction and allowed the case to proceed. The dispute is a preview of the cross-border litigation discussed in Chapter 7. Another example involves privacy claims rather than defamation. In 2016, a Florida jury awarded the wrestler Terry Bollea, known as Hulk Hogan, $140 million against Gawker Media for publishing excerpts of a sex tape. Gawker filed for bankruptcy, and the case settled for $31 million. It then emerged that the litigation had been financed by the investor Peter Thiel, who had been the subject of Gawker coverage years earlier and who acknowledged that he had funded several lawsuits against the company. Many observers viewed the Bollea claim itself as legitimate; the episode's broader significance was the demonstration that a wealthy third party could fund litigation against a publication in order to destroy it. Some of the most consequential recent cases involve speech about sexual misconduct. After a 2018 op-ed in which the actor Amber Heard described herself as a survivor of domestic abuse without naming anyone, her former husband Johnny Depp sued her in Virginia. In 2022, a jury found that she had defamed him and awarded damages, which were reduced to $10.35 million because of Virginia's statutory cap on punitive damages; it also found for her on part of her counterclaim. In 2020, by contrast, Depp had lost a libel claim in England against the publisher of The Sun, which had called him a "wife beater," after the High Court found that the allegations were substantially true. The contrast between the two results became a staple of debates about whether defamation suits are being used to silence accusers. Legislatures have responded. California enacted a statute in 2023 making communications about sexual assault, harassment and discrimination privileged when made without malice, and allowing a defendant who prevails against a defamation claim based on such communications to recover fees. In 2025, a federal judge in New York dismissed a $400 million defamation suit by the director and actor Justin Baldoni and his production company against the actress Blake Lively and the New York Times arising from her complaints of harassment on a film set. These cases show that the categories of protected speech on which SLAPP debates focus are not limited to reporting by professional journalists. SLAPPs abroad The phenomenon is not American. In the United Kingdom, a series of cases involving wealthy claimants and investigative journalists in the early 2020s prompted a government consultation and eventual legislation. Catherine Belton's Putin's People, published by HarperCollins in 2020, attracted claims from several Russian businessmen and the state oil company Rosneft; one claimant, Roman Abramovich, settled in December 2021 in return for amendments to future editions. The journalist Tom Burgis and his publisher were sued by the mining company Eurasian Natural Resources Corporation over his book Kleptopia; the claim was struck out in 2022. The journalist Carole Cadwalladr was sued personally by the businessman Arron Banks over a TED talk and a tweet about his funding of the campaign for Brexit; she won at trial in 2022, but the Court of Appeal ruled in 2023 that one of her defenses had ceased to apply after a certain date and awarded Banks damages. Elsewhere in Europe, the Maltese journalist Daphne Caruana Galizia faced dozens of civil and criminal defamation suits when she was murdered by a car bomb in October 2017. Many of the civil suits continued against her family after her death. Her case became the emblem of the European campaign against SLAPPs and gave its name, informally, to the EU directive that followed. The lesson The consistent lesson of these cases is that the problem is procedural, not substantive. The substantive law of defamation, especially in the United States, already favors defendants in cases involving public controversy. What it does not do is decide those cases quickly or place the cost of meritless claims on the parties who bring them. The remedy, if there is one, must therefore be procedural as well: a mechanism that forces the plaintiff to show early that the claim has merit, stops expensive discovery until that showing is made, and shifts the defendant's costs to the plaintiff if it cannot be. That is the design of the anti-SLAPP statute. Chapter 6: Anti-SLAPP Statutes: Design, Spread and Limits If the SLAPP works by imposing cost and delay, the remedy is to take away both. Every serious anti-SLAPP statute is built around the same three ideas. It lets a defendant file an early motion that forces the plaintiff to show, before expensive discovery begins, that the claim has real merit. It pauses discovery and other proceedings while the motion is decided. And it makes the plaintiff pay the defendant's legal fees if the claim fails that test. Around those three ideas, the details vary enormously, and the details decide whether a statute works. California and the first generation The first American anti-SLAPP laws were modest. Washington enacted a statute in 1989 giving immunity to people who communicate information to government agencies, and New York in 1992 protected people who spoke out about applications for permits and similar government approvals. Both focused on petitioning in the narrow sense that Pring and Canan had studied. The statute that defined the field was California's, enacted in 1992 as section 425.16 of the Code of Civil Procedure. Its preamble declared that the legislature had found a disturbing increase in lawsuits brought primarily to chill the valid exercise of constitutional rights of freedom of speech and petition. In 1997, after courts had read the statute narrowly, the legislature amended it to direct that it be construed broadly. The California statute creates what it calls a special motion to strike, which courts analyze in two steps. At the first step, the defendant must show that the claim arises from protected activity. The statute defines this in four categories: statements made before a legislative, executive or judicial proceeding; statements made in connection with an issue under consideration by such a body; statements made in a place open to the public or a public forum in connection with an issue of public interest; and any other conduct in furtherance of the constitutional rights of petition or free speech in connection with a public issue or an issue of public interest. The defendant need not show that the plaintiff intended to chill speech, or that speech was actually chilled; the California Supreme Court so held in Equilon Enterprises v. Consumer Cause, Inc. in 2002. If the defendant makes that showing, the burden shifts. At the second step, the plaintiff must demonstrate a probability of prevailing on the claim. California courts treat this as a summary-judgment-like inquiry: the plaintiff must present admissible evidence that, if credited, would support a judgment in its favor. The court does not weigh the evidence or decide credibility; it asks whether the plaintiff's showing is legally sufficient and factually substantiated. In a defamation case involving a public figure, that means evidence from which a jury could find actual malice by clear and convincing evidence. While the motion is pending, discovery is stayed, although the court may allow specified discovery for good cause, for example where the plaintiff needs evidence of the defendant's state of mind that only the defendant possesses. A defendant who wins the motion is entitled to recover attorneys' fees and costs; the statute says the defendant "shall" recover them. A plaintiff can recover fees only if the court finds that the motion was frivolous or solely intended to cause delay. And an order granting or denying the motion is immediately appealable. California's statute has generated more appellate case law than any other, and its doctrine is the most developed in the country. The California Supreme Court has refined it repeatedly. In Baral v. Schnitt in 2016, it held that a motion may target specific allegations of protected activity within a cause of action that also rests on unprotected conduct. In FilmOn.com Inc. v. DoubleVerify Inc. in 2019, it held that in deciding whether a statement was made in connection with an issue of public interest, courts must consider the context, including the audience, the speaker and the purpose, and that a confidential report prepared for commercial clients did not qualify merely because it touched on a topic of public interest. The statute has also been criticized for overbreadth. Defendants in ordinary commercial, employment and landlord-tenant disputes have invoked it, and the right of immediate appeal lets a losing movant delay a case for a year or more. In 2003 the legislature responded to some of these concerns by enacting section 425.17, which exempts certain actions brought in the public interest and certain commercial speech by businesses about their own or competitors' products. The tension between broad protection and abuse of the protection itself runs through every anti-SLAPP regime. Texas and New York Texas enacted its Citizens Participation Act in 2011, and for several years it was among the broadest anti-SLAPP laws in the country. It applied to any legal action that was based on, related to or in response to a party's exercise of the right of free speech, petition or association, defined expansively. The plaintiff had to establish by "clear and specific evidence" a prima facie case for each essential element of the claim, a standard the Texas Supreme Court explained in In re Lipsky in 2015 as requiring more than general allegations but not a heightened burden of proof. Even if the plaintiff met that burden, the defendant could win by establishing each element of a valid defense. Fees were mandatory, and early versions of the statute also mandated sanctions against the plaintiff. The statute's breadth produced exactly the problem critics of California's law had warned about. Because nearly every lawsuit is related in some way to communication, the Texas act was invoked in trade-secret disputes, noncompetition cases, family-law matters and ordinary business torts. In 2019 the legislature narrowed it. The amended act applies to actions based on or in response to protected activity, removing the "relates to" language; it defines a "matter of public concern" more precisely; it exempts a list of claim types; and it makes sanctions discretionary rather than mandatory. The core protection for journalism and public debate remains. New York's original 1992 statute was one of the narrowest in the country, applying only to suits by applicants for public permits against their critics. In November 2020 the legislature transformed it. The amended law applies to any claim based on a communication in a place open to the public or a public forum in connection with an issue of public interest, or any other lawful conduct in furtherance of free speech or petition in connection with an issue of public interest. It requires a plaintiff whose claim is covered to establish that the claim has a substantial basis in law, stays discovery pending the motion, and makes fee awards mandatory for a defendant who prevails. The most distinctive feature of the New York law is substantive. For any covered claim, a plaintiff seeking damages must prove by clear and convincing evidence that the communication was made with knowledge of its falsity or reckless disregard for the truth. In effect, New York extended the actual-malice requirement to private-figure plaintiffs whenever the speech concerns an issue of public interest, going beyond what Gertz requires. The law's significance was illustrated in January 2024, when a New York court ordered Donald Trump to pay the New York Times and three of its reporters about $392,000 in legal fees after dismissing his suit over the paper's reporting on his family's finances and taxes. The uniform act The variation between state statutes created its own problems. A publisher operating nationally faced different levels of protection in each state, and plaintiffs could choose the most favorable forum. In 2020 the Uniform Law Commission, the body of state-appointed commissioners that drafts model legislation, approved the Uniform Public Expression Protection Act, known as UPEPA, as a template for states to adopt. UPEPA draws on California's structure but tightens it. Its scope covers communications in or relating to governmental proceedings and the exercise, on a matter of public concern, of the freedoms of speech, press, assembly, petition and association guaranteed by the federal or state constitution. It excludes claims against governmental units and officials acting in their official capacity, certain commercial speech, and certain consumer-protection and enforcement actions. A defendant must file the motion within sixty days after being served. Filing the motion stays all other proceedings, including discovery, although the court may allow specified discovery if a party shows it is necessary. The court must hear the motion within sixty days and rule within sixty days after the hearing. The court dismisses the claim if the defendant shows that the act applies and either the plaintiff fails to establish a prima facie case as to each essential element of the claim, or the defendant shows that the plaintiff failed to state a claim or that there is no genuine issue of material fact and the defendant is entitled to judgment as a matter of law. Fees and costs are mandatory for a prevailing defendant; the plaintiff can recover fees only if the motion was frivolous or filed solely to cause delay. Either side may appeal immediately. The prima facie standard was chosen deliberately to avoid a constitutional problem that had struck down earlier statutes. Washington's 2010 anti-SLAPP law required a plaintiff to establish by clear and convincing evidence a probability of prevailing, which the Washington Supreme Court held in Davis v. Cox in 2015 violated the state constitutional right to a jury trial, because it required the judge to weigh evidence and decide disputed facts. The Minnesota Supreme Court struck down that state's statute on similar grounds in Leiendecker v. Asian Women United of Minnesota in 2017. UPEPA's prima facie test, which asks only whether the plaintiff has evidence that would support a verdict if believed, avoids that difficulty. Adoption has been rapid. Washington, whose earlier statute had been struck down, was the first state to enact UPEPA in 2021. Kentucky followed in 2022, and adoptions accelerated after that, including Minnesota and Pennsylvania in 2024 and Ohio, Idaho, Iowa, Montana and Delaware in 2025. Michigan, which had no anti-SLAPP law at all, enacted the uniform act in late 2025, and South Dakota and Missouri followed in 2026. By mid-2026, roughly a third of the states had adopted UPEPA in some form, and more than thirty states plus the District of Columbia had an anti-SLAPP statute of some kind, although the strength of those statutes still ranged from comprehensive to nominal. The principal features of the leading models are compared in Table 2. Table 2. Key features of four leading anti-SLAPP regimes. Feature California (1992) Texas (2011, amended 2019) New York (amended 2020) UPEPA (2020 model) Plaintiff's burden Probability of prevailing, shown with admissible evidence Clear and specific evidence of a prima facie case Substantial basis in law; actual malice for damages Prima facie case on each element Discovery Stayed; specified discovery for good cause Suspended; limited discovery for good cause Stayed; limited discovery by order All proceedings stayed; specified discovery by order Fees to prevailing defendant Mandatory Mandatory Mandatory Mandatory Immediate appeal Yes Yes Ordinary state appellate practice Yes The federal court problem Anti-SLAPP statutes are creatures of state law. When a defamation case is filed in federal court, usually because the parties are citizens of different states, the federal court must decide whether the state statute applies. Under the doctrine that stems from Erie Railroad Co. v. Tompkins in 1938, federal courts in such cases apply state substantive law but federal procedural law, and where a Federal Rule of Civil Procedure answers the same question as a state law, the federal rule governs if it is valid. The Supreme Court applied this principle in Shady Grove Orthopedic Associates v. Allstate Insurance Co. in 2010, holding that a New York rule barring class actions for statutory penalties could not be applied in federal court because Rule 23 of the federal rules governed the same question. The federal courts of appeals have divided on whether anti-SLAPP motions survive this analysis. The First Circuit, in Godin v. Schencks in 2010, and the Ninth Circuit, beginning with United States ex rel. Newsham v. Lockheed Missiles & Space Co. in 1999, have applied state statutes in federal court, reasoning that they do not conflict with the federal rules and serve substantive state interests. The Ninth Circuit later adjusted its approach, holding in Planned Parenthood Federation of America v. Center for Medical Progress in 2018 that when an anti-SLAPP motion attacks the legal sufficiency of a complaint, federal courts apply the ordinary motion-to-dismiss standard, and when it attacks the factual sufficiency, they apply the summary-judgment standard and allow discovery. Judges on that court have continued to question whether the state statute belongs in federal court at all. Most other circuits have refused to apply the statutes. The District of Columbia Circuit, in an opinion by then-Judge Brett Kavanaugh in Abbas v. Foreign Policy Group, LLC in 2015, held that the District's anti-SLAPP act could not be applied because Federal Rules 12 and 56 answer the question of when a court may dismiss a claim before trial. The Eleventh Circuit reached the same conclusion about Georgia's statute in Carbone v. Cable News Network, Inc. in 2018, as did the Tenth Circuit about New Mexico's in Los Lobos Renewable Power, LLC v. AmeriCulture, Inc. in 2018, the Fifth Circuit about Texas's in Klocke v. Watson in 2019, and the Second Circuit about California's in La Liberte v. Reid in 2020. On January 20, 2026, the Supreme Court decided Berk v. Choy, a medical-malpractice case that did not involve defamation at all. Delaware law required a plaintiff suing for malpractice to file an affidavit of merit from a qualified expert along with the complaint. The Court held, in an opinion by Justice Amy Coney Barrett, that the requirement could not be applied in federal court because it conflicted with Rules 8 and 12, which specify what a complaint must contain and when a claim may be dismissed. The case said nothing directly about anti-SLAPP statutes, but its reasoning closely parallels the analysis of the circuits that have refused to apply them. Commentators across the spectrum read it as a strong signal that anti-SLAPP motions, which likewise require a plaintiff to produce evidence at the outset of a case, will not survive in federal court, and that the First and Ninth Circuits' approach is now difficult to defend. The practical consequence is a strategic gap. A plaintiff who can establish diversity jurisdiction may prefer to file in federal court specifically to avoid the state's anti-SLAPP law, and a defendant sued in federal court may lose the protections the state legislature intended. Whether the fee-shifting provisions of anti-SLAPP statutes, as distinct from their dismissal procedures, might still apply in federal court as substantive state law is an open question, and some courts have been receptive to that argument. The obvious solution is a federal statute. Bills to create a federal anti-SLAPP procedure have been introduced in Congress repeatedly since 2015. The most ambitious versions would create a special motion to dismiss for federal claims, allow defendants sued under state law in state court to remove SLAPPs to federal court, and provide for fee-shifting. None has reached a vote in either chamber. After Berk, the absence of federal legislation matters more than it did before. The comparative picture Other common-law jurisdictions have developed their own models. Ontario amended its Courts of Justice Act in 2015 to add a procedure for dismissing proceedings that limit debate on matters of public interest. The Ontario approach includes an explicit weighing step: even if a plaintiff shows that its claim has substantial merit and that the defendant has no valid defense, the court must dismiss if the harm to the plaintiff is not sufficiently serious that the public interest in permitting the proceeding outweighs the public interest in protecting the expression. The Supreme Court of Canada, in 1704604 Ontario Ltd. v. Pointes Protection Association in 2020, described this weighing as the crux of the analysis. British Columbia adopted similar legislation in 2019. The Canadian model gives courts more discretion than American statutes, and correspondingly less predictability. Australia has been slower. Only the Australian Capital Territory has a dedicated anti-SLAPP statute, and national defamation reforms beginning in 2021 have focused instead on a serious-harm threshold and a public-interest defense. The United Kingdom and the European Union, whose reforms are closely tied to the problem of cross-border litigation, are discussed in the next chapter. What anti-SLAPP laws cannot do Even the best statute has limits. It helps only defendants sued in states that have one, and after Berk possibly only in state court. It does not deter plaintiffs indifferent to fee awards, such as the very wealthy, or those who can shield assets. It does not prevent suits in foreign jurisdictions. It does not protect against subpoenas for sources or documents in litigation to which the journalist is not a party. And an overbroad statute can itself be abused, allowing powerful defendants to shut down legitimate claims by less powerful plaintiffs, including claims by private individuals who have genuinely been defamed. The best-designed laws answer these concerns by focusing their protection on speech of public concern, using a prima facie standard that respects the right to a jury trial, allowing targeted discovery where the plaintiff genuinely needs it, and requiring fast decisions. On those measures, UPEPA represents the current consensus. Its spread has substantially narrowed the geographic gaps in protection within the United States. The gaps that remain are federal and international. Chapter 7: Libel Tourism and the Borders of Reputation Defamation law is national, but publication is not. A book printed in New York can be bought online in London; an article posted in Washington can be read in Melbourne; a broadcast produced in Paris can be streamed in Chicago. Each place where a statement is read may apply its own law to it, and those laws differ profoundly. For decades, claimants who could not win under the American rules exploited those differences by suing in countries whose law was more favorable to them. The practice became known as libel tourism, and the response to it, first in the United States and then in England and across Europe, forms the third layer of protection for public debate. London, the libel capital Until 2014, England and Wales offered a claimant almost everything the American common law had offered before Sullivan. Defamatory statements were presumed false, and the defendant had to prove their truth. There was no general requirement that the defendant have been at fault. Damage to reputation was presumed. The claimant did not have to show that anyone had actually thought less of him. Defenses for responsible journalism existed but were narrow and uncertain; the House of Lords created a qualified privilege for responsible reporting on matters of public interest in Reynolds v. Times Newspapers Ltd. in 1999, and strengthened it in Jameel v. Wall Street Journal Europe in 2006, but in practice defendants found the defense hard to establish. Costs magnified the imbalance. English litigation was among the most expensive in the world, and under the English rule the losing party paid the winner's costs. Conditional fee arrangements, under which a claimant's lawyer could charge a success fee of up to double the normal rate if the claim succeeded, and after-the-event insurance premiums were recoverable from the losing defendant. A publisher who lost a libel case could face a bill several times larger than the damages. The rules on jurisdiction made London accessible to claimants from anywhere. Under a principle traced to Duke of Brunswick v. Harmer in 1849, each publication of a defamatory statement was a separate cause of action, and publication occurred wherever a statement was read. A foreign claimant who could show a reputation in England, and some publication there, could sue in the English courts. In Berezovsky v. Michaels in 2000, the House of Lords allowed two Russian businessmen to sue the American magazine Forbes in England over an article that had sold around 1,900 copies there and hundreds of thousands in North America. In Polanski v. Condé Nast Publications Ltd. in 2005, it allowed the film director Roman Polanski, a fugitive from American justice who would not travel to England for fear of extradition, to give evidence by video link in his suit against Vanity Fair. Other courts sometimes drew limits. In Jameel v. Dow Jones & Co. in 2005, the Court of Appeal struck out a claim as an abuse of process when only five people in England had read the article, three of them members of the claimant's own camp. But the general rule favored claimants. The High Court of Australia adopted a similar approach to online publication in Dow Jones & Co. v. Gutnick in 2002. An article in the financial weekly Barron's, published on a website hosted in New Jersey, discussed the business dealings of an Australian businessman. The court held that publication occurred where the article was downloaded and read, and that he could therefore sue in his home state of Victoria under Australian law. The decision confirmed that internet publishers could be sued anywhere their content was accessed. The Mahfouz cases The case that turned libel tourism into an American political issue involved the Saudi banker Khalid bin Mahfouz. After the attacks of September 11, 2001, several books and articles alleged links between him and the financing of terrorism. He brought or threatened a number of claims in England. In 2007, Cambridge University Press agreed to withdraw and destroy remaining copies of a book it had published on Islamic charities, Alms for Jihad, after he complained. One of the targets, the American author Rachel Ehrenfeld, refused to engage. Her book Funding Evil, published in the United States in 2003, was not marketed in England, but a small number of copies, reported as twenty-three, had been purchased there through online booksellers, and a chapter was available on the internet. Bin Mahfouz sued her in London. She declined to appear, and in 2005 the High Court entered a default judgment against her, awarding damages and declaring the allegations false. Ehrenfeld then sued bin Mahfouz in federal court in New York, seeking a declaration that the English judgment could not be enforced against her in the United States. The case turned on whether New York courts had personal jurisdiction over bin Mahfouz, who had never done business in New York in the relevant sense. In Ehrenfeld v. Bin Mahfouz in 2007, the New York Court of Appeals, answering a question certified by the Second Circuit, held that the state's long-arm statute did not reach him. The foreign plaintiff could obtain an English judgment, and the American author could not obtain an American court's declaration that it was unenforceable until he chose to try to enforce it. The American response American courts had refused to enforce foreign libel judgments before. In Bachchan v. India Abroad Publications Inc. in 1992, a New York trial court declined to enforce an English judgment in favor of an Indian businessman against a news service that had reported allegations linking him to a Swedish arms-procurement scandal. Enforcing a judgment rendered under English law, which did not place the burden of proving falsity on the plaintiff, would be repugnant to the public policy embodied in the First Amendment. In Telnikoff v. Matusevitch in 1997, the Maryland Court of Appeals reached the same conclusion about an English judgment arising from a letter published in the Daily Telegraph in a dispute between two Soviet émigrés. These decisions rested on the general principle that courts need not enforce foreign judgments contrary to fundamental public policy. But they were case-by-case, and they offered no protection against the threat of a foreign judgment hanging over a writer indefinitely. The legislative response came first in New York. In 2008 the state enacted the Libel Terrorism Protection Act, sometimes called Rachel's Law, which allowed New York courts to exercise jurisdiction over foreign plaintiffs who had obtained defamation judgments against New York residents, for the purpose of declaring those judgments unenforceable, and barred recognition of foreign defamation judgments unless the foreign court had provided at least as much protection for freedom of speech and press as the United States and New York constitutions. Illinois, Florida, California and other states adopted similar laws. Congress followed with the Securing the Protection of our Enduring and Established Constitutional Heritage Act, known as the SPEECH Act, which passed both chambers without opposition and was signed into law on August 10, 2010. Codified at sections 4101 to 4105 of Title 28 of the United States Code, it provides that a federal or state court shall not recognize or enforce a foreign judgment for defamation unless the court determines either that the foreign law applied provided at least as much protection for freedom of speech and press as the First Amendment and the relevant state's constitution and law, or that the party opposing recognition would have been found liable for defamation by an American court applying those protections. The burden of establishing either condition falls on the party seeking enforcement. The court must also be satisfied that the foreign court's exercise of personal jurisdiction complied with American due-process standards, and a judgment against an online service provider must be consistent with Section 230 of the Communications Decency Act. An American person against whom a foreign defamation judgment has been entered may bring an action in federal court for a declaration that it is unenforceable, and a defendant who successfully opposes enforcement is entitled to a reasonable attorney's fee in most cases. The statute has been applied in only a small number of reported cases, which itself suggests it has had a deterrent effect. The leading appellate decision is Trout Point Lodge, Ltd. v. Handshoe, decided by the Fifth Circuit in 2013. The owners of a Nova Scotia resort had obtained a Canadian defamation judgment against a Mississippi blogger who had linked them to a political corruption scandal. The Fifth Circuit held the judgment unenforceable under the SPEECH Act: the plaintiffs had not shown that Canadian defamation law provided protection equivalent to American law, which, among other things, would have required them to prove falsity, and they had not shown that the blogger would have been liable under American law. The SPEECH Act is an effective shield against enforcement in the United States. It has two important limits. It does nothing to prevent a foreign judgment from being enforced against assets outside the United States, so a publisher or author with assets or business in England or elsewhere remains exposed. And it does not prevent the foreign suit itself. A defendant who ignores a foreign claim risks a default judgment that may restrict travel or business abroad; a defendant who defends it bears the cost of foreign litigation. The SPEECH Act solved the enforcement problem, not the forum problem. England reforms The pressure for reform in England came as much from within as from abroad. In 2008 the British Chiropractic Association sued the science writer Simon Singh over a Guardian column describing some chiropractic treatments for children as "bogus." The High Court ruled that the word implied deliberate dishonesty, which Singh would have to prove true. The Court of Appeal reversed in 2010, holding that the statement was an expression of opinion, and the association dropped the claim. The case, together with others involving scientists and doctors, fuelled a campaign by English PEN, Index on Censorship and Sense About Science that attracted wide public support. The result was the Defamation Act 2013, which came into force in England and Wales on January 1, 2014. Its most important provisions were these: 1. Serious harm. Under section 1, a statement is not defamatory unless its publication has caused or is likely to cause serious harm to the claimant's reputation. For a business trading for profit, serious harm requires serious financial loss. In Lachaux v. Independent Print Ltd. in 2019, the Supreme Court of the United Kingdom held that this requires proof of actual or likely impact, assessed on the facts, not merely the inherent tendency of the words. 1. Public interest. Section 4 replaced the Reynolds privilege with a statutory defense for publication on a matter of public interest where the defendant reasonably believed that publishing was in the public interest. The Supreme Court considered it in Serafin v. Malkiewicz in 2020. 2. Single publication. Section 8 introduced a single publication rule, so that the limitation period runs from the first publication rather than restarting with each download of the same material. 3. Jurisdiction. Section 9 provides that where the defendant is not domiciled in the United Kingdom, the court has no jurisdiction unless it is satisfied that England and Wales is clearly the most appropriate place in which to bring the action. The provision was aimed squarely at libel tourism. The Act also created defenses for website operators and for peer-reviewed statements in scientific or academic journals, and removed the presumption of trial by jury. Separately, reforms to litigation funding that took effect in 2013 ended the recoverability of success fees and insurance premiums from losing defendants in most cases. The reforms appear to have reduced classic libel tourism, in which foreign claimants sued foreign defendants in London over publications with little English connection. They did less about a problem that became more prominent in the following decade: wealthy foreign claimants suing English journalists and publishers, who are domiciled in England and therefore outside section 9, over investigative reporting. The cases described in Chapter 5 involving Catherine Belton, Tom Burgis and others were of this kind. Section 9 protects foreign defendants from English courts; it does nothing for English defendants. Parliament responded in the Economic Crime and Corporate Transparency Act 2023, which contains the first statutory definition of a SLAPP in English law and requires court rules providing for the early dismissal of SLAPP claims relating to the disclosure of information about economic crime where the claimant has not shown that the claim is more likely than not to succeed at trial, together with protections against adverse costs. The provisions came into effect in 2025. They are limited to claims concerning economic crime, and campaigners have argued that a general anti-SLAPP law is needed. As of mid-2026, the government had indicated that it would focus on implementing the existing provisions rather than legislating more broadly, though pressure in Parliament continued. Scotland, which has its own law of defamation, reformed it in the Defamation and Malicious Publication (Scotland) Act 2021, which introduced a serious-harm threshold and other changes, and campaigners have pressed for dedicated anti-SLAPP legislation there as well. Europe's directive In continental Europe, the problem of cross-border SLAPPs has been shaped by the European Union's rules on jurisdiction. Under the Brussels I Regulation as interpreted by the Court of Justice of the European Union in eDate Advertising GmbH v. X in 2011, a person claiming to have been defamed online may sue in the member state where the publisher is established, in the member state where the claimant has the center of his interests for all the damage caused, or in each member state where the content was accessible for the damage caused there. The EU's rules on the law applicable to non-contractual obligations exclude defamation, so the law applied depends on the forum. The combination allows claimants considerable room to choose a favorable court. After the murder of Daphne Caruana Galizia, the European Parliament and civil-society groups campaigned for EU legislation. The result was Directive (EU) 2024/1069, adopted in April 2024, on protecting persons who engage in public participation from manifestly unfounded claims or abusive court proceedings. Because the EU's competence in civil procedure is limited to matters with cross-border implications, the directive applies only to such cases, although the Commission issued a parallel recommendation in 2022 urging member states to adopt similar protections for purely domestic cases, and the Council of Europe adopted a recommendation on SLAPPs for its forty-six member states in 2024. The directive requires member states to provide for the early dismissal of manifestly unfounded claims, with the burden on the claimant to substantiate that the claim is not manifestly unfounded. It allows courts to require the claimant to provide security for costs and damages, requires that a claimant who brings an abusive proceeding can be ordered to bear all costs, including the defendant's full legal fees, and provides for penalties or other deterrent measures. It permits nongovernmental organizations to support defendants in proceedings. Two provisions address the problem the SPEECH Act addressed in the United States. Member states must ensure that recognition and enforcement of a judgment from a non-EU country in proceedings against a person in the EU engaged in public participation are refused if those proceedings are considered manifestly unfounded or abusive under the law of the member state where recognition is sought. And a person in the EU who is the target of such abusive proceedings in a non-EU country may sue for damages and costs in the courts of his or her own member state. Member states were required to implement the directive by May 7, 2026. Only a handful did so on time; by one account, fewer than a quarter of the member states bound by the directive had completed transposition by the deadline, and the European Commission began proceedings against those that had not. How national courts interpret "manifestly unfounded" and "abusive" will determine the directive's real effect. The directive's provisions on third-country proceedings have obvious relevance to American litigation. When Greenpeace International, headquartered in Amsterdam, sued Energy Transfer in the Netherlands in 2025, it relied on Dutch law and on the principles underlying the directive rather than on the directive itself, which did not apply retroactively to the American suits. The Amsterdam court's decision in June 2026 to accept jurisdiction nevertheless suggests how European courts may increasingly be asked to police litigation brought elsewhere. The irony is striking: tools designed to protect European speakers against litigation in foreign courts may now be invoked against litigation in American courts. The comparative picture The five systems discussed in this and the preceding chapter differ on the questions that most affect a publisher's exposure, as Table 3 sets out. Table 3. How five systems allocate the main burdens in defamation and SLAPP litigation. Issue United States England and Wales Canada (Ontario) European Union directive Who proves falsity Plaintiff, on matters of public concern Defendant proves truth Defendant proves truth Left to national law Fault or public-interest defense Actual malice for public figures; at least negligence otherwise Statutory public-interest defense Responsible communication defense (2009) Left to national law Threshold for a claim No general harm threshold Serious harm No general threshold; weighing on anti-SLAPP motion Not addressed Early dismissal State anti-SLAPP statutes Economic-crime SLAPPs only Statutory motion with public-interest weighing Manifestly unfounded cross-border claims Foreign judgments SPEECH Act bars enforcement absent equivalent protection Ordinary recognition rules Ordinary recognition rules Refusal of abusive non-EU judgments The Canadian responsible communication defense was recognized by the Supreme Court of Canada in Grant v. Torstar Corp. in 2009. It protects publication on matters of public interest where the publisher was diligent in trying to verify the allegation, taking into account the seriousness of the allegation, its urgency and the reliability of its sources. What borders still decide The combined effect of the SPEECH Act, the Defamation Act 2013 and the EU directive is that the most extreme forms of libel tourism are now much harder to carry out. A foreign judgment against an American author is essentially unenforceable in the United States; a claimant with a weak connection to England will struggle to sue a foreign defendant there; and EU courts will soon be obliged to refuse enforcement of abusive foreign judgments. What remains is subtler. Publishers with assets, staff or distribution abroad can still be sued where those things are. Journalists who travel face the risk of judgments or even criminal defamation proceedings in countries where defamation remains a crime. And the reach of the internet means that every publication is potentially a publication everywhere. The protection a writer enjoys therefore depends not only on the law where he or she lives but also on the law of every place his or her work is read, and the cost of defending in the least protective of those places. Chapter 8: The Reporter's Privilege and Confidential Sources Much of the journalism that has mattered most in American history depended on people who could not afford to be named. The Washington Post's Watergate reporting relied on a senior FBI official whose identity remained secret for more than thirty years, until W. Mark Felt acknowledged in 2005 that he had been the source known as Deep Throat. Reporting on torture at Abu Ghraib, warrantless surveillance, sexual abuse within churches and sexual harassment by powerful men in entertainment and business drew on sources who spoke only after being promised confidentiality. A source who fears being fired, sued, prosecuted or physically harmed will not speak without that promise, and a reporter's promise is worth only as much as the law allows the reporter to keep it. The protection of sources is the fourth layer of the system this book describes. It differs from the others in one important respect. Actual malice, anti-SLAPP statutes and the SPEECH Act protect journalists from liability for what they publish. Source protection concerns compulsion: whether a court, a prosecutor or the police can force a journalist to reveal who told them something, or seize the materials that would reveal it. The threat is usually not a damages judgment but a subpoena, a search warrant, or a jail cell for contempt. Branzburg and its ambiguity The Supreme Court has addressed the question directly only once. Branzburg v. Hayes, decided in 1972, consolidated three cases. Paul Branzburg, a reporter for the Louisville Courier-Journal, had written articles about the manufacture and use of illegal drugs in Kentucky, based on observation of people who had been promised anonymity, and was subpoenaed by state grand juries to identify them. Paul Pappas, a television reporter in Massachusetts, and Earl Caldwell, a reporter for the New York Times, had covered the Black Panther Party and were subpoenaed by grand juries investigating it. By a vote of five to four, the Court held that the First Amendment does not give reporters a privilege to refuse to appear and testify before a grand jury about information relevant to a criminal investigation. Justice Byron White's majority opinion reasoned that the public interest in law enforcement and in effective grand jury proceedings outweighed the uncertain burden on newsgathering that would result from requiring reporters to testify, like other citizens, about crimes they had witnessed. The Court was skeptical that the flow of information to the press would dry up, and it worried about the difficulty of defining who qualifies as a journalist. Congress and the states, it noted, remained free to create statutory privileges. The majority opinion also said that newsgathering is not without First Amendment protection, and that grand jury investigations conducted in bad faith or for purposes of harassment would raise different issues. And Justice Lewis Powell, who supplied the fifth vote, wrote a short concurrence emphasizing the limited nature of the holding. A reporter called to testify, he wrote, would have access to the court on a motion to quash if the testimony sought bore only a remote and tenuous relationship to the subject of the investigation, or if there were other reasons to believe the reporter was being harassed. Claims of privilege should be judged case by case by striking a proper balance between freedom of the press and the obligation of all citizens to give relevant testimony. Justice Potter Stewart's dissent, joined by Justices Brennan and Thurgood Marshall, proposed a three-part test that has influenced courts ever since. The government should be required to show that there is probable cause to believe the reporter has information clearly relevant to a specific probable violation of law; that the information cannot be obtained by alternative means less destructive of First Amendment rights; and that there is a compelling and overriding interest in the information. Justice William O. Douglas dissented separately, arguing for an absolute privilege. The combination of a narrow majority, a concurrence that seemed to endorse balancing, and an influential dissent left lower courts room to interpret. Many federal courts of appeals concluded that Branzburg rejected a privilege only before grand juries acting in good faith, and recognized a qualified privilege in other settings, particularly civil cases. Under the approach adopted in circuits such as the Second, Third and District of Columbia Circuits, a party seeking to compel disclosure of confidential sources in a civil case must generally show that the information is highly relevant, goes to the heart of the claim, and cannot be obtained from other sources. The Second Circuit extended a weaker form of the privilege to nonconfidential materials, such as unpublished footage, in Gonzales v. National Broadcasting Co. in 1999. Other courts were more skeptical. In McKevitt v. Pallasch in 2003, Judge Richard Posner, writing for the Seventh Circuit, questioned whether any federal constitutional privilege existed beyond the general requirement that subpoenas be reasonable. In criminal cases, and above all before grand juries, the qualified privilege has proved weak. When a special counsel investigated the disclosure of the identity of the CIA officer Valerie Plame, the District of Columbia Circuit held in 2005 that New York Times reporter Judith Miller and Time reporter Matthew Cooper had no First Amendment privilege to refuse to testify before the grand jury. Miller spent eighty-five days in jail before testifying after her source released her from her promise. In United States v. Sterling in 2013, the Fourth Circuit held that the Times reporter James Risen had no privilege to refuse to testify at the criminal trial of a former CIA officer accused of leaking information to him; the government ultimately chose not to call him. The freelance videographer Josh Wolf spent 226 days in federal custody in 2006 and 2007 for refusing to provide a grand jury with video of a protest in San Francisco, the longest term served by an American journalist for such a refusal. When the qualified privilege gives way Even in civil cases, the qualified privilege is only qualified. Its limits were demonstrated in the litigation arising from the investigation of Wen Ho Lee, a scientist at the Los Alamos National Laboratory who was investigated as a suspected spy in 1999 and ultimately pleaded guilty only to a single count of mishandling information. Lee sued the government under the Privacy Act for leaking information about him and subpoenaed reporters who had written about the investigation. The courts held that he had overcome the privilege, and reporters were held in contempt. In 2006, to end the dispute, five news organizations agreed to contribute $750,000 to a settlement between Lee and the government, an extraordinary step taken so that their reporters would not have to identify their sources. A more recent case followed the same path and ended differently. In 2017, the reporter Catherine Herridge, then at Fox News, published stories about an FBI counterintelligence investigation of Yanping Chen, a Chinese-American scientist who had founded a graduate school in Virginia. The stories drew on investigative materials, including photographs, that Chen contended had been leaked by government officials in violation of the Privacy Act. Chen was never charged. She sued the FBI and other agencies and, after failing to identify the leaker through discovery, subpoenaed Herridge. In February 2024, Judge Christopher Cooper of the federal district court in Washington held Herridge in civil contempt for refusing to identify her sources and imposed a fine of $800 per day, which was stayed pending appeal. The court applied the District of Columbia Circuit's qualified privilege and found that Chen had satisfied it: the information went to the heart of her claim, and she had exhausted reasonable alternatives. In September 2025 the District of Columbia Circuit affirmed, and on July 2, 2026, the Supreme Court declined Herridge's application to stay the order while she sought further review. The Herridge case matters because it shows the privilege failing in the setting where it is supposed to be strongest: a civil case in which the reporter is not a party. It also shows how Privacy Act suits can turn a journalist into the only available witness against the government, since the government officials who leaked are rarely identifiable any other way. State shield laws Because the federal constitutional privilege is limited and uncertain, most practical protection for sources comes from the states. Nearly every state protects journalists' sources in some form, most by statute and some by judicial decision; Wyoming is commonly identified as the only state with neither a shield statute nor a recognized privilege. The protections vary considerably. Some are absolute for confidential sources. New York's shield law, section 79-h of the Civil Rights Law, gives absolute protection to information obtained under a promise of confidentiality and qualified protection to nonconfidential material. California placed its shield in the state constitution in 1980, providing that a journalist cannot be held in contempt for refusing to disclose sources or unpublished information, although the California Supreme Court has held that the protection may yield to a criminal defendant's constitutional right to a fair trial. New Jersey strengthened its statute after the New York Times reporter Myron Farber was jailed for forty days in 1978 for refusing to turn over materials in a murder trial. Other states offer only qualified protection, applying tests similar to Justice Stewart's. The statutes also differ on whom they protect. Older statutes often covered only employees of newspapers, magazines and broadcasters; newer ones and courts interpreting older ones have wrestled with whether they reach freelancers, documentary filmmakers, authors, student journalists and bloggers. In Obsidian Finance Group, LLC v. Cox in 2014, the Ninth Circuit held that a blogger was entitled to the same First Amendment protection in a defamation case as the institutional press, but the state shield law at issue in the trial court had not protected her, because she did not fit its categories. State shield laws have a significant limitation: they generally do not apply in federal court. Under Rule 501 of the Federal Rules of Evidence, privileges in federal court are governed by federal common law, except that in civil cases where state law supplies the rule of decision, such as diversity defamation cases, state privilege law applies. A reporter subpoenaed in a federal criminal investigation or a federal-question civil case, like a Privacy Act suit, cannot rely on the shield law of her home state. Searches, seizures and the Privacy Protection Act Subpoenas at least give the journalist an opportunity to object in court before complying. Search warrants do not. In Zurcher v. Stanford Daily in 1978, the Supreme Court held that the Fourth Amendment did not bar police from searching the offices of a student newspaper, which was not suspected of any crime, for photographs of a violent demonstration. Congress responded two years later with the Privacy Protection Act of 1980, which generally prohibits federal, state and local officials from searching for or seizing a journalist's work product or documentary materials held in connection with a purpose to disseminate information to the public. The statute has exceptions, including where there is probable cause to believe that the journalist has committed a crime to which the materials relate, and where immediate seizure is necessary to prevent death or serious bodily injury. Where it applies, officials must usually proceed by subpoena rather than warrant, and the journalist can sue for damages if they do not. The statute's importance was illustrated vividly in August 2023, when police in Marion, Kansas, raided the offices of the weekly Marion County Record and the homes of its co-owner and a city council member, seizing computers and phones in an investigation into how the paper had obtained publicly available information about a local restaurant owner's driving record. The co-owner, Joan Meyer, who was ninety-eight years old, died the day after the raid. The county prosecutor withdrew the warrant within days, finding insufficient evidence to support it. The raid drew national condemnation, and in November 2025 Marion County agreed to pay $3 million and to issue an apology to settle claims brought by the newspaper and others. A similar episode in San Francisco in 2019, when police used sledgehammers to enter the home of the freelance journalist Bryan Carmody in an effort to identify the source of a leaked police report, ended with the city paying a settlement and the warrants being quashed. The executive branch's shifting rules At the federal level, the most important constraints on subpoenas to journalists in criminal matters have come not from the courts or Congress but from the Justice Department itself. Since 1970, department guidelines have required high-level approval before prosecutors subpoena reporters or their records, and have directed prosecutors to exhaust alternatives first. The guidelines are internal, and journalists cannot enforce them in court. Their weaknesses became evident in 2013, when it was revealed that the department had secretly obtained two months of telephone records for lines used by Associated Press reporters and editors, and that an FBI affidavit in another leak investigation had described the Fox News reporter James Rosen as a possible co-conspirator in order to obtain a warrant for his emails. The department revised its guidelines in 2014 and 2015. In 2021 it emerged that during the previous administration the department had secretly obtained telephone and email records of reporters at the Washington Post, CNN and the New York Times in leak investigations. Attorney General Merrick Garland responded with a policy, formalized as a regulation in October 2022, that generally barred compulsory process to obtain information from or records of members of the news media acting within the scope of newsgathering, with narrow exceptions. On April 25, 2025, Attorney General Pam Bondi rescinded that policy, citing the need to pursue leaks of classified and sensitive information, and restored a framework under which subpoenas, court orders and search warrants directed at journalists are permitted in leak investigations with senior approval. The episode showed the fragility of protection that depends on executive policy: what one attorney general grants, the next can withdraw. The obvious alternative is a federal shield statute. Versions have been introduced in Congress for decades. The most recent, the Protect Reporters from Exploitative State Spying Act, known as the PRESS Act, would bar the federal government from compelling journalists to disclose sources or unpublished information, and from compelling telephone and internet providers to disclose journalists' records, subject to narrow exceptions for preventing terrorism and imminent violence. It passed the House of Representatives without opposition in January 2024 but never received a vote in the Senate. In November 2024, the President-elect publicly urged Republicans to kill the bill, and it has not been enacted. The main sources of protection, and their limits, are summarized in Table 4. Table 4. The layers of protection for journalists' sources in the United States. Source of protection Where it applies Strength Who can change it First Amendment qualified privilege Federal and many state courts; strongest in civil cases Qualified; weak before grand juries Courts State shield laws State courts; federal diversity cases Absolute to qualified, varying by state State legislatures and courts Privacy Protection Act of 1980 Searches and seizures by officials at every level Statutory bar with exceptions; damages remedy Congress Justice Department media policy Federal prosecutors and investigators Internal only; not enforceable by journalists Attorney General (tightened 2022, rescinded 2025) PRESS Act (proposed) All federal proceedings and providers Near-absolute with narrow exceptions Not enacted Promises and their enforcement One final case shows that source protection runs in both directions. In 1982, a political operative named Dan Cohen gave reporters for two Minnesota newspapers documents about a rival candidate's criminal record in exchange for a promise that his name would not be used. The editors overruled the reporters and identified him, and he was fired. He sued for breach of promise, and in Cohen v. Cowles Media Co. in 1991, the Supreme Court held that the First Amendment did not bar enforcing the promise under the generally applicable doctrine of promissory estoppel. The press, the Court said, has no special immunity from laws of general applicability. The case gives sources a legal claim against journalists who break their word, and gives journalists a reason to be precise about what they promise. A comparative note European law offers sources stronger protection than American federal law. In Goodwin v. United Kingdom in 1996, the European Court of Human Rights held that an order requiring a trainee journalist to reveal the source of information about a company's finances violated Article 10 of the European Convention on Human Rights, declaring that protection of journalistic sources is one of the basic conditions for press freedom. Later decisions have required that any order to disclose be reviewed by an independent judicial body before it takes effect. The European Media Freedom Act, a regulation adopted by the European Union in 2024, goes further, restricting member states from using detention, sanctions, searches or intrusive surveillance software to obtain information about journalists' sources, subject to limited exceptions. In the United Kingdom, section 10 of the Contempt of Court Act 1981 provides a statutory presumption against compelled disclosure. The digital problem The legal protections described in this chapter were designed around a world in which the evidence of a source's identity was in a reporter's notebook or memory. Today, it is usually in records held by third parties: telephone carriers, email providers, messaging platforms and cloud services. A subpoena to a provider does not require the journalist's cooperation and may not even be disclosed to her. That is why the Garland policy and the PRESS Act addressed records held by providers, and why the 2025 rescission matters so much. It is also why many newsrooms now use encrypted messaging applications and anonymous submission systems, and why the most effective source protection may increasingly be technical rather than legal. The law protects a promise of confidentiality only imperfectly. Encryption, when it works, protects it absolutely. Chapter 9: Anonymity, Unmasking and the Defendant's Dilemma The previous chapter dealt with journalists who know who their sources are and refuse to say. This chapter concerns two related problems. The first arises when the speaker is anonymous to everyone: a person who criticizes a company, an official or a neighbor online under a pseudonym, and who is then sued as "John Doe" by a plaintiff seeking to learn his identity. The second arises when a publisher that relied on a confidential source is itself sued for defamation and must choose between revealing the source and defending the case with one hand tied. Both problems test how far the law's protection of anonymity extends when a claim of defamation is on the other side of the scale. The constitutional tradition of anonymous speech Anonymity has deep roots in American political speech. The essays now known as the Federalist Papers were published under the pseudonym Publius; their Anti-Federalist opponents wrote as Brutus, Cato and the Federal Farmer. The Supreme Court has recognized that tradition as constitutionally significant. In Talley v. California in 1960, it struck down a Los Angeles ordinance banning the distribution of handbills that did not identify their author. In McIntyre v. Ohio Elections Commission in 1995, it held that Ohio could not punish a woman for distributing unsigned leaflets opposing a proposed school tax levy. Justice John Paul Stevens's opinion for the Court described anonymous pamphleteering as an honorable tradition of advocacy and dissent and observed that anonymity is a shield from the tyranny of the majority. A speaker may choose anonymity to avoid retaliation, to avoid social ostracism, or simply so that readers will judge the message rather than the messenger. The Court has recognized a related right in the privacy of association. In NAACP v. Alabama ex rel. Patterson in 1958, it held that Alabama could not compel the NAACP to disclose its membership lists, given the danger of reprisal against members at the height of the civil rights struggle. In Americans for Prosperity Foundation v. Bonta in 2021, it struck down California's requirement that charities disclose their major donors to the state attorney general, holding that compelled disclosure requirements are subject to exacting scrutiny. None of these cases involved defamation. The right to speak anonymously has never included a right to defame anonymously; a false and defamatory statement is no more protected because its author is unknown. The question in the unmasking cases is procedural: how much must a plaintiff show before a court will strip away a speaker's anonymity, given that the unmasking itself may be the plaintiff's real goal? How unmasking works A plaintiff who believes he has been defamed by an anonymous online post cannot serve a complaint on a pseudonym. The usual procedure is to file suit against a "Doe" defendant and then issue a subpoena to the platform that hosted the post, and later to the internet service provider, for information identifying the account holder. Many platforms have policies of notifying users before complying, giving them an opportunity to move to quash the subpoena, often anonymously through counsel. The danger is obvious. If unmasking requires nothing more than filing a complaint, anyone can learn the identity of a critic simply by alleging defamation, and then retaliate outside the courtroom through dismissal from employment, social pressure or harassment, without ever pursuing the claim. Employers have used Doe lawsuits to identify employees who criticized them anonymously on message boards; public officials have used them to identify constituents who mocked them online. For such plaintiffs, the lawsuit is a means of obtaining a name, not a judgment. Early courts applied lenient standards. A Virginia trial court in 2000, in a case involving a subpoena to America Online, required only that the plaintiff show a good-faith basis for believing that it had been the victim of actionable conduct. The courts that followed developed tougher tests. Dendrite and Cahill The most influential test came from New Jersey. In Dendrite International, Inc. v. Doe No. 3, decided by the Appellate Division of the New Jersey Superior Court in 2001, a software company sought to identify anonymous posters who had criticized it on a Yahoo message board devoted to its stock. The court established a four-part procedure. First, the plaintiff must undertake efforts to notify the anonymous posters that they are the subject of a subpoena or application for disclosure, and give them a reasonable opportunity to oppose it. Second, the plaintiff must identify the exact statements alleged to be actionable. Third, the plaintiff must establish a prima facie cause of action, supported by evidence as to each element of the claim within the plaintiff's control. Fourth, even if the plaintiff clears those hurdles, the court must balance the defendant's First Amendment right of anonymous speech against the strength of the prima facie case and the necessity of disclosure. Applying this test, the court declined to order the identification of the poster in question, because Dendrite had not shown that the statements had caused it harm. The Delaware Supreme Court adopted a streamlined version in Doe v. Cahill in 2005. A town councilman in Smyrna, Delaware, sued an anonymous blogger who had posted comments questioning his mental fitness and leadership. The court held that the plaintiff must make reasonable efforts to notify the anonymous defendant and must present evidence sufficient to survive a motion for summary judgment on the elements of the claim within his control. It dropped the separate balancing step, reasoning that the summary-judgment standard itself adequately protected the defendant's rights. Importantly, it did not require the plaintiff to prove actual malice at this stage, since a plaintiff cannot know the defendant's state of mind without knowing who the defendant is. The court then held that the councilman's claim failed because the statements, read in the context of an online blog known for opinion and hyperbole, could not reasonably be understood as asserting facts. Many other jurisdictions have adopted one version or another of these tests. The Arizona Court of Appeals in Mobilisa, Inc. v. Doe in 2007 combined Cahill's summary-judgment standard with Dendrite's balancing step. Courts in the District of Columbia, Maryland, New Hampshire, Texas and elsewhere have used similar approaches, and federal district courts in California and elsewhere have applied them in federal cases. The Ninth Circuit, in In re Anonymous Online Speakers in 2011, held that the appropriate level of scrutiny depends on the nature of the speech: political, religious and literary speech receives the highest protection, and commercial speech between business competitors less. The tests work. They force plaintiffs to show real evidence of falsity and harm before unmasking, and they frequently result in the protection of speakers whose statements, read in context, are opinion or hyperbole. Their weakness is that they are judge-made and inconsistent, and a speaker's protection depends on where the plaintiff chooses to sue and where the platform is located. The Supreme Court of Virginia showed how complex the jurisdictional issues can be in Yelp, Inc. v. Hadeed Carpet Cleaning, Inc. in 2015. A carpet-cleaning company sought to unmask reviewers who it alleged had never been its customers. The lower courts ordered Yelp to comply, but the Supreme Court of Virginia vacated that order on the ground that Virginia courts lacked power to enforce a subpoena against a company based in California. Anti-SLAPP statutes supply additional protection. Under many of them, including UPEPA, an anonymous defendant can move to dismiss through counsel without revealing his identity, and discovery, including discovery aimed at unmasking, is stayed until the motion is decided. California provides for an award of fees to a person who successfully moves to quash a subpoena seeking his identity in connection with protected speech. Section 230 and why unmasking matters Unmasking matters so much in the United States because of Section 230 of the Communications Decency Act of 1996. Its central provision states that no provider or user of an interactive computer service shall be treated as the publisher or speaker of any information provided by another information content provider. The Fourth Circuit gave the provision a broad reading in Zeran v. America Online, Inc. in 1997, holding that it barred a defamation claim against an online service for failing to remove defamatory posts even after it had been notified of them. Courts across the country have followed Zeran, and the result is that platforms are generally immune from defamation liability for content created by their users. There are limits. A platform that materially contributes to the unlawfulness of content may itself be an information content provider and lose the immunity, as the Ninth Circuit held in Fair Housing Council of San Fernando Valley v. Roommates.com, LLC in 2008 about a website that required users to answer discriminatory questions. And the immunity does not apply to federal criminal law, intellectual property claims and a few other categories. The Supreme Court had an opportunity to address the scope of Section 230 in Gonzalez v. Google LLC in 2023, but declined to do so, resolving the case on other grounds. Section 230's significance for defamation is double-edged. On one side, it protects the infrastructure of online speech. Without it, platforms facing liability for every user post would have strong incentives to remove anything a complainant claimed was defamatory, and critics of powerful people would be the first to disappear. It also blocks some of the most aggressive SLAPPs, as when Representative Nunes's suit against Twitter over parody accounts was dismissed. The SPEECH Act, discussed in Chapter 7, reinforces this by requiring that foreign judgments against online service providers be consistent with Section 230. On the other side, it means that a person defamed online usually has only one defendant to sue: the original speaker. If that speaker is anonymous, unmasking is the only path to any remedy at all. That places heavy weight on the unmasking tests, which must protect legitimate anonymous criticism while still giving genuinely defamed plaintiffs a way to identify and sue those responsible. Other legal systems have struck different balances. The European Court of Human Rights held in Delfi AS v. Estonia in 2015 that holding a commercial news portal liable for clearly unlawful anonymous comments posted on its site did not violate the Convention, though it limited that holding the following year in a case involving less offensive comments on a nonprofit site. English courts can order an intermediary to disclose the identity of a wrongdoer under the equitable jurisdiction recognized in Norwich Pharmacal Co. v. Customs and Excise Commissioners in 1974. Ireland's reformed defamation law, most of which came into force in March 2026, gives its Circuit Court power to order platforms to identify anonymous posters of defamatory material. The American approach remains the most protective of both platforms and anonymous speakers. The defendant's dilemma The final problem brings the story back to journalism. Suppose a newspaper publishes a story based on information from a confidential source, and the subject of the story sues for defamation. To defend itself, the newspaper may need to show that it had a reliable source and good reason to believe what the source said, since that evidence is central to disproving actual malice or negligence. But proving that it had a reliable source may require identifying the source, which the newspaper has promised not to do. Courts have handled this dilemma in several ways. Most recognize that the qualified privilege discussed in Chapter 8 applies even when the journalist is a party, but that it can be overcome when the source's identity goes to the heart of the plaintiff's claim, and a plaintiff trying to prove actual malice can often make that showing. The more difficult question is what happens when the journalist refuses to comply with an order to disclose. Courts cannot easily jail a corporate defendant, and holding reporters in contempt may not produce the information. Some courts have instead imposed evidentiary consequences. In Downing v. Monitor Publishing Co. in 1980, the New Hampshire Supreme Court held that if a media defendant refused to disclose its sources after being ordered to do so, the jury could be instructed to presume that there was no source. Other courts have barred defendants from relying on the undisclosed source at trial, or have entered default judgments. Such sanctions leave the defendant with a choice between breaking a promise and losing the case. The dilemma shapes editorial practice. Many news organizations require that anonymous sources be known to at least one editor, that their information be corroborated by other sources or documents whenever possible, and that stories explain why anonymity was granted. The purpose is partly journalistic credibility and partly legal protection: a story that can be defended with documents and on-the-record witnesses does not require the newspaper to reveal a confidential source. The St. Amant decision discussed in Chapter 2 identified reliance on an unverified anonymous telephone call as a classic basis for inferring reckless disregard. The lesson for journalists is that anonymous sourcing is protected when it is careful and vulnerable when it is not. Anonymity and its costs The law's protection of anonymous speech rests on the recognition that people will say true and important things only if they can do so without being identified. It also accepts that some people will abuse anonymity to spread falsehoods. The unmasking tests, anti-SLAPP statutes and the qualified reporter's privilege all attempt to separate the two early, without forcing speakers to reveal themselves unless a plaintiff can show a real claim. That task has become harder. Online harassment campaigns, coordinated disinformation and synthetic media have made anonymous speech a vehicle for serious harm in ways its defenders in the 1990s did not anticipate. At the same time, the same technologies have made it easier than ever for powerful actors to identify, surveil and retaliate against their critics. The balance the courts have struck in cases like Dendrite and Cahill does not resolve these tensions, but it offers a principled way to manage them: anonymity yields only to evidence, not to allegation. Conclusion: Protection Is Only as Strong as Its Weakest Layer The preceding chapters have described a system built in stages, each responding to a failure of the one before. Sullivan changed what a plaintiff must prove. Anti-SLAPP statutes changed how quickly and at whose expense that proof must be offered. The SPEECH Act and its foreign counterparts changed where a claim can be pressed and enforced. Shield laws, the Privacy Protection Act and the unmasking tests govern who can be compelled to reveal what. Seen as a whole, the system is impressive. Seen from the position of any individual speaker, it is only as strong as the weakest layer that happens to apply. What the layers mean for a real defendant Consider two writers who publish the same investigative article about the same corporate executive. The first is a staff reporter at a national newspaper in a state with a strong anti-SLAPP law, sued in state court. She has the actual-malice standard, an early motion that stays discovery, a mandatory fee award if the claim fails, an employer with insurance and in-house counsel, and a state shield law that protects her sources. The second is a freelancer in a state with a weak statute or none, sued in federal court on diversity grounds. She has the actual-malice standard and nothing else: no early dismissal, no stay of discovery, no fee recovery, no insurer, and a qualified federal privilege that may or may not protect her sources. The law of defamation is formally the same for both. Their practical freedom to publish is not. That gap is the central lesson of this book. The strength of American press protection is usually measured by the substantive rule, and on that measure it is the strongest in the world. But substantive rules protect only those who can afford to reach the stage at which they apply. The procedural, jurisdictional and evidentiary layers decide who can. Where the gaps are widest Four gaps stand out as of late 2026. The first is in federal court. The Supreme Court's decision in Berk v. Choy has made it likely that state anti-SLAPP procedures will not apply in federal diversity cases anywhere in the country. Plaintiffs will notice. The rational response for a plaintiff seeking to impose cost on a critic from another state will be to file in federal court, and only Congress can close that door. A federal anti-SLAPP statute, applying to federal claims and allowing removal of state-law SLAPPs to federal court, has been proposed in some form for a decade. It is now the single most important reform available. The second is federal source protection. The qualified privilege recognized by the lower federal courts has proved unable to protect a journalist in a routine civil case, as the Herridge litigation showed. The Justice Department's restraint depends on the attorney general of the day, as the rescission of the 2022 regulations in April 2025 showed. The PRESS Act passed the House without opposition and then died. Without a federal statute, a journalist's promise to a source in a matter involving the federal government is a promise that federal courts and prosecutors can override. The third is abroad. The SPEECH Act protects American defendants' assets in the United States, but not their assets elsewhere or their ability to travel, and it does not spare them the cost of defending foreign suits. The European Union's directive and the United Kingdom's limited reforms are meaningful steps, but most EU member states missed the directive's deadline, and the British government has so far declined to extend its protections beyond cases involving economic crime. The emerging practice of suing in one's home court to recover the costs of a SLAPP brought elsewhere, tested in the Greenpeace litigation in Amsterdam, may produce new conflicts between American and European courts rather than resolving old ones. The fourth is the pressure on Sullivan itself. The Court has declined every invitation to reconsider it, and there is no sign of a majority for doing so. But the arguments against it have become mainstream in some legal and political circles, and the settlements paid by major media companies to a sitting President suggest that the practical protection of Sullivan can be eroded without any change in doctrine. When a corporation settles a weak claim because of regulatory exposure or political risk, it signals to every future plaintiff that such claims have value. What should not change It does not follow that every proposal to reform defamation law should be resisted. The critics of Sullivan are right that the public-figure category has expanded in ways that can leave ordinary people who became briefly prominent without an effective remedy, and that the actual-malice standard offers little incentive to verify. They are right, too, that the internet has made reputational injury easier to inflict and harder to repair. There is room for reforms that would give defamed people better access to correction and vindication without exposing publishers to ruinous damages for honest errors, including the declaratory-judgment and correction models that have been proposed for decades and never widely enacted. Lower courts could, within Gertz, apply the limited-purpose public-figure category more carefully to people who did not seek attention. But the core of Sullivan should stand. Its premise, that a rule requiring critics of the powerful to guarantee the truth of every factual statement will suppress true statements along with false ones, is at least as sound today as it was in 1964. The recent cases in which plaintiffs have won large recoveries, against Fox, Newsmax and Giuliani, demonstrate that the standard does not protect deliberate disinformation. The recent cases in which powerful plaintiffs have sued critics for amounts in the billions demonstrate why the standard remains necessary. A return to the common law would not primarily benefit the private individuals whose plight critics invoke. It would benefit those with the resources to sue. What follows for those who publish For journalists and publishers, the practical lessons follow from how the law actually works. Actual malice is proved from internal records, so the best protection is a newsroom whose records show verification: calls made, documents checked, contrary evidence considered and addressed. A prompt correction does not erase liability, but it is powerful evidence of good faith, as the Palin litigation showed. Reliance on anonymous sources should be deliberate, documented and corroborated, because a story that can be defended without revealing a source is a story that does not force a choice between the source and the case. Writers who work without institutional support should know whether their state has an anti-SLAPP statute, whether they have any form of media liability insurance, and whether organizations that offer pro bono defense for journalists, such as press-freedom legal clinics and advocacy groups, can help. For media organizations, the lesson of the last two years is that settlements of weak claims have consequences beyond the individual case. A company that pays to resolve a suit it would likely have won makes the next suit more likely, not only against itself but against every other publisher. Organizations with the resources to litigate meritless claims to judgment perform a service for those without them. What follows for everyone else It is tempting to see these questions as matters for lawyers and large media companies. They are not. The anti-SLAPP statutes described in this book protect the resident who speaks against a development at a zoning hearing, the parent who complains to a school board, the patient who writes an honest review of a doctor, and the employee who reports harassment. The unmasking tests protect the anonymous critic of a local official. The SPEECH Act protects any American author whose book is bought abroad. The protections were built for journalism, but they belong to anyone who takes part in public life. The system has come a long way since an Alabama jury awarded a city commissioner $500,000 against a newspaper for an advertisement that did not mention his name. The lesson of that case was that a law of reputation, applied without regard to its effect on public debate, can become an instrument of power. The lesson of the sixty years since is that the protection of public debate requires not a single rule but a structure, and that the structure has to be maintained. Most of the work that remains is legislative, and most of it is unglamorous: statutes on procedure, privilege and jurisdiction. It is also where the freedom to speak about the powerful will, in practice, be won or lost. Notes Introduction and Chapter 1 1. New York Times Co. v. Sullivan, 376 U.S. 254 (1964). The facts of the advertisement, the Alabama circulation figures, the trial instructions and the $500,000 award are drawn from the Court's opinion. 1. Chaplinsky v. New Hampshire, 315 U.S. 568 (1942). 2. Beauharnais v. Illinois, 343 U.S. 250 (1952). 3. Coleman v. MacLennan, 78 Kan. 711, 98 P. 281 (1908). 4. NAACP v. Button, 371 U.S. 415 (1963), source of the "breathing space" formulation. 5. Anthony Lewis, Make No Law: The Sullivan Case and the First Amendment (Random House, 1991), for the background of the Alabama litigation and the suits against the ministers. Chapter 2 1. Garrison v. Louisiana, 379 U.S. 64 (1964). 1. St. Amant v. Thompson, 390 U.S. 727 (1968). 2. Curtis Publishing Co. v. Butts and Associated Press v. Walker, 388 U.S. 130 (1967). 3. Harte-Hanks Communications, Inc. v. Connaughton, 491 U.S. 657 (1989). 4. Herbert v. Lando, 441 U.S. 153 (1979). 5. Bose Corp. v. Consumers Union of United States, Inc., 466 U.S. 485 (1984). 6. Anderson v. Liberty Lobby, Inc., 477 U.S. 242 (1986). 7. Philadelphia Newspapers, Inc. v. Hepps, 475 U.S. 767 (1986). 8. Masson v. New Yorker Magazine, Inc., 501 U.S. 496 (1991). 9. Hustler Magazine, Inc. v. Falwell, 485 U.S. 46 (1988). 10. Milkovich v. Lorain Journal Co., 497 U.S. 1 (1990). 11. US Dominion, Inc. v. Fox News Network, LLC, Delaware Superior Court, opinion on summary judgment (March 31, 2023); settlement announced April 18, 2023. 12. Newsmax settlement with Dominion Voting Systems, announced August 18, 2025. 13. Freeman v. Giuliani, No. 1:21-cv-03354 (D.D.C.), jury verdict December 15, 2023. 14. Palin v. New York Times Co., Second Circuit decision ordering a new trial (2024); jury verdict for the defendant on retrial, Southern District of New York (April 22, 2025). Chapter 3 1. Rosenblatt v. Baer, 383 U.S. 75 (1966). 1. Monitor Patriot Co. v. Roy, 401 U.S. 265 (1971). 2. Rosenbloom v. Metromedia, Inc., 403 U.S. 29 (1971). 3. Gertz v. Robert Welch, Inc., 418 U.S. 323 (1974). 4. Chapadeau v. Utica Observer-Dispatch, Inc., 38 N.Y.2d 196 (1975). 5. Time, Inc. v. Firestone, 424 U.S. 448 (1976). 6. Hutchinson v. Proxmire, 443 U.S. 111 (1979). 7. Wolston v. Reader's Digest Association, Inc., 443 U.S. 157 (1979). 8. Dun & Bradstreet, Inc. v. Greenmoss Builders, Inc., 472 U.S. 749 (1985). 9. Waldbaum v. Fairchild Publications, Inc., 627 F.2d 1287 (D.C. Cir. 1980). 10. Dameron v. Washington Magazine, Inc., 779 F.2d 736 (D.C. Cir. 1985). 11. Sandmann v. New York Times Co., 78 F.4th 319 (6th Cir. 2023). Chapter 4 1. McKee v. Cosby, 139 S. Ct. 675 (2019) (Thomas, J., concurring in the denial of certiorari). 1. Berisha v. Lawson, 141 S. Ct. 2424 (2021) (Thomas, J., and Gorsuch, J., dissenting from the denial of certiorari). 2. Coral Ridge Ministries Media, Inc. v. Southern Poverty Law Center, 142 S. Ct. 2453 (2022) (Thomas, J., dissenting from the denial of certiorari). 3. Wynn v. Associated Press, petition for certiorari denied March 24, 2025. 4. Florida House Bill 991 (2023 session), which died without enactment. 5. Trump v. O'Brien, New Jersey Superior Court, Appellate Division (2011). 6. Pierre N. Leval, "The No-Money, No-Fault Libel Suit: Keeping Sullivan in Its Proper Place," 101 Harvard Law Review 1287 (1988). 7. Walters v. OpenAI, L.L.C., Superior Court of Gwinnett County, Georgia, order granting summary judgment (May 2025). Chapter 5 1. George W. Pring and Penelope Canan, SLAPPs: Getting Sued for Speaking Out (Temple University Press, 1996). 1. Bell Atlantic Corp. v. Twombly, 550 U.S. 544 (2007); Ashcroft v. Iqbal, 556 U.S. 662 (2009). 2. Eastern Railroad Presidents Conference v. Noerr Motor Freight, Inc., 365 U.S. 127 (1961); United Mine Workers v. Pennington, 381 U.S. 657 (1965). 3. Greenpeace International, public materials on the North Dakota judgment and the Dutch proceedings; Amsterdam District Court ruling on jurisdiction (June 2026). 4. Depp v. News Group Newspapers Ltd. [2020] EWHC 2911 (QB). 5. California Civil Code section 47.1. 6. Banks v. Cadwalladr [2023] EWCA Civ 219. Chapter 6 1. California Code of Civil Procedure sections 425.16 and 425.17. 1. Equilon Enterprises v. Consumer Cause, Inc., 29 Cal. 4th 53 (2002). 2. Baral v. Schnitt, 1 Cal. 5th 376 (2016). 3. FilmOn.com Inc. v. DoubleVerify Inc., 7 Cal. 5th 133 (2019). 4. Texas Civil Practice and Remedies Code, chapter 27 (Texas Citizens Participation Act), as amended in 2019. 5. In re Lipsky, 460 S.W.3d 579 (Tex. 2015). 6. New York Civil Rights Law sections 70-a and 76-a; New York Civil Practice Law and Rules 3211(g), as amended in 2020. 7. Uniform Law Commission, Uniform Public Expression Protection Act (2020), with official comments. 8. Davis v. Cox, 183 Wash. 2d 269 (2015). 9. Leiendecker v. Asian Women United of Minnesota, 895 N.W.2d 623 (Minn. 2017). 10. Erie Railroad Co. v. Tompkins, 304 U.S. 64 (1938); Shady Grove Orthopedic Associates, P.A. v. Allstate Insurance Co., 559 U.S. 393 (2010). 11. Godin v. Schencks, 629 F.3d 79 (1st Cir. 2010); United States ex rel. Newsham v. Lockheed Missiles & Space Co., 190 F.3d 963 (9th Cir. 1999); Planned Parenthood Federation of America, Inc. v. Center for Medical Progress, 890 F.3d 828 (9th Cir. 2018). 12. Abbas v. Foreign Policy Group, LLC, 783 F.3d 1328 (D.C. Cir. 2015); Carbone v. Cable News Network, Inc., 910 F.3d 1345 (11th Cir. 2018); Los Lobos Renewable Power, LLC v. AmeriCulture, Inc., 885 F.3d 659 (10th Cir. 2018); Klocke v. Watson, 936 F.3d 240 (5th Cir. 2019); La Liberte v. Reid, 966 F.3d 79 (2d Cir. 2020). 13. Berk v. Choy, Supreme Court of the United States (January 20, 2026). 14. 1704604 Ontario Ltd. v. Pointes Protection Association, 2020 SCC 22. Chapter 7 1. Reynolds v. Times Newspapers Ltd. [2001] 2 AC 127 (HL); Jameel v. Wall Street Journal Europe SPRL [2006] UKHL 44. 1. Duke of Brunswick v. Harmer (1849) 14 QB 185. 2. Berezovsky v. Michaels [2000] 1 WLR 1004 (HL); Polanski v. Condé Nast Publications Ltd. [2005] UKHL 10; Jameel v. Dow Jones & Co. [2005] EWCA Civ 75. 3. Dow Jones & Co. v. Gutnick [2002] HCA 56. 4. Ehrenfeld v. Bin Mahfouz, 9 N.Y.3d 501 (2007). 5. Bachchan v. India Abroad Publications Inc., 154 Misc. 2d 228 (N.Y. Sup. Ct. 1992); Telnikoff v. Matusevitch, 347 Md. 561 (1997). 6. SPEECH Act, Pub. L. No. 111-223, 124 Stat. 2380 (2010), codified at 28 U.S.C. sections 4101 to 4105. 7. Trout Point Lodge, Ltd. v. Handshoe, 729 F.3d 481 (5th Cir. 2013). 8. British Chiropractic Association v. Singh [2010] EWCA Civ 350. 9. Defamation Act 2013; Lachaux v. Independent Print Ltd. [2019] UKSC 27; Serafin v. Malkiewicz [2020] UKSC 23. 10. Economic Crime and Corporate Transparency Act 2023, sections 194 and 195; House of Commons Library, The impact of strategic lawsuits against public participation (SLAPPs), Research Briefing CBP-10955 (July 2026). 11. Defamation and Malicious Publication (Scotland) Act 2021. 12. eDate Advertising GmbH v. X and Martinez v. MGN Ltd., Joined Cases C-509/09 and C-161/10 (CJEU, 2011). 13. Directive (EU) 2024/1069 of the European Parliament and of the Council of 11 April 2024; Commission Recommendation (EU) 2022/758; Council of Europe, Recommendation CM/Rec(2024)2. 14. European Association of Private International Law, "Status of the Anti-SLAPP Directive Transpositions at the 7 May 2026 Deadline" (May 2026). 15. Grant v. Torstar Corp., 2009 SCC 61. Chapter 8 1. Branzburg v. Hayes, 408 U.S. 665 (1972). 1. Gonzales v. National Broadcasting Co., 194 F.3d 29 (2d Cir. 1999); McKevitt v. Pallasch, 339 F.3d 530 (7th Cir. 2003). 2. In re Grand Jury Subpoena, Judith Miller, 397 F.3d 964 (D.C. Cir. 2005). 3. United States v. Sterling, 724 F.3d 482 (4th Cir. 2013). 4. Chen v. Federal Bureau of Investigation (D.D.C.), contempt order of February 29, 2024; affirmed by the D.C. Circuit (2025); application for stay denied by the Supreme Court (July 2, 2026). 5. New York Civil Rights Law section 79-h; California Constitution, article I, section 2(b); In re Farber, 78 N.J. 259 (1978). 6. Obsidian Finance Group, LLC v. Cox, 740 F.3d 1284 (9th Cir. 2014). 7. Federal Rule of Evidence 501. 8. Zurcher v. Stanford Daily, 436 U.S. 547 (1978); Privacy Protection Act of 1980, 42 U.S.C. section 2000aa. 9. Marion County, Kansas, settlement announced November 2025. 10. 28 C.F.R. section 50.10 (2022); Attorney General memorandum rescinding the policy (April 25, 2025). 11. PRESS Act, H.R. 4250, 118th Congress, passed by the House of Representatives in January 2024. 12. Cohen v. Cowles Media Co., 501 U.S. 663 (1991). 13. Goodwin v. United Kingdom (1996) 22 EHRR 123; Regulation (EU) 2024/1083 (European Media Freedom Act); Contempt of Court Act 1981, section 10. Chapter 9 1. Talley v. California, 362 U.S. 60 (1960); McIntyre v. Ohio Elections Commission, 514 U.S. 334 (1995). 1. NAACP v. Alabama ex rel. Patterson, 357 U.S. 449 (1958); Americans for Prosperity Foundation v. Bonta, 594 U.S. 595 (2021). 2. Dendrite International, Inc. v. Doe No. 3, 342 N.J. Super. 134, 775 A.2d 756 (App. Div. 2001). 3. Doe v. Cahill, 884 A.2d 451 (Del. 2005). 4. Mobilisa, Inc. v. Doe, 217 Ariz. 103, 170 P.3d 712 (Ct. App. 2007). 5. In re Anonymous Online Speakers, 661 F.3d 1168 (9th Cir. 2011). 6. Yelp, Inc. v. Hadeed Carpet Cleaning, Inc., 289 Va. 426 (2015). 7. 47 U.S.C. section 230; Zeran v. America Online, Inc., 129 F.3d 327 (4th Cir. 1997); Fair Housing Council of San Fernando Valley v. Roommates.com, LLC, 521 F.3d 1157 (9th Cir. 2008) (en banc); Gonzalez v. Google LLC, 598 U.S. 617 (2023). 8. Delfi AS v. Estonia, Application No. 64569/09 (ECtHR, Grand Chamber, 2015); Norwich Pharmacal Co. v. Customs and Excise Commissioners [1974] AC 133. 9. Downing v. Monitor Publishing Co., 120 N.H. 383 (1980). Further Reading Anthony Lewis, Make No Law: The Sullivan Case and the First Amendment (Random House, 1991).** The classic narrative account of Sullivan, from the advertisement through the Supreme Court's decision, by a journalist who covered the Court for the New York Times. Still the best introduction to why the case mattered. Samantha Barbas, Actual Malice: Civil Rights and Freedom of the Press in New York Times v. Sullivan (University of California Press, 2023).** A legal historian's detailed reconstruction of the case, drawing on archival material, which places it firmly in the context of the civil rights movement and the Southern campaign of libel suits against the national press. Lee Levine and Stephen Wermiel, The Progeny: Justice William J. Brennan's Fight to Preserve the Legacy of New York Times v. Sullivan (American Bar Association, 2014).** Traces the internal history of the Supreme Court's defamation cases from Sullivan through the 1980s, including Gertz and Dun & Bradstreet, using the justices' papers. George W. Pring and Penelope Canan, SLAPPs: Getting Sued for Speaking Out (Temple University Press, 1996).** The study that named the phenomenon. Its case material is dated, but its analysis of why meritless suits succeed in silencing their targets remains the foundation of the field. Pierre N. Leval, "The No-Money, No-Fault Libel Suit: Keeping Sullivan in Its Proper Place," Harvard Law Review, vol. 101 (1988).** A federal judge's proposal for a declaratory action that lets a plaintiff establish falsity without proving fault or recovering damages. Essential reading for anyone interested in reforms that fall short of overruling Sullivan. Robert D. Sack, Sack on Defamation: Libel, Slander, and Related Problems (Practising Law Institute, regularly updated).** The standard practitioner's treatise on American defamation law, written by a judge of the Second Circuit who spent his early career as a media lawyer. Comprehensive and readable. Reporters Committee for Freedom of the Press, Anti-SLAPP Legal Guide and The Reporter's Privilege Compendium (online, regularly updated).** State-by-state guides to anti-SLAPP statutes and to the law of reporter's privilege, maintained by the leading legal-support organization for American journalists. The most practical starting point for anyone who needs to know the law of a particular state. Uniform Law Commission, Uniform Public Expression Protection Act (2020), with official comments.** The model statute and the drafters' explanations of its choices, including the prima facie standard adopted to avoid the jury-trial problems that doomed earlier statutes. Directive (EU) 2024/1069 on protecting persons who engage in public participation from manifestly unfounded claims or abusive court proceedings. The text of the European Union's anti-SLAPP directive, readily available from the EU's official legal database. Its recitals are a clear statement of the European rationale for procedural protection. House of Commons Library, The impact of strategic lawsuits against public participation (SLAPPs), Research Briefing CBP-10955 (2026). A concise, neutral account of the development of SLAPP law in the United Kingdom, including the economic-crime provisions of the 2023 Act and the arguments for broader legislation. Hashtags: #MediaLaw #DefamationLaw #LibelLaw #LibelTourism #AntiSLAPP #AntiSLAPPLegislation #StrategicLawsuits #FreedomOfSpeech #FreedomOfExpression #PressFreedom #FirstAmendment #MediaRights #JournalismLaw #ActualMalice #NewYorkTimesVSullivan #PublicFigureDoctrine #ReputationProtection #FreeSpeechProtection #DefamationLitigation #MediaLiability #CrossBorderDefamation #InternationalDefamationLaw #SPEECHAct #UKDefamationAct #EUSLAPPDirective #ReporterPrivilege #JournalisticSources #ShieldLaws #OnlineDefamation #LegalProtectionForJournalists

Download the Book (PDF): Introduction Type the name of a famous scientist into a search engine and, alongside the list of links, a panel appears. It gives a birth date, a death date, a photograph, a spouse, the institutions where the person worked, the prizes they won, and a short row of "people also search for" names. None of this is a document. Nobody wrote that panel as a page. It is assembled on the fly from a store of facts about a thing the search engine has decided exists, a thing with a stable identifier that is distinct from every other thing that happens to share its name. The panel is the visible tip of a knowledge graph. The same structure, under other names, sits behind a good deal of modern computing. Banks use graphs of companies, subsidiaries and directors to find who ultimately controls an account. Pharmaceutical researchers use graphs of genes, proteins, compounds, diseases and published findings to propose which existing drug might be repurposed for a new indication. Retailers use graphs of products, attributes and compatibility to answer "will this lens fit my camera?" Streaming services, logistics companies, intelligence agencies and hospital systems all keep, somewhere in their infrastructure, a network of named things joined by named relationships. And since the arrival of large language models, the knowledge graph has had a second career: as a way of keeping a fluent but unreliable text generator tied to facts that someone can check. This book is about that structure: what it is, how it is built, what it is good for, and where it fails. Its argument can be put in a sentence. The value of a knowledge graph lies not in the graph, the nodes and edges and the database that stores them, but in the explicit commitments it records: what kinds of things exist, which particular thing each mention refers to, how things relate, and where each claim came from. Those commitments are expensive to make and expensive to keep. They are also exactly what statistical language models lack. That is why the two technologies need each other, and why the hard work in this field has always been, and remains, the work of deciding and maintaining identity and meaning rather than the work of storing edges. Why identity is the hard part It is easy to underestimate the problem a knowledge graph solves, because human readers solve it without noticing. When a newspaper mentions "Jordan" in a story about a basketball contract, a peace summit and a river, a reader assigns the word to a person, a country and a body of water without effort. When an annual report refers to "the Company", "Acme", "Acme Holdings plc" and "our group", the reader understands that these are, depending on context, the same entity or deliberately different ones. When a medical record says "MI" a clinician reads myocardial infarction, while in a different record the same letters might mean mitral insufficiency. Software does none of this for free. A database that stores strings stores strings; "Paris" the city, "Paris" the Texas town, "Paris" the mythological prince and "Paris" the celebrity are, to it, the same five characters. Every system that tries to answer questions about the world must at some point decide what things there are and which words point at which things. Most systems make those decisions implicitly, in code, in column names, in the heads of the analysts who know which table to trust. A knowledge graph makes them explicitly, as data. Each thing gets an identifier. Each identifier gets a type. Each relationship between identifiers has a name with a defined meaning. Each fact can carry a record of where it came from and when it was true. Once those decisions are written down, a great deal follows. Facts from different sources can be merged because they are attached to the same identifier. Questions can be answered by following relationships rather than by matching words. Inconsistencies become visible: if one source says a company was founded in 1998 and another says 2003, the conflict is a property of a single node and can be flagged. And a machine can reason, in a modest way, from what is stated to what follows: if every subsidiary of a sanctioned entity is restricted, and this company is a subsidiary of that one, then this company is restricted. Why this matters more now For a decade the knowledge graph was largely an infrastructure concern, important to search engines and to a handful of data-rich industries, invisible to everyone else. Large language models changed its standing in two directions at once. First, they made extraction cheaper. Turning the text of contracts, papers, filings and web pages into structured facts used to require bespoke machine-learning pipelines trained for each type of relationship. A capable language model can now be asked, in plain words, to read a paragraph and list the companies, people and relationships it mentions in a given format. The output is imperfect, but it lowered the cost of building a first graph by something like an order of magnitude, and it put graph construction within reach of teams that would never have attempted it. Second, they made grounding urgent. Language models produce fluent, confident text whether or not it is true. They do not, in any robust sense, know which entity a name refers to; they have absorbed statistical associations among words, which usually line up with facts and sometimes do not. They cannot say where a claim came from. They do not reliably distinguish what was true in 2019 from what is true now. Organisations that want to put such models in front of customers, regulators or clinicians need a way to tie generated answers to sources of record. Retrieval of documents is one answer. Retrieval of structured, identified, sourced facts is another, and for many questions a better one. These two developments pull in opposite directions, and much of the interest of the present moment lies in the tension. The same models that make graphs cheaper to build are unreliable enough that a graph built carelessly with them inherits their errors, then lends those errors the authority of structured data. A graph used to ground a model is only as good as the identity and provenance decisions underneath it. If those decisions were made by the model itself, the grounding is partly circular. What this book covers The chapters run from foundations to practice. The first explains what a knowledge graph is, distinguishes the two main technical traditions (the standards-based semantic web and the property graph databases popular with developers), and traces how the idea moved from academic project to the infrastructure of web search. The second deals with ontology design: how to decide what kinds of things and relationships a graph will recognise, and why that design work determines almost everything downstream. The third and fourth chapters cover construction. Chapter 3 is about extracting entities and relationships from unstructured text, from hand-written patterns through supervised learning to the current generation of language-model extraction, and about how to judge whether an extraction pipeline is good enough. Chapter 4 is about the problem this book regards as central: entity resolution and entity linking, the work of deciding that two records or two mentions refer to the same real thing, and attaching them to one identifier. The fifth and sixth chapters look at the two large settings where knowledge graphs earn their keep: web search, with its public graphs and structured markup, and the enterprise, where graphs integrate data that was never designed to fit together. Chapter 7 addresses language models directly: how graphs are used to ground generation, what the evidence says about when this helps, and where it does not. Chapter 8 deals with what happens after launch: quality, provenance, time, and the governance that keeps a graph from decaying into a confident record of things that were once true or were never true at all. The book is written for readers who are technically curious but not specialists: engineers and data scientists meeting graphs for the first time, product and data leaders deciding whether to invest in one, and anyone who wants to understand why search engines and enterprise AI systems behave as they do. It assumes no knowledge of formal logic or graph theory. Where a technical term is needed it is explained in place. What this book does not do A short book has to choose. This one does not teach a query language in detail, does not survey graph database products, and does not cover the mathematics of graph neural networks, though each of these is mentioned where it bears on the argument. It treats knowledge graph embeddings, the techniques that turn nodes and edges into vectors for machine learning, only insofar as they matter for completing and checking graphs. It also does not pretend that a knowledge graph is the right answer to every data problem. Several chapters spend time on cases where it is not, because the most common way to waste money on this technology is to build a graph to answer questions a well-designed table already answers. The thread running through all of it is the one stated at the start. Storage is the easy part. Deciding what exists, what is the same as what, and what is true according to whom: that is the work, and it is human work that machines can assist but not replace. Chapter 1: Things, Not Strings On 16 May 2012 Google announced a change to its search engine under a slogan that has since become the shorthand for the whole field: "things, not strings." The company said its new Knowledge Graph already contained more than 500 million objects and more than 3.5 billion facts about and relationships between them. The point of the slogan was that a search engine had, until then, matched the characters a user typed against the characters in documents. From now on it would also try to work out which thing in the world the user meant, and answer with facts about that thing. The idea was not new in 2012. It had a long history in artificial intelligence, in library science, and in the semantic web movement of the early 2000s. What was new was scale, commercial commitment and visibility. Within a few years every major search engine, and soon most large technology companies, had something they called a knowledge graph. The term spread faster than any agreed definition of it, and today it is applied to everything from a carefully curated scientific database to a loose network of text snippets an algorithm has linked together. Before going further it is worth being precise about what the term should mean. Statements about identified things At its simplest a knowledge graph is a collection of statements, each of which connects two things by a named relationship. "Marie Curie — award received — Nobel Prize in Physics." "Nobel Prize in Physics — conferred by — Royal Swedish Academy of Sciences." "Marie Curie — spouse — Pierre Curie." Each statement has three parts: a subject, a predicate (the relationship), and an object. For this reason the basic unit is usually called a triple. Draw every subject and object as a point and every predicate as a labelled arrow between points, and you have a graph in the mathematical sense: nodes joined by edges. That much describes a data structure, and data structures are not very interesting on their own. Three further features turn a pile of triples into a knowledge graph. The first is identity. Each node stands for one particular thing and has an identifier that does not depend on its name. In Wikidata, the large collaborative graph maintained by the Wikimedia movement, Marie Curie is Q7186. The label "Marie Curie" is attached to Q7186 as a property, alongside "Maria Skłodowska-Curie" and the forms of her name in hundreds of languages. If another person named Marie Curie exists in the graph, she has a different identifier. The identifier is the thing; names are just facts about it. The second is typing. Nodes belong to classes (Q7186 is an instance of "human"), relationships have defined meanings, and the classes and relationships are themselves described: "human" is a subclass of "person", "award received" is expected to link a person or organisation to an award. This layer of description is the ontology or schema, and it is the subject of the next chapter. Without it a graph is a set of arrows whose labels mean whatever each contributor thought they meant. The third is context. Real facts are rarely timeless and never sourceless. Marie Curie received the physics prize in 1903, shared with Pierre Curie and Henri Becquerel; the claim is supported by the Nobel Foundation's own records. A mature knowledge graph can attach such qualifiers and references to individual statements, so that a consumer can ask not only "what is true?" but "when, according to whom, and how sure?" Putting these together, a reasonable working definition is this: a knowledge graph is a body of statements about identified entities, organised by an explicit model of the kinds of entities and relationships that exist, with enough context attached to the statements to judge when and why each holds. The definition is deliberately demanding. Many systems marketed as knowledge graphs have the first feature and little of the other two, and much of what follows in this book concerns the consequences. Two technical traditions Anyone who starts building a knowledge graph quickly finds that the field has two dialects, which grew up in different communities and still use different vocabularies for overlapping ideas. The first is the semantic web tradition, standardised by the World Wide Web Consortium. Its data model is RDF, the Resource Description Framework, in which every subject and predicate (and most objects) is a web address, an IRI, that acts as a globally unique identifier. The idea, set out by Tim Berners-Lee, James Hendler and Ora Lassila in a 2001 Scientific American article, was that data published on the web could link to data published elsewhere simply by reusing each other's identifiers, building a web of data alongside the web of documents. RDF comes with a family of companion standards: RDFS and OWL for describing classes and properties and their logical relationships, SPARQL for querying, SHACL for validating that data conforms to expected shapes, and SKOS for thesauri and controlled vocabularies. Systems that store RDF are called triplestores. The strengths of this tradition are interoperability and formality. Because identifiers are global, two organisations can merge their graphs without negotiating a mapping, provided they used the same identifiers for the same things. Because the schema languages have formal semantics, a reasoner can derive new facts that follow logically from stated ones, and can detect certain contradictions. Large public graphs such as DBpedia, which extracts structured data from Wikipedia, and many scientific resources in the life sciences are published in RDF. Wikidata, though it has its own internal data model, offers a SPARQL query service over an RDF export. The weaknesses are equally real. The standards are layered and their documentation assumes comfort with formal logic. Attaching information to a statement, as opposed to a node, has historically been awkward in RDF; the original mechanism, reification, multiplied a single fact into several triples, and the community has spent years on cleaner alternatives, culminating in the RDF 1.2 work on quoted triples. Many developers found the whole stack heavy for the problems they actually had. The second tradition is the property graph, popularised by graph databases such as Neo4j from the late 2000s. In a property graph, nodes and edges both carry labels and arbitrary key-value properties. An edge from a person to a company labelled WORKS_AT can carry a start date, a job title and a source directly on itself. Identifiers are local to the database rather than global web addresses. Queries are written in languages designed for traversing paths, of which Cypher became the most widely used; a pattern such as "find people who work at a company that supplies a company I own" reads almost like a sketch of the path. In April 2024 the International Organization for Standardization published GQL (ISO/IEC 39075), the first ISO standard database language since SQL, which gives property graphs a common query language heavily influenced by Cypher. Property graphs appeal to software engineers because they map naturally onto application data and are easy to start with. Their weakness is the mirror image of RDF's strength: with no global identifiers and, often, no formal schema, two property graphs built by different teams do not merge by themselves, and meaning lives in conventions rather than in machine-readable definitions. Many property graph deployments add schema constraints and naming rules to compensate. The two traditions have been converging for years. Products increasingly support both models, mappings between them are well understood, and the arguments between their advocates are less heated than they were. The choice still matters, but it matters less than the modelling and identity decisions that either one must record. Table 1 summarises the practical differences, with the relational database included as the baseline that most organisations already have. Table 1. Three ways to store connected data. Feature Relational database RDF triplestore Property graph Basic unit Row in a table Triple (subject, predicate, object) Node or edge with properties Identifiers Local keys Global IRIs Local IDs Schema Fixed tables, enforced Ontology (RDFS, OWL), validated with SHACL Optional, often by convention Facts about facts Extra columns or tables Reification or RDF 1.2 quoted triples Properties on edges Typical query language SQL SPARQL Cypher, GQL Strongest at Known, stable questions Integration across sources, inference Path traversal, developer speed A small graph, worked through It helps to see these ideas operate on a small scale. Imagine a regional bank that wants to know whether any of its business customers are connected to a company that has just been placed under sanctions. Its data includes a customer list, a feed from the national company register, and a list of sanctioned entities published by the government. In a knowledge graph, each company, person and address becomes a node with its own identifier. The company register contributes statements such as "Harbour Logistics Ltd — has director — Elena Marsh" and "Harbour Logistics Ltd — registered address — 14 Quay Street" and "Harbour Logistics Ltd — shareholder — Northgate Holdings SA, 60 percent." The bank's own data contributes "Customer account 30071 — held by — Harbour Logistics Ltd." The sanctions list contributes "Northgate Holdings SA — subject of — sanctions designation, dated this week." Each statement on its own is unremarkable, and each comes from a different system that knows nothing about the others. Joined at shared nodes, they form a path: account 30071 is held by Harbour Logistics, which is 60 percent owned by Northgate Holdings, which is sanctioned. A query that asks for every customer account within three ownership steps of a sanctioned entity will find it. So will a query asking which other companies share Elena Marsh as a director, or share the address on Quay Street, which is how investigators look for networks of related shell companies. Notice what made this possible. It was not the graph database; the same query could be written, more laboriously, against tables. It was the fact that "Harbour Logistics Ltd" in the bank's customer file, "HARBOUR LOGISTICS LIMITED" in the register and the counterparty named in a payment instruction were all attached to the same node. If they had not been, the path would have been broken at its first link and the query would have returned nothing, a silence that looks exactly like a clean result. The value of the small graph lies almost entirely in that identity decision, which is the point this book will keep returning to. The example also shows why context matters. The ownership statement needs a date, because Northgate may have sold its stake last year, and a source, because the register and a commercial data provider may disagree. A graph that records only "Harbour Logistics — shareholder — Northgate" cannot tell an investigator whether the link is current or whether to believe it. Why not just use tables? A natural objection is that every one of these structures can be emulated in an ordinary relational database. A table with three columns holds triples perfectly well. Why does the graph framing matter? There are three good answers and one bad one. The bad answer is performance mythology: the claim that graph databases are always faster for connected data. Sometimes they are, particularly for queries that follow an unknown number of hops, such as "find every company ultimately controlled by this person through any chain of ownership." Relational databases handle such recursive queries less gracefully. But for many workloads a well-indexed relational store is as fast or faster, and the choice of storage engine is rarely what determines whether a knowledge graph project succeeds. The first good answer is schema flexibility. A relational schema fixes in advance which attributes each kind of thing has. Real-world entities do not cooperate: some people have three nationalities, some companies have no fixed address, a new regulatory requirement adds a relationship nobody anticipated. In a graph, adding a new kind of relationship means adding new edges, not migrating tables. This matters most when data comes from many sources with different shapes, which is exactly the integration problem knowledge graphs are usually deployed to solve. The second good answer is that relationships become first-class. In a relational design, the fact that a person is a director of a company is often implicit in a foreign key, and its meaning is documented, if at all, outside the database. In a graph the relationship is named, typed and queryable in itself. That makes it possible to ask questions about the structure of the data, such as which relationships connect two entities, without knowing in advance which tables to join. The third good answer, and the most important for this book's argument, is that the graph framing forces identity to the surface. Because every statement must point at nodes, someone has to decide which node each piece of data belongs to. In a collection of tables it is possible to postpone that decision forever, and many organisations do: the customer table, the account table and the marketing list each hold their own version of "the customer", and nobody reconciles them until a regulator asks. A knowledge graph does not make the reconciliation easy, but it makes it unavoidable. From projects to infrastructure The modern history of the field runs through a handful of projects that shaped how practitioners think. Cyc, begun by Douglas Lenat in 1984, attempted to hand-encode the common-sense knowledge an intelligent system would need, in a rich logical language. It demonstrated both the power of formal knowledge and the cost of producing it manually; decades of work produced millions of assertions and few mass-market applications. WordNet, developed at Princeton from the mid-1980s, organised English words into sets of synonyms linked by relationships such as "is a kind of" and "is a part of", and became one of the most widely used resources in natural language processing. Neither was called a knowledge graph at the time, but both established that structured knowledge about meaning could be built and used. The semantic web era of the 2000s produced the standards described above and the Linked Open Data movement, which encouraged publishers to expose datasets in RDF with links to each other. DBpedia (from 2007) and YAGO (also 2007, combining Wikipedia with WordNet's class hierarchy) became hubs of this network. Freebase, launched by the start-up Metaweb in 2007, took a more pragmatic approach: a collaboratively edited graph with a simpler data model and a friendly interface. Google acquired Metaweb in 2010, and Freebase became one of the seeds of the Google Knowledge Graph. Freebase itself was closed in 2016, with much of its content offered for migration to Wikidata. Wikidata, launched in 2012 by Wikimedia Deutschland, is now the largest openly editable knowledge graph and the most important public one. It began as a way to store facts shared across Wikipedia's language editions, so that a population figure updated once would update everywhere, and grew into a general-purpose graph of well over a hundred million items, edited by tens of thousands of volunteers and a great many automated tools. Its data model embodies the three features described earlier: every item has a language-independent identifier, properties are themselves defined items with documentation and constraints, and individual statements carry qualifiers (such as start and end dates) and references. Alongside these public efforts, companies built private graphs: Microsoft's for Bing, Amazon's product graph, LinkedIn's economic graph of people, jobs and skills, Facebook's social graph. In 2019 a group of researchers from several of these companies published an article in Communications of the ACM comparing their industry knowledge graphs. Their common observations were sobering and instructive: the graphs were enormous, never complete, built from many sources of uneven quality, and dominated in cost not by storage or query but by the work of entity resolution, schema evolution and quality control. What a graph is for It is worth closing this chapter with a plain statement of what knowledge graphs are good at, since the rest of the book will spend much of its time on their difficulties. They are good at integration: bringing facts about the same things from many sources into one place, keyed to shared identities. They are good at questions whose answers depend on relationships, especially chains of relationships of unknown length. They are good at explanation, because an answer derived by following edges can show the path it followed. They are good at serving facts to other systems, from search panels to recommendation engines to language models, in a form those systems can rely on. And they are good at making disagreement visible, because conflicting claims about the same thing end up attached to the same node. They are not good at holding nuance that resists being broken into discrete statements, such as the argument of a legal opinion or the reasoning in a scientific paper. They are not cheap to keep accurate. And they are only as reliable as the decisions about identity and meaning on which they rest. Those decisions start with the ontology. Chapter 2: Deciding What Exists Every knowledge graph rests on a set of decisions about what kinds of things exist and how they can relate. Is a hospital a building, an organisation, or both? Is a "customer" a person, an account, or a relationship between a person and a company? Does a drug treat a disease, or is the treatment relationship really a claim made in a particular trial about a particular population? The written form of these decisions is called an ontology, or, in less formal settings, a schema or data model. The word comes from philosophy, where it means the study of what exists. In computing it means something humbler and more practical: an explicit, shared specification of the categories and relationships a system will use. Ontology design is the least glamorous part of building a knowledge graph and the most consequential. Extraction pipelines can be retrained, entity resolution can be rerun, storage engines can be swapped. A poor ontology, once data has been loaded against it and applications have been built on top of it, is very hard to change. It determines what questions the graph can answer, which sources can be integrated without distortion, and whether two teams using the graph mean the same thing when they use the same word. The building blocks An ontology has a small number of parts, whatever language it is written in. Classes are the kinds of things: Person, Organisation, Drug, Disease, Product, Contract. Classes are usually arranged in a hierarchy, so that Pharmaceutical Company is a subclass of Company, which is a subclass of Organisation. Membership is inherited upward: anything that is a pharmaceutical company is also a company and an organisation, and any rule that applies to organisations applies to it. Properties, or relationships, connect things. Some connect two entities (employs, located in, treats, supplies). Others connect an entity to a literal value such as a date, a number or a string (date of birth, revenue, chemical formula). Properties can be given a domain and a range, which state what kinds of things they are expected to connect: "employs" runs from an Organisation to a Person. Properties can also be arranged in hierarchies ("chief executive of" is a more specific form of "employed by") and given logical characteristics, such as being symmetric (if A is married to B, B is married to A), transitive (if A is part of B and B is part of C, A is part of C), or inverse to another property (subsidiary of and parent of). Individuals, or instances, are the particular things: this company, this drug, this person. Strictly they belong to the graph's data rather than its ontology, but the boundary is blurrier than it sounds, and deciding whether something is a class or an individual is one of the classic modelling problems. Finally, constraints and rules state what must or must not hold: every person has at most one date of birth; every drug in the formulary must have an active ingredient; nothing can be both a person and an organisation. In the semantic web stack, OWL expresses many of these as logical axioms that a reasoner can use, and SHACL expresses them as shapes against which data can be validated. In property graph systems they are typically enforced by database constraints and application code. Two jobs an ontology does It helps to see that an ontology serves two different purposes, which pull in somewhat different directions. The first is communication. An ontology is a contract among the people and systems that produce and consume a graph's data. When one team loads "supplier" relationships and another queries them, both need to agree on what counts: a company that has delivered goods in the past year, any company with a signed master agreement, or any company appearing in the procurement system? Written definitions, examples and scope notes carry this agreement. Much of the practical value of an ontology lies in this documentation, not in its formal logic. The second is computation. An ontology lets software check data, infer new facts and integrate sources automatically. If the ontology says that "headquartered in" runs from an organisation to a place, a validator can reject a statement that a person is headquartered somewhere. If it says that "subsidiary of" is transitive, a query for all subsidiaries of a parent can return indirect ones without anyone writing the recursion. These purposes diverge when it comes to formality. Formal axioms enable automated checking and inference, but they are hard for most practitioners to read, easy to get subtly wrong, and can make reasoning slow or produce surprising conclusions. A famous class of error arises when a modeller uses a domain declaration as if it were a constraint. In OWL, declaring that the domain of "employs" is Organisation does not forbid a person from employing someone; it licenses the reasoner to conclude that anyone who employs someone is an organisation. A graph with a sole trader in it will then quietly classify that person as an organisation. The lesson is not that formal semantics are bad, but that formality should be adopted deliberately, where its consequences are understood, rather than by default. Reuse before you build The single most valuable rule in ontology design is to reuse existing vocabularies wherever they fit. A great deal of careful modelling already exists, and adopting it brings interoperability with other datasets as well as saving effort. For general web content, the dominant vocabulary is schema.org, founded in 2011 by Google, Microsoft (Bing), Yahoo and later joined by Yandex. It defines hundreds of types (Person, Organization, Event, Product, Recipe, MedicalCondition and many more) and properties for describing them. Its design is deliberately pragmatic: definitions are informal, the hierarchy is shallow, and properties are permissive about what values they accept. Chapter 5 discusses its role in search. For many enterprise graphs, schema.org is a reasonable starting point for common entities such as people, organisations, places and events. Specialised domains have far richer resources. In finance, the Financial Industry Business Ontology (FIBO), maintained by the EDM Council, models legal entities, ownership and control, securities, loans and derivatives in formal detail. In the life sciences, the Gene Ontology describes the functions of gene products, SNOMED CT provides a clinical terminology with hundreds of thousands of concepts used in electronic health records, and the OBO Foundry coordinates a family of biomedical ontologies built to common principles. Libraries and cultural heritage institutions use CIDOC CRM and various bibliographic models. Governments publish vocabularies for statistical data, organisational structures and public services. Reuse has its own hazards. Large standard ontologies can be far more detailed than an application needs, and adopting them wholesale imports complexity and commitments nobody on the team understands. The usual practice is to reuse selectively: adopt the identifiers and definitions for the classes and properties that are actually needed, map local terms to standard ones where the meanings genuinely coincide, and extend with local terms where they do not. What should be avoided is the opposite pattern, inventing a private term for something a well-known vocabulary already defines well, which guarantees later mapping work. How to decide: competency questions Ontologies go wrong most often not through technical error but through lack of focus. A team sets out to "model the business" and produces an elaborate hierarchy of everything the business touches, most of which no application ever queries. Or the ontology is designed around the shape of the first data source loaded, and the second source does not fit. The most reliable discipline for avoiding this was proposed in the 1990s by Michael Grüninger and Mark Fox: start from competency questions. These are concrete questions, phrased in ordinary language, that the knowledge graph must be able to answer. "Which of our suppliers have a parent company in a country under export controls?" "Which approved drugs share a molecular target with this compound?" "Which customers hold products that will be affected by this rate change?" Each question implies classes, properties and constraints. Collecting twenty or thirty of them from the people who will actually use the graph, and checking that the draft ontology can express the answer to each, keeps the design anchored to need. Competency questions also expose hidden ambiguity early. Ask a compliance team and a sales team what "customer" means and the questions they want answered will reveal that they mean different things. The ontology can then model the difference explicitly, perhaps with separate classes for a legal party, an account and a commercial relationship, instead of papering over it with one overloaded term that later breaks every report. From a question to a model Consider how a single competency question shapes a design. A manufacturer's risk team asks: "Which of our finished products depend, through any number of supply tiers, on a component made in a facility located in a region affected by a given disruption?" Read closely, the question names at least five classes: Product, Component, Supplier, Facility and Region. It implies relationships: a product contains components; a component is made by a supplier; the supplier operates facilities; facilities are located in regions. It also implies that "contains" can nest, since components contain sub-components, so the relationship must be traversable to any depth, which in OWL terms suggests a transitive property and in a property graph suggests a variable-length path query. And it implies a distinction that the purchasing system may not make: between the supplier as a legal entity that signs contracts and the facility as a physical place where things are made. Many purchasing systems record only the former. A model that conflates the two will answer the question wrongly whenever a supplier has plants in several regions, which most large suppliers do. The same question also reveals what does not need to be modelled yet. Nothing in it requires prices, delivery schedules, contract terms or the internal structure of the supplier's organisation. Those may matter for other questions later, and the ontology can grow to accommodate them; but building them in now, before anyone has asked for them, adds work and invites inconsistency. A dozen such questions, each analysed this way, will typically produce a compact ontology of a few dozen classes and relationships that covers most of what the first applications need, which is a far better starting point than an elaborate model of everything. Classic modelling dilemmas A handful of design questions recur in almost every project. None has a universally right answer, but each has consequences that should be chosen rather than stumbled into. The first is whether something should be a class or an individual. Is "Aspirin" a class, whose instances are particular tablets, or an individual drug? Is "Boeing 737" a class of aircraft or an individual product model? The answer depends on the questions to be asked. A hospital pharmacy tracking batches needs aspirin as a class with individual lots as instances; a drug-interaction database needs aspirin as an individual. Some graphs need both, and modelling patterns exist (sometimes called punning, or separating a product model from its physical instances) to handle it. The second is whether a relationship should be an edge or a node. Simple relationships are edges. But "Alice was chief financial officer of Acme from 2015 to 2021, according to Acme's annual report" is more than an edge; it carries dates, a role and a source. One option is to attach these as properties of the edge, which property graphs allow directly and RDF 1.2 now supports more cleanly. Another is to turn the relationship into a node of its own, an Employment or Position, which then links to the person, the organisation and the role. This is sometimes called reification or the n-ary relation pattern. The node approach is heavier but more expressive, and it is the right choice whenever a relationship has a life of its own: when it can be amended, disputed, or connected to other things. The third is how to model time. Most facts change. Companies change names and owners, people change jobs, borders move, drug approvals are withdrawn. A graph that stores only current facts cannot answer historical questions and cannot explain why a past decision was made. A graph that stores every fact with validity intervals is more truthful but harder to query, because every question must specify "as of when." Wikidata handles this with start-time and end-time qualifiers on statements; financial graphs often keep full temporal history because regulators require it. The design decision is how much history the competency questions actually require. The fourth is the treatment of uncertainty and disagreement. If two sources disagree on a company's founding date, does the graph store one value, both values with their sources, or a value with a confidence score? Storing one value is simple and lossy. Storing all claims with provenance, and letting consumers choose a policy for resolving them, is more honest and is the approach taken by the most robust public graphs. Chapter 8 returns to this. The fifth is granularity. A graph of corporate relationships might say that one company "owns" another. A more detailed one distinguishes direct shareholding, indirect ownership through intermediaries, voting control, and beneficial ownership, each with percentages. Finer distinctions answer more questions but cost more to populate and are more likely to be filled inconsistently. A useful rule is to model at the level of detail that the best available source supports, and no finer: an ontology that distinguishes eight kinds of control is worthless if every source simply says "owned by." Taxonomies, thesauri and the light end of the spectrum Not every knowledge organisation problem needs a full ontology. Many organisations' most useful structured knowledge is a taxonomy: a hierarchy of topics, product categories or skills used to tag and navigate content. A thesaurus adds synonyms and related-term links. These lighter structures are often published using SKOS, the W3C's Simple Knowledge Organization System, which deliberately avoids the logical commitments of OWL: a SKOS "broader" link says only that one concept is more general than another, not that every instance of the narrower is an instance of the broader. The distinction matters because teams frequently confuse the two. A product taxonomy that files "Laptop bags" under "Laptops" is perfectly sensible for navigation; a shopper looking at laptops may want bags. As an ontology class hierarchy it is nonsense, because a laptop bag is not a kind of laptop, and any reasoning over it will produce absurdities. Many real knowledge graphs contain both kinds of structure. The discipline is to keep them separate and use each for its purpose: taxonomies to organise and tag, ontologies to define what things are. Ontologies as living documents The final point about ontology design is that it is never finished. The 2019 comparison of industry knowledge graphs mentioned in Chapter 1 singled out schema evolution as one of the persistent difficulties: new kinds of entities appear, definitions turn out to be ambiguous, two classes are discovered to overlap, and every change must be propagated through data that was loaded under the old rules. Organisations that manage this well treat the ontology like a published interface. Changes are proposed, reviewed and versioned. Deprecated terms are kept with pointers to their replacements, rather than deleted, so that old data and old queries do not silently break. Each class and property has an owner who can answer questions about its meaning. And the competency questions are kept as a test suite: when the ontology changes, someone checks that the questions it was built to answer can still be answered. This is governance, and it is unglamorous. It is also the reason some knowledge graphs remain useful for a decade while others become, within two years, a store of data whose meaning nobody can confidently explain. The ontology is where the graph's commitments about what exists are written down. The next step is populating it, and for most organisations most of the relevant knowledge is not in structured databases at all but in text. Chapter 3: Reading Facts Out of Text Most of what organisations know is written down rather than tabulated. Contracts describe obligations between parties. Scientific papers report that a compound inhibits an enzyme. News articles report that one company has acquired another, that an executive has resigned, that a factory has closed. Clinical notes record symptoms, diagnoses and prescriptions in a clinician's shorthand. Regulatory filings list subsidiaries, directors and risks. If a knowledge graph is to reflect what an organisation, or the world, actually knows, it has to be able to read. The task of turning unstructured text into structured statements is called information extraction. It has been studied for more than three decades, and its methods have passed through several generations, each of which made the problem cheaper without making it go away. This chapter explains what the task involves, how the methods developed, what language models changed, and how to decide whether an extraction pipeline is trustworthy enough to feed a graph. The parts of the task Extracting facts from a sentence such as "In March, Northwind Pharmaceuticals completed its acquisition of the Swiss biotech Alpenbio for $1.2 billion" involves several distinct subtasks, which older systems performed in separate stages and newer ones often combine. Named entity recognition finds the spans of text that refer to entities and assigns each a broad type: "Northwind Pharmaceuticals" is an organisation, "Alpenbio" is an organisation, "Swiss" indicates a location, "$1.2 billion" is a monetary amount, "March" is a date. Relation extraction decides which relationships the text asserts between the entities found: Northwind acquired Alpenbio; Alpenbio is located in Switzerland. The relationships are normally drawn from a fixed set defined by the ontology, so this step is in effect a classification: for each pair of entities, which, if any, of the known relationships does this sentence express? Event extraction goes further, recognising that the sentence describes an event (an acquisition) with participants in roles (acquirer, target), a time, and attributes (price). Events are naturally modelled as nodes, following the pattern described in the previous chapter for relationships with a life of their own. Coreference resolution links mentions within a document that refer to the same thing. If the next sentence says "The company said the deal would close its pipeline gap," something must decide that "the company" is Northwind rather than Alpenbio. Normalisation converts values to standard forms: "March" to a full date using the article's publication date, "$1.2 billion" to a number and currency, "Swiss" to the country Switzerland. And finally entity linking, the subject of the next chapter, attaches each mention to a specific node in the graph: this "Northwind Pharmaceuticals" is the company with identifier such-and-such, not a similarly named firm elsewhere. Each stage can fail, and errors compound. A pipeline whose stages are each 90 percent accurate can easily produce correct complete facts much less than 90 percent of the time. This arithmetic, more than any single technique, is what makes extraction hard. Four generations of method The first generation relied on hand-written rules. In 1992 the computational linguist Marti Hearst showed that simple lexical patterns reliably signal certain relationships: "X such as Y", "Y and other X", "X, including Y" all suggest that Y is a kind of X. The insight generalised. Through the 1990s, the US government's Message Understanding Conferences drove the development of systems that filled templates (about terrorist incidents, corporate management changes, satellite launches) using carefully engineered patterns. Rule-based extraction is precise when the patterns match and silent when they do not; it cannot handle the endless variety of ways people phrase things. But it remains useful today for well-structured text such as regulatory filings, where the phrasing is formulaic, and as a high-precision supplement to statistical methods. The second generation used supervised machine learning. Annotators labelled thousands of sentences with entities and relationships, and statistical models learned to generalise from the examples. This greatly improved coverage but created a new bottleneck: labelled data. Every new relationship type needed new annotation, and annotation by trained people is slow and expensive. Two ideas tried to relieve the bottleneck. Distant supervision, introduced by Mike Mintz and colleagues in 2009, used an existing knowledge base (Freebase) as a source of noisy labels: if the knowledge base says a person was born in a city, then any sentence mentioning both is assumed to express that relationship. The assumption is often wrong, but at scale the noise can be partly averaged out. Weak supervision frameworks, such as Stanford's Snorkel, generalised the idea, letting engineers write many rough labelling heuristics whose disagreements a model then reconciles. Meanwhile, open information extraction systems, beginning with TextRunner at the University of Washington in 2007, abandoned a fixed set of relationships altogether and extracted any subject-verb-object pattern from web text, trading precision and consistency for breadth. The third generation used pretrained neural language models. From 2018, models such as BERT, trained on vast amounts of text before being fine-tuned on a specific task, raised the accuracy of entity recognition and relation classification substantially and reduced the amount of labelled data needed. Sequence-to-sequence models went a step further, treating extraction as translation from a sentence into a linearised list of triples, which allowed a single model to perform recognition and relation extraction together. The fourth generation, current since roughly 2023, uses large general-purpose language models instructed in natural language. Rather than training a model for each relationship, an engineer describes the ontology and the desired output format in a prompt, perhaps with a few examples, and asks the model to read a passage and return the entities and relationships it finds. Modern models can return structured output conforming to a supplied schema, which removes most of the parsing problems that plagued early attempts. The trade-offs among these generations are summarised in Table 2. The pattern is clear: each step reduced the up-front cost of building an extractor and increased its flexibility, while making its behaviour harder to predict and audit. Table 2. Generations of information extraction. Approach What it needs Strengths Weaknesses Hand-written rules Expert time per pattern Precise, auditable, cheap to run Brittle, low coverage Supervised and weakly supervised models Labelled or heuristically labelled examples Good accuracy on trained relations Labelling cost; fixed relation set Fine-tuned pretrained transformers Modest labelled data, GPU training High accuracy, joint extraction Retraining for new relations Prompted large language models A schema description and examples Fast to start, flexible, handles new relations Cost at scale, inconsistency, invented facts What large language models changed, and what they did not The practical effect of prompted extraction is hard to overstate. A team that would once have spent months annotating data and training a relation classifier can produce a working first draft of a graph from a document collection in days. The models handle paraphrase, long-distance dependencies and implicit relationships that defeated earlier systems. They can be asked to extract new relationship types by editing a prompt. They can perform several subtasks at once: finding entities, resolving pronouns, normalising dates and assigning relationships in one pass. But several problems persist or take new forms. The first is invention. A language model asked to list the relationships in a passage will sometimes list relationships that are plausible but not stated. Told that a passage mentions a chief executive and a company, it may assert an employment relationship the text never makes; given a drug and a disease mentioned in the same paragraph, it may assert a treatment relationship when the text was describing a side effect. Older extractors made errors too, but their errors were mostly omissions or misclassifications of what was present. A generative model can produce statements with no textual basis at all. The defence is to require the model to quote the supporting span for each extracted fact and then to check, mechanically, that the quoted span exists in the source and contains the entities named. The second is inconsistency. The same model given the same passage twice may return different results, or label the same relationship with different predicates in different documents. At small scale this is a nuisance; across millions of documents it produces a graph in which the same fact is represented several ways. Constraining output to a fixed schema, setting generation parameters for determinism where the provider allows it, and normalising predicates after extraction all help. The third is ontology drift. If the prompt invites the model to propose new relationship types, it will, and they will be overlapping and inconsistent: "acquired", "purchased", "bought", "completed acquisition of", "took over". Open extraction of this kind is useful for exploration, but a graph meant for reliable querying needs extraction tied to the controlled vocabulary of the ontology, with a separate, human-reviewed process for adding new terms. The fourth is cost and throughput. Running a large model over every sentence of a large corpus is expensive. Many production systems use a cascade: cheap filters or small fine-tuned models to find passages likely to contain relevant facts, and a large model only for those. Some teams use a large model to generate training data for a smaller, faster model that then runs at scale, an old idea from the weak-supervision era in a new form. The fifth problem is the most basic: extraction is not linking. A model that extracts "Northwind Pharmaceuticals acquired Alpenbio" has produced a triple of strings, not a statement about identified things. Until each string is attached to the right node, the triple cannot be merged with anything else the graph knows. Language models can help with linking, as Chapter 4 explains, but it is a distinct problem with its own failure modes, and pipelines that skip it produce what might be called a graph of names rather than a graph of things. Text that is not quite prose Much of the text that organisations want to extract from is not flowing prose at all. Annual reports list subsidiaries in tables spanning several pages. Scientific papers put their key results in tables and figure legends. Contracts are organised into numbered clauses whose meaning depends on definitions given pages earlier ("the Supplier", "the Effective Date"). Invoices, purchase orders and lab reports follow templates. Product listings mix a title, a list of bullet-point features and a table of specifications. Each form needs its own handling. Tables need to be recovered with their structure intact before extraction, since a model that sees a flattened table as a stream of words will misalign rows and columns. Documents scanned from paper need optical character recognition first, and its errors propagate. Defined terms in contracts need to be resolved to the parties they stand for before any relationship can be attached to the right entity. The last few years have brought much better tools for document layout analysis, and multimodal language models can read page images directly, including tables, which has removed some of these steps. But the general lesson holds: a large share of extraction errors in practice come not from misunderstanding language but from misreading the document's structure, and time spent getting the structure right is rarely wasted. A worked case shows how the stages combine. Suppose a team wants a graph of corporate events (acquisitions, executive appointments, plant closures) from a business news archive. A practical pipeline might first classify articles to find those likely to report such events, using a small, cheap model. It would then run a larger model over the relevant articles with a prompt that lists the event types and roles the ontology defines, asks for structured output, and requires a quoted supporting sentence for each event. A mechanical check would discard any event whose quotation does not appear in the article. Dates would be normalised against the article's publication date. Entity mentions would be passed to a linking stage. And events reported by several articles would be merged into one, with all the supporting articles recorded, which also provides a useful signal: an acquisition reported by twenty outlets is far more likely to be real than one reported by a single blog. None of these steps is exotic. Together they turn a capable but unreliable reader into a pipeline whose output can be measured and trusted to a known degree. Evaluating an extraction pipeline Whether an extraction pipeline is good enough depends on what the graph will be used for, and that judgement requires measurement. The standard measures are precision (of the facts extracted, what fraction are correct) and recall (of the facts actually stated in the text, what fraction were extracted). Academic benchmarks such as the ACE corpora, the TAC knowledge base population tasks and TACRED provide labelled data for comparing systems, but they rarely match the domain, document types and ontology of a real project. Published benchmark scores should be read as rough indications of relative capability, not as predictions of performance on one's own documents. The only reliable evaluation is a sample of one's own documents annotated by people who understand the domain. A few hundred carefully annotated passages, covering the main document types and the most important relationships, are usually enough to estimate precision and recall to within a few percentage points and, more usefully, to see what kinds of errors the system makes. Error analysis is often more valuable than the headline numbers. A pipeline that misses relationships expressed in tables, or confuses parent and subsidiary when the sentence is in the passive voice, or invents relationships when two entities appear in a list, has a specific defect that can be addressed. The right balance between precision and recall depends on the use. A graph that will drive automated decisions, such as blocking a payment to a sanctioned entity's subsidiary, needs very high precision on the relevant relationships, and is better off missing a fact than asserting a false one, provided misses are caught elsewhere. A graph meant to help analysts discover leads can tolerate lower precision in exchange for recall, since a human will check each lead. Many graphs serve both purposes, which argues for recording a confidence score and the extraction method on each fact, so that each consumer can apply its own threshold. Keeping the evidence The most important design decision in an extraction pipeline is easy to overlook: keep the link from every extracted fact back to the exact passage it came from. Store the document identifier, the location of the supporting span, the version of the extractor that produced it, and the time it was run. This evidence trail has several uses. It allows any fact in the graph to be checked by a person in seconds, which builds trust and makes errors cheap to correct. It allows facts to be re-extracted when a better model becomes available, and the results compared. It allows a fact to be withdrawn automatically when its source document is retracted or superseded. And, as Chapter 7 discusses, it allows a language model answering a question from the graph to cite its sources in a form a reader can verify. A graph without this trail is a set of assertions whose basis nobody can reconstruct. A graph with it is a structured index into the documents themselves, which is a much more defensible thing to put in front of a regulator, a clinician or a customer. Extraction, then, produces candidate statements with evidence. Turning those candidates into statements about the right things is the next and harder step. Chapter 4: Who Is Who Of all the problems involved in building a knowledge graph, the one that most determines its value is deciding when two pieces of information are about the same thing. A supplier database lists "Intl. Business Machines Corp." while a news feed mentions "IBM" and a contract names "International Business Machines Corporation (Armonk, NY)." A hospital system holds three records for a patient whose surname was misspelled once and changed by marriage once. A scientific database has two authors called "J. Wang" at the same university, and a paper that mentions one of them. Unless these are correctly merged or correctly kept apart, every downstream fact is attached to the wrong node, and every answer the graph gives inherits the error. The field has two names for two forms of this problem. Entity resolution (also called record linkage, deduplication or entity matching) works with structured records: given two databases, or one database with duplicates, which records describe the same real-world entity? Entity linking works with text: given a mention in a document, which node in the graph does it refer to, if any? The techniques overlap, and both come down to the same question of identity. This chapter treats them together and argues that they deserve more investment than any other stage of graph construction. Why identity is hard Identity looks simple from a distance and dissolves on contact. Consider some of the ways it goes wrong. Names vary. A single company can appear under its full legal name, an abbreviation, a former name, a trading name, a ticker symbol and a misspelling. A single person can appear with and without a middle initial, with a maiden or married surname, with a name transliterated differently from another script, with a nickname. Names collide. Many different people share common names; in some naming cultures a small number of surnames cover a large fraction of the population. Many different companies share generic names, and one corporate group may contain dozens of legal entities with nearly identical names in different countries ("Acme Ltd", "Acme GmbH", "Acme Holdings Ltd", "Acme UK Ltd"). A town called Springfield exists in many American states. Attributes conflict. Two records for the same person may have different addresses because one is out of date, different dates of birth because of a data entry error, different phone numbers because both are valid. Two records for different people may share an address because they live together. The right granularity is unclear. Is "Google" the same entity as "Alphabet Inc."? For a consumer search, near enough. For a regulatory filing, certainly not: Alphabet is the parent holding company, Google LLC a subsidiary. Is a hospital the same entity as the trust that operates it? Is a product sold under different names in different countries one product or several? These are partly ontology questions, as Chapter 2 discussed, and entity resolution cannot be done well until they are settled. And entities change. Companies merge, split and rename themselves. People move, change jobs and change names. A record that correctly matched an entity five years ago may describe a different situation today. The anatomy of entity resolution Record linkage has a long history in statistics and public administration. The term was used by Halbert Dunn in 1946 to describe assembling the records of an individual's life, and the formal model still at the heart of many systems was published by Ivan Fellegi and Alan Sunter in 1969. Their insight was to treat matching as a problem of weighing evidence: each field that agrees or disagrees between two records shifts the probability that they describe the same entity by an amount that depends on how informative that field is. Agreement on a rare surname is strong evidence; agreement on a common one is weak. Agreement on date of birth is strong; agreement on gender is weak. Modern entity resolution pipelines have four stages, whatever methods they use within each. The first is standardisation. Before comparison, values are cleaned into consistent forms: legal suffixes such as "Ltd", "Limited", "plc" and "Inc." normalised or separated from the name, addresses parsed into components, phone numbers put in international format, names split into given and family parts, characters transliterated consistently. This step is unglamorous and decisive; a large share of matching errors in practice come from inconsistent formatting that a few rules would have fixed. The second is blocking. Comparing every record with every other record grows with the square of the number of records; for ten million records that is about fifty trillion comparisons. Blocking reduces this by only comparing records that share some cheap key, such as the first three letters of the surname plus the postcode, or a phonetic code of the company name. Modern systems often block on several keys at once, or use approximate nearest-neighbour search over vector embeddings of the records, to reduce the chance that a true match is never compared. Blocking is a trade-off: aggressive blocking is fast but misses matches whose keys differ; generous blocking catches more but costs more. The third is pairwise matching. For each candidate pair produced by blocking, the system decides how likely it is that they refer to the same entity. Methods range from hand-weighted rules, through the probabilistic Fellegi-Sunter model, to machine-learning classifiers trained on labelled pairs, to neural models that compare text representations of the two records, to large language models asked directly whether two descriptions refer to the same entity. Each pair receives a score. The fourth is clustering. Pairwise scores must be turned into groups of records, each group representing one entity. This is harder than it sounds, because matching is not reliably transitive in practice: record A may match B and B match C, while A and C clearly do not match. A naive approach that merges everything connected by a match creates sprawling clusters in which a few weak links have chained together dozens of distinct entities. Good clustering algorithms consider the whole set of scores among a group and are willing to break weak links to keep clusters coherent. Entity linking: from mention to node Entity linking applies the same logic to mentions in text. Given the sentence "Jordan scored 32 points as the Bulls beat the Knicks," the system must decide that "Jordan" refers to a particular basketball player and not to the country, the river, or any of the many other people with that name, and that "the Bulls" means the Chicago team. The standard approach has two steps. Candidate generation retrieves a short list of nodes that the mention might refer to, typically using a dictionary of names and aliases collected from the graph itself, from anchor texts in Wikipedia (the words people use when linking to an article), and from previous linking decisions. Disambiguation then chooses among the candidates using context. Three kinds of evidence matter. Prior probability: some referents are simply more common, and an unqualified "Paris" is most often the French capital. Local context: words near the mention, here "scored", "points", "Bulls", favour a sports interpretation. And coherence: the entities mentioned together in a document tend to be related, so a document that also mentions the Knicks and Chicago is about the basketball player, and choosing the referents of all mentions jointly is more accurate than choosing each alone. The modern history of entity linking runs from "wikification" systems in the late 2000s, which linked words in text to Wikipedia articles, through systems such as TagMe and AIDA that exploited coherence among entities, to neural approaches in which both mentions in context and candidate entities are encoded as vectors and compared. Facebook's BLINK model (2020) showed that a dense retrieval approach could link to millions of entities with high accuracy, including entities never seen in training, using only their textual descriptions. Large language models can now perform linking directly, given a mention, its context and a list of candidates with descriptions. Two difficulties are specific to linking. The first is the absent entity: many mentions refer to things not yet in the graph, such as a small private company, an ordinary person, or a newly announced product. A linker that always picks the best available candidate will wrongly attach these to some similarly named node, creating false facts about a real entity. Good systems must be able to say "none of the above" and create a new placeholder node, which is in turn a candidate for later resolution. The second is that context is often thin. A mention in a short tweet, a table cell or a product title offers little evidence, and the linker must rely heavily on priors that are wrong in exactly the cases where the mention is interesting. Language models as matchers, and their limits Large language models have made a real difference to both entity resolution and entity linking. They handle variant names, abbreviations and transliterations that would require extensive rule sets. They bring background knowledge: a model knows, in most cases, that "Big Blue" is a nickname for IBM and that "the Fed" in an American financial article means the Federal Reserve. Asked to compare two records and explain whether they describe the same entity, a capable model will usually give a sensible answer with sensible reasons. Several research groups have reported that general-purpose models match or approach specialised entity-matching systems on standard benchmarks without task-specific training. There are three reasons not to hand identity over to a language model entirely. The first is that a model's background knowledge is a strength on famous entities and a liability on obscure ones. Asked about two records for small, similarly named companies, the model has no real knowledge and will reason from surface similarity, while sounding as confident as it does about IBM. Since the graph's hardest and most valuable identity decisions are usually about entities that are not famous, the model is least reliable where it matters most. The second is cost and consistency at scale. Blocking may still leave hundreds of millions of candidate pairs; sending each to a large model is expensive, and the model's decisions may not be consistent from one pair to the next in the way transitive clustering needs. The practical pattern is to use cheaper methods for the clear cases and reserve the language model, or a human, for the ambiguous middle band. The third is accountability. When a bank wrongly merges a customer with a sanctioned individual, or wrongly keeps apart two records that should have triggered a fraud alert, someone has to explain why. A rule-based or probabilistic matcher can show which fields agreed and how they were weighted. A language model can produce an explanation, but the explanation is generated text rather than a trace of the actual computation, and auditors are rightly wary of it. Measuring whether resolution works Entity resolution needs evaluation as much as extraction does, and its evaluation has its own traps. The obvious measures are pairwise: of the pairs the system declared matches, what fraction truly are (precision), and of the true matching pairs, what fraction did it find (recall). But pairwise measures can mislead, because the number of pairs grows with the square of cluster size. A single wrong merge of two large clusters, say two corporate groups with fifty records each, creates thousands of false pairs and swamps the score, while many small errors elsewhere barely register. Cluster-level measures, which ask how closely the system's groups correspond to the true groups, give a more balanced view. In practice teams report both. The hard part, again, is obtaining the truth to compare against. A useful approach is to sample clusters produced by the system and have reviewers judge each one: is every record in this cluster the same entity, and are there records elsewhere that belong in it? The first question is easy to answer by inspection; the second requires searching, which is why recall is usually estimated less precisely than precision. Sampling should deliberately over-represent the difficult cases, such as clusters with many records, records with common names and records from the least reliable sources, because errors concentrate there and a uniform sample may miss them. Thresholds deserve explicit attention. Every matcher produces scores, and someone must choose the score above which pairs are merged, the score below which they are kept apart, and the band between, which goes to review. Moving these thresholds trades false merges against missed matches, and the right trade depends on consequences. For a marketing database, a few missed duplicates mean a customer receives two catalogues. For a fraud system, a missed match may let a banned person open a new account; a false merge may freeze an innocent person's funds. These consequences are business decisions, and the thresholds that encode them should be set by people who own the consequences, not left at whatever default a vendor shipped. Identifiers that solve the problem upstream The cheapest entity resolution is the one that does not need to happen, because the data arrives carrying a shared identifier. Several important domains have built such identifiers precisely because downstream matching proved so costly. After the 2008 financial crisis, regulators found that they could not quickly determine which firms were exposed to Lehman Brothers, in part because the same legal entities appeared under different names and codes in different systems. The response was the Legal Entity Identifier, a twenty-character code defined by the ISO 17442 standard and overseen by the Global Legal Entity Identifier Foundation. Many jurisdictions now require LEIs for parties to financial transactions, and the LEI data includes information about direct and ultimate parent entities. In research, ORCID gives each researcher a persistent identifier that follows them across name changes and institutions, and DOIs identify publications. In the life sciences, identifiers for genes, proteins and chemical compounds are maintained by long-lived databases and cross-referenced extensively. Public knowledge graphs play a similar role for everything else. Wikidata records, for many of its items, the identifiers that the same entity has in hundreds of other databases: national library authority files, company registers, stock exchanges, film databases, biological databases. Linking a local record to a Wikidata item therefore often links it, at one remove, to a large network of external identifiers. This is one of the most valuable and least advertised services Wikidata provides. The lesson for anyone designing a graph is to capture and propagate identifiers at every opportunity. Where an authoritative identifier exists, store it, match on it, and require it of data suppliers. Where none exists, mint stable internal identifiers, never reuse them, and keep a record of every merge and split so that a past decision can be understood and reversed. Decisions, not facts The deepest point about entity resolution is that its outputs are decisions, not observations. No source states that record 4471 and record 9013 describe the same person; a matcher concludes it, with some confidence, on the basis of evidence. That decision can be wrong, can become wrong as new evidence arrives, and may be right for one purpose and wrong for another. Graphs that treat merges as irreversible facts accumulate errors silently. Two records are merged; facts from both now hang on one node; a year later new evidence shows they were different people, and nobody can now tell which facts came from which. Graphs that treat merges as revisable decisions keep the original source records intact, store the match decisions separately with their scores and evidence, and compute the merged view from them. This costs more storage and more complexity. It is the only design that allows identity errors, which are certain to occur, to be corrected without corrupting everything attached to them. Human review belongs in this process, not as a last resort but as a designed stage. Pairs in the uncertain band of scores go to reviewers with the evidence laid out side by side. Their decisions are stored, used to retrain or recalibrate the automatic matcher, and audited. The volume of human review is a cost to be managed, but eliminating it entirely is a choice to accept a certain rate of undetected identity errors, and that rate should be measured and chosen deliberately rather than discovered. Everything else in the graph depends on this layer. Extraction produces claims; ontology gives them meaning; but only identity resolution decides what they are claims about. Chapter 5: The Graph Behind the Search Box Web search is where knowledge graphs became part of ordinary life, and it remains the setting in which their benefits and their failure modes are most visible to the public. A search engine sits between billions of people and an unimaginably large, contradictory, constantly changing body of text. The knowledge graph is the part of the search engine that tries to say, not "here are pages containing your words," but "here is the thing you mean, and here is what is known about it." This chapter looks at how that works, where the facts come from, what structured markup on the web contributes, and how the arrival of generated answers in search has made the underlying graph more important rather than less. What the graph does for a search engine When Google introduced its Knowledge Graph in 2012 it described three benefits: finding the right thing, getting the best summary, and going deeper and broader. Those three still capture what an entity layer does inside a search engine, though the mechanisms have become far more elaborate. Finding the right thing is disambiguation. The query "taj mahal" might be about the monument, the blues musician, or a local restaurant. A search engine with an entity layer can recognise that the query matches several entities, estimate from the user's context and from aggregate behaviour which is most likely, and, where the ambiguity is real, offer the alternatives explicitly. This is the entity linking problem from the previous chapter, applied to queries that are often only two or three words long, with almost no surrounding context. It works as well as it does because the search engine has enormous amounts of evidence about what people who type a given query go on to click. Getting the best summary is the knowledge panel and its relatives: the box of key facts about a person, place, organisation, work or product that appears beside or above the results. The panel is assembled from facts in the graph, selected according to the type of entity (a film gets its director, release date and cast; a company gets its founders, headquarters and stock price) and often according to what users typically search for next. Going deeper and broader is navigation through relationships: from a novel to its author to the author's other books, from a city to its notable landmarks, from a disease to its symptoms and treatments. These "people also search for" and carousel features follow edges in the graph. Beyond these visible features, the entity layer improves ranking in less obvious ways. Recognising that a query and a page are about the same entity, even when they use different words for it, helps match them. Knowing that a page is about a well-identified entity, and that its claims are consistent with the graph, can count as evidence of quality. Google's documentation and public statements over the years have emphasised understanding entities and their relationships as central to how search works, though the details of ranking are, understandably, not published. The scale is large. In May 2020 Google stated that its Knowledge Graph had grown to more than 500 billion facts about five billion entities. Whatever the precise figures today, no human team curates a graph of that size. It is built overwhelmingly by the automated extraction and resolution methods described in the previous two chapters, applied to the open web and to licensed data, with human review concentrated on high-visibility entities and reported errors. Where the facts come from The sources that feed search engine knowledge graphs fall into a few broad groups. Open collaborative resources were central from the beginning. Freebase, acquired by Google in 2010, was one of the original seeds. Wikipedia, with its infoboxes of structured facts and its carefully linked articles, has been a key source of both facts and the short descriptions shown in panels. Wikidata, which now holds the structured data that many Wikipedia infoboxes display, is used by many organisations as a backbone of identifiers and facts. The importance of these resources to commercial search gives them an odd position: volunteer-edited projects whose accuracy affects what billions of people see. Licensed data covers domains where accuracy and timeliness matter commercially: sports results, financial market data, film and television listings, music catalogues, business listings and so on. Search engines contract with providers who maintain such data professionally. Entity owners themselves contribute. Businesses provide information about their locations, opening hours and contact details through dedicated tools such as Google Business Profile. Notable people and organisations can, after verification, claim their knowledge panels and suggest corrections. And the open web is mined at scale. Extraction systems read pages to find facts, and the structured markup that publishers embed in their pages, discussed next, makes this much easier. Each source has characteristic errors. Collaborative sources can be vandalised or simply wrong, and the delay between an error being introduced and being corrected can be long enough for it to be copied elsewhere. Licensed data can lag. Self-reported information can be promotional. Extracted information inherits every problem described in Chapter 3. The graph's job is to reconcile these, which brings back the question of provenance and conflict resolution that runs through this book. Structured data on the web From the start, the semantic web vision imagined that publishers would describe their content in machine-readable form. For years this happened only patchily. The turning point was schema.org, launched in 2011 by Google, Microsoft and Yahoo, with Yandex joining soon after, which gave publishers a single, pragmatic vocabulary that all the major search engines promised to understand. A publisher using schema.org adds a small block of structured data to a web page, today most commonly in a format called JSON-LD placed in the page's code, stating for example that the page describes a Recipe, with a given name, author, cooking time, list of ingredients and average rating. Or that it describes an Event with a date, location and ticket price; or a Product with a brand, price and availability; or an Organization with a logo, address and links to its official social media profiles. Search engines use this markup in two main ways. The first is to generate richer search results: recipe results with cooking times and star ratings, event listings with dates, product results with prices. Search engines document which kinds of markup make a page eligible for which features. The second, less visible, is to feed the entity layer. A property such as sameAs, which states that this organisation is the same entity as the one described at a given Wikidata or Wikipedia address, is a direct entity-linking signal from the publisher. Markup on an organisation's own website stating its official name, founders and logo is evidence the search engine can weigh against other sources. The effect has been to make a large part of the web partially self-describing. The Web Data Commons project, which analyses the Common Crawl web archive, has tracked the spread of structured markup over many years and found it on a substantial and growing fraction of pages. This is arguably the most successful realisation of the original semantic web idea, achieved not through formal ontologies but through a simple shared vocabulary and a clear incentive: pages with markup became eligible for more prominent displays in search. The incentive also creates problems. Publishers mark up what benefits them, and sometimes mark up things that are not true, such as reviews that do not exist or prices that do not apply. Search engines respond with guidelines, manual actions and algorithms that compare the markup with the visible page content. Structured data, like any other self-reported evidence, has to be weighed rather than trusted. Beyond the results page The same entity layer serves many products besides the list of results. Voice assistants depend on it heavily, because a spoken answer has no room for ten blue links: asked how tall a mountain is or who directed a film, an assistant must identify the entity, look up the property, and say a single value. Maps rely on a graph of places, with their names, categories, opening hours and relationships to the businesses that occupy them. Shopping features rely on product graphs that resolve the same item sold by many merchants under different titles into one product with comparable offers. News features cluster articles by the entities they discuss, so that coverage of a company or a public figure can be grouped and followed. In each case the common requirement is the one this book has emphasised: a stable identifier for the thing, to which facts and content from many places can be attached. A restaurant whose name changes, or which shares its name with a restaurant in another city, poses the same identity problem as a company or a person. Errors propagate across products too. A business wrongly marked as permanently closed in the places graph loses customers on maps, in search and in voice answers at once. Wikidata deserves particular mention in this context, because it has become a kind of public switchboard of identity. Many of its items record identifiers from dozens or hundreds of external databases, from national library authority files to film, music, sports and scientific databases. A search engine, a museum, a news organisation or a researcher who links their own records to Wikidata items gains, at a stroke, connections to all those other identifier systems. This is why public graphs matter even to organisations with large private ones: they supply a shared vocabulary of identity that no single company could impose, and that anyone can use to align their data with everyone else's. When the graph is wrong in public Because knowledge panels present facts with the visual authority of a reference work, their errors attract attention. Over the years panels have shown wrong photographs of people, attached one person's biography to another with the same name, reported living people as dead, and repeated vandalism that had been inserted into a collaborative source. Each such error can usually be traced to one of the stages discussed in earlier chapters: an extraction that misread a sentence, an entity resolution decision that merged two people, a source that was wrong at the moment it was ingested. These incidents illustrate a structural difference between a list of links and a knowledge panel. A list of links presents sources and leaves the reader to judge them. A panel presents conclusions, usually without visible sources, and invites the reader to take them as settled. The more a search engine shifts from pointing at documents to stating facts, the more it takes on responsibility for the accuracy of those facts, and the more its identity and provenance machinery matters. Search engines have developed mechanisms in response: feedback links on panels, verification processes that let entities claim and correct their own panels, priority review for sensitive categories such as health and recently deceased people, and editorial policies about which sources may be used for which kinds of facts. None of this eliminates errors, but it is an implicit acknowledgement that a graph presented to the public needs a governance process, not just a pipeline. Generated answers and the return of the graph From 2023 the major search engines began placing text generated by large language models at the top of results. Google's AI Overviews, launched broadly in the United States in May 2024 and extended to many other countries since, and the conversational answers offered by Microsoft's Bing and by newer search products, all compose a summary answer to the query, drawing on retrieved web pages and, to varying and generally undisclosed degrees, on structured knowledge. One might have expected generated answers to make the knowledge graph redundant: why maintain a structured store of facts if a model can read the pages and write the answer? In practice the opposite has happened. The early weeks of generated search answers produced widely reported errors, some absurd, in which a model summarised satirical or joking content as if it were fact, or combined information about two different entities. These are exactly the failure modes a curated entity layer is designed to prevent: failing to distinguish reliable from unreliable sources, and failing to keep distinct things distinct. The structured graph therefore has at least three roles in generated search. It disambiguates the query, so that the system retrieves documents about the right entity. It supplies short, verified facts, such as dates, heights, populations and office holders, that are better looked up than generated. And it provides a check against which generated statements can be compared: if the generated text says a person was born in one year and the graph says another, something is wrong. How each search engine combines these is proprietary, but the general logic is the same as that discussed for enterprise systems in Chapter 7. For publishers and organisations, the shift raises the stakes of being correctly represented in the entity layer. If an increasing share of searches end with a generated answer rather than a click, then what the answer says about an organisation depends more on what the search engine believes about it as an entity, and less on what any particular page says. Organisations have responded by attending more carefully to their structured data, their entries in public graphs such as Wikidata, and the consistency of their name, address and key facts across the sources search engines draw on. Some of this is legitimate maintenance; some shades into attempts to manipulate public graphs, which those communities police with varying success. Either way, it is a recognition that in generated search, identity in the graph is the foundation everything else stands on. What an organisation can control For any organisation that cares how it appears in search, the practical question is what it can do about its representation in a graph it does not own. The honest answer is that it can influence the evidence but not the conclusion. It can publish accurate structured data about itself on its own website: its official name, logo, address, founding date, leadership and links to its official profiles elsewhere, using schema.org markup and the sameAs property to connect to its entries in public graphs. It can keep its business listings consistent across the services that search engines draw on, since inconsistent names, addresses and phone numbers across directories are a common cause of duplicate or conflated entities. It can ensure that authoritative third-party sources, such as company registers and industry databases, hold correct and current information. Where it meets the community's notability and sourcing standards, it can see that public graphs such as Wikidata carry accurate, referenced statements, while respecting those communities' rules on conflicts of interest. And where search engines offer verification and correction processes, it can use them. What it cannot do is assert its way into the graph. Search engines weigh self-description against independent evidence, precisely because self-description is the evidence most likely to be self-serving. An organisation whose own claims are contradicted by registries, news coverage and public graphs will usually find that the independent sources win. That is, on balance, the right outcome, and a reminder that the graph's value to everyone depends on its not being easily captured by anyone. Lessons from the open web Search is the extreme case of the problems this book discusses. Its sources are vast, uncoordinated and adversarial. Its users ask about everything. Its errors are public. The lessons it teaches apply, in milder form, to every other setting. Entity disambiguation is worth more than any amount of fact extraction, because a fact attached to the wrong entity is worse than no fact. Provenance and source weighting are essential, because sources disagree and some are wrong on purpose. Simple shared vocabularies, adopted because they bring a concrete benefit to the people who use them, achieve more than elaborate ones adopted out of principle. And the more a system presents conclusions instead of sources, the more it needs governance to stand behind them. The enterprise, where knowledge graphs face a different version of the same problems, is the subject of the next chapter. Chapter 6: Knowledge Graphs Inside the Enterprise Inside a large organisation, the knowledge graph faces a different version of the problem it solves on the web. The sources are fewer and more trustworthy, but they were never designed to fit together. A bank has a core banking system, a credit card platform, a mortgage system inherited through an acquisition, a customer relationship management tool, a sanctions screening service, a trade surveillance system and a data warehouse that copies from most of them on a schedule. Each has its own idea of what a customer is, its own identifiers, its own field names, and its own blind spots. The questions that matter most, such as who really controls this account, what is our total exposure to this corporate group, which customers will be affected by this policy change, cut across all of them. Enterprise knowledge graphs exist to answer such questions. They are less visible than search engine graphs and, in aggregate, probably more numerous. This chapter looks at where they have proved their value, what they actually take to build, and how to tell whether an organisation needs one. The integration problem The classic enterprise data architecture moves data from operational systems into a central warehouse, reshaping it into tables designed for reporting. This works well for questions that were anticipated when the warehouse was designed. It works poorly for questions that cut across the warehouse's structure, for sources that change frequently, and for data whose meaning depends on relationships rather than attributes. A knowledge graph approaches integration differently. Instead of reshaping every source into a fixed table design, it maps each source onto a shared ontology and a shared set of entity identifiers. The customer in the mortgage system and the customer in the card platform become the same node, if entity resolution says they are the same person. The products they hold, the addresses associated with them, the companies they direct, and the transactions between them all become edges. A new source can be added by mapping it to the ontology and resolving its entities, without redesigning what is already there. And questions can be asked in terms of the business's concepts, rather than in terms of the table structures of the systems that happen to hold the data. This is sometimes described as a semantic layer: a model of the business's concepts and their relationships that sits between the raw data and the people and applications that use it. The idea has been rediscovered several times under different names, including enterprise data models, master data management, data fabrics and, most recently, semantic layers for analytics and AI. What the knowledge graph adds to these is an explicit, queryable representation of relationships and of identity. Where enterprise graphs earn their keep A few domains have produced enough successful deployments to show where the approach is strongest. Financial crime and ownership are the most established. Detecting money laundering, sanctions evasion and fraud is fundamentally about relationships: who owns whom, who directs what, which accounts send money to which, which addresses and phone numbers are shared. Criminals deliberately use chains of shell companies across several jurisdictions to obscure the person who ultimately benefits. Following those chains is a graph traversal of unknown depth, the kind of query relational databases handle clumsily and graphs handle naturally. Regulators in many jurisdictions require banks to identify the ultimate beneficial owners of corporate customers, and the investigative journalism behind the Panama Papers in 2016, in which the International Consortium of Investigative Journalists worked through millions of leaked documents from a law firm specialising in offshore companies, made heavy use of graph databases to map relationships among companies, intermediaries and people. The same patterns now run inside compliance departments, combining internal customer data with external corporate registries, LEI data and sanctions lists. Life sciences are a second stronghold. Biological knowledge is naturally a network of genes, proteins, pathways, compounds, diseases, symptoms, and the published findings that connect them, and the field has a long tradition of structured, curated databases with stable identifiers. Pharmaceutical companies and research groups build knowledge graphs that integrate these public resources with internal experimental data and with relationships extracted from the literature. One use is drug repurposing: finding an approved drug that may work for a new disease because the graph shows it acts on a target implicated in that disease. A widely cited example came in early 2020, when the company BenevolentAI used its biomedical knowledge graph to identify baricitinib, an arthritis drug, as a candidate treatment for COVID-19, a hypothesis published in The Lancet in February of that year; the drug was later tested in clinical trials and authorised for use in hospitalised patients. The example is instructive in both directions. The graph proposed a plausible hypothesis quickly; clinical trials, not the graph, established whether it worked. Product and catalogue data is a third. Retailers and manufacturers manage millions of products with attributes, variants, compatibilities, components and suppliers. A product graph can answer questions such as which accessories fit which models, which products contain a component affected by a recall, or which substitutes are available when an item is out of stock. Amazon has published research on its product knowledge graph, which faces in concentrated form the extraction and resolution problems discussed in earlier chapters, since much product information arrives as free text from many sellers describing the same items differently. Supply chains are a fourth, increasingly important area. Disruptions of recent years exposed how little many companies knew about their suppliers' suppliers. Mapping a supply network several tiers deep, combining purchasing data with external information about corporate ownership and facility locations, is a graph problem. So is the regulatory requirement, growing in several jurisdictions, to demonstrate that products are free of forced labour or deforestation further up the chain. Internal knowledge management is a fifth, more mixed area. Organisations have tried to build graphs of their people, skills, projects, documents and decisions, to answer questions like "who here knows about this?" or "where was this decision made?" These have had some success, but are vulnerable to the decay discussed in Chapter 8, since the facts change constantly and nobody's job is to keep them current. What it takes to build one The common thread in successful enterprise graphs is that they began with a specific, valuable question, not with an ambition to model the organisation. A compliance team needed to see beneficial ownership; a research group needed to find candidate targets; a supply chain team needed tier-two visibility. The graph was built to answer that question, proved its value, and then grew as other teams found they could reuse its entities and relationships. The work divides into the stages this book has already described, and the effort is distributed among them very unevenly. Choosing and setting up a graph database is a matter of weeks. Designing the initial ontology, with competency questions from the people who need the answers, takes longer and requires domain experts' time. Mapping the first few structured sources to the ontology is substantial but tractable. Extraction from documents, if needed, is a project of its own. And entity resolution, across sources that were never designed to share identifiers, is almost always the largest and longest piece, and the piece that determines whether the graph's answers can be trusted. Organisations that underestimate this distribution of effort tend to follow a recognisable path. They buy a graph database, load several sources into it with only light reconciliation, produce an impressive visualisation of everything connected to everything, and then discover that the graph cannot answer the business's questions reliably because it contains four versions of each major customer and no one knows which relationship came from which source. The project is then judged a failure of the technology, when it was a failure to do the identity work. Two organisational features distinguish deployments that last. The first is ownership. Someone, usually a small central team, owns the ontology and the identity layer: they decide what classes and relationships mean, they run entity resolution, and they handle disputes. Business teams own the data in their own domains but conform to the shared model. Without a central owner, the ontology fragments as each team extends it to suit itself. The second is a feedback path. Users who find errors must have a way to report them, and reports must lead to corrections, both in the graph and, where possible, in the pipeline or source that produced the error. A build in outline To make the distribution of effort concrete, consider how a mid-sized bank might build a beneficial ownership graph for its corporate customers. The competency questions come first, from the compliance team: Who are the individuals who ultimately own or control more than a threshold share of this customer? Which of our customers share an ultimate owner, director or registered address with a sanctioned or high-risk party? Which customers' ownership has changed since we last reviewed them? These questions fix the core ontology: legal entities, natural persons, addresses, ownership and control relationships with percentages and dates, directorships, and sanctions designations. The bank adopts FIBO's definitions for legal entities and control where they fit, and records LEIs wherever they exist. The sources are mapped next: the bank's customer onboarding records, the national company register's ownership and director data, a commercial provider's international ownership data, the LEI database's parent relationships, and the official sanctions lists. Each source is loaded with its own provenance, so that every ownership statement records where it came from and when. Then comes resolution, which takes most of the time. Customer records must be matched to register entries, register entries to the commercial provider's entities, and individuals across all of them, including the sanctioned individuals whose names may be transliterated in several ways. Pairs that the automatic matcher scores in the uncertain middle band go to a small review team, whose decisions feed back into the matcher. Conflicting ownership percentages from different sources are kept side by side, with a rule for which to prefer in reports and a flag for analysts when the difference matters. Only after this does the graph begin to answer its questions reliably. The traversal that finds an ultimate owner through four layers of holding companies is a few lines of query. The months of work were in making sure that the four layers were correctly connected, that each company in the chain was the right company, and that the person at the top was not confused with someone of the same name. Banks that have built such graphs typically describe the same pattern: the technology was the easy part. Standards, vendors and the build-or-buy question Enterprise graph builders face a technology market that has matured considerably. Native graph databases from several vendors, graph capabilities added to major relational and cloud databases, RDF triplestores with reasoning, and platforms that combine storage with tools for ontology management, entity resolution and visualisation are all widely available. The publication of the GQL standard in 2024 may, over time, reduce lock-in to particular vendors' query languages for property graphs, as SQL did for relational databases. In regulated domains, shared ontologies reduce the work considerably. A bank that models legal entities and ownership using FIBO, and identifies counterparties with LEIs, starts with definitions that regulators, auditors and other institutions already understand. A life sciences company that uses standard identifiers for genes, compounds and diseases can integrate public data with little mapping effort. The build-or-buy decision is rarely about the database. It is about whether to acquire external data that is already resolved and linked, such as commercial corporate ownership data, curated biomedical graphs, or product data from industry registries, or to build that resolution in-house. Buying resolved data is often the cheaper route for the external world; the organisation's own internal data must always be resolved by the organisation itself. The semantic layer for software agents Since 2024 a further use has emerged that makes the enterprise graph more important still. Organisations are deploying language-model agents: systems that take a goal, plan steps, call internal tools and databases, and act on the results. An agent asked to "prepare the quarterly exposure report for our top twenty counterparties" must find out which counterparties are the top twenty, by what measure, across which legal entities, and from which systems. Left to explore raw databases with cryptic table names, it will make plausible guesses, and some of them will be wrong in ways that are hard to detect. A knowledge graph, or the ontology at its core, gives such an agent a map. It says what a counterparty is in this organisation, which legal entities roll up into which groups, where the authoritative exposure figures live, and what the relationships between them mean. The agent can query that map before acting, and its actions can be checked against it. In effect the ontology becomes documentation written for machines as well as people, and the identity layer becomes the shared reference that keeps several agents, and the humans supervising them, talking about the same things. This use is young, and its practices are still forming. But it extends the argument of this chapter rather than changing it. Agents amplify whatever the organisation's data already gets right or wrong. Where identity has been resolved and meaning written down, they can work quickly and verifiably. Where it has not, they will reproduce, at machine speed, the same inconsistencies that human analysts used to work around by knowing which spreadsheet to trust. A common failure: the hairball One failure pattern is distinctive enough to name. Teams new to graphs often begin by loading everything they can reach and visualising the result. The picture is a dense tangle of nodes and edges, sometimes called a hairball, which looks impressive in a demonstration and answers nothing. Every customer is connected to every other through shared branches, products, generic addresses such as a registered agent's office, and placeholder values such as "unknown" that were loaded as if they were real entities. The hairball is a symptom of two missing disciplines. Without competency questions, there is no basis for deciding which relationships are meaningful and which are noise. Without careful resolution and cleaning, placeholder values and shared service providers become false hubs that connect unrelated entities. A single node for "N/A" linked to forty thousand customers as their employer will dominate every path-finding query. The cure is not a better visualisation but a narrower scope, a cleaner identity layer and explicit rules for which values may become nodes. When not to build a knowledge graph Because the term carries prestige, organisations sometimes build knowledge graphs for problems that do not need them. Several signals suggest that a graph is the wrong tool. If the important questions are about aggregates over a stable set of attributes, such as revenue by region and month, a data warehouse is the right tool and a graph adds complexity without benefit. If the data comes from one or two well-designed systems with shared identifiers, the integration problem a graph solves does not exist. If the relationships involved are few and fixed, a relational design with a handful of join tables is simpler to build and maintain. And if nobody can name the competency questions the graph would answer, or the people who would be accountable for its accuracy, the project is likely to produce a graph that nobody uses. Conversely, a graph is likely to be worth the effort when several of the following hold: many heterogeneous sources describing overlapping entities; questions that depend on chains of relationships of variable length; a schema that must keep evolving as new kinds of entities and relationships appear; a need to explain answers by showing the path that produced them; and a need to serve trusted facts to other systems, including, increasingly, language models. That last need has transformed the conversation about enterprise knowledge graphs since 2023. Many organisations that had never considered building one began to do so because they wanted to put language models to work on their own data and found that the models, left to themselves, could not be trusted with it. How graphs help, and how much, is the subject of the next chapter. Chapter 7: Grounding Language Models Large language models write fluently about almost anything. They summarise, translate, explain, draft and converse with a facility that would have seemed implausible a decade ago. They also state falsehoods with exactly the same fluency, invent citations, confuse people who share a name, report facts that were true when their training data was collected but are no longer, and cannot reliably say where any particular claim came from. The industry's word for the most visible of these failures is hallucination, though the term understates how structural the problem is. A language model is trained to produce likely text, and likely text is not the same thing as true text. This chapter is about the use of knowledge graphs to make language models more reliable, a practice often called grounding. It sets out why the problem arises, describes the main patterns for combining graphs with models, reviews what the evidence says about when graphs help, and identifies the limits of the approach. The central claim is that graphs help language models in exact proportion to the quality of the identity, meaning and provenance commitments they contain. A graph that merely restructures the same text the model could have read gives little; a graph that carries resolved entities, defined relationships and traceable sources gives a great deal. Why language models need help A language model's knowledge, such as it is, lives in billions of numerical parameters adjusted during training to predict text. Facts that appear often and consistently in the training data are usually reproduced correctly. Facts that appear rarely, inconsistently, or not at all are reproduced unreliably, and the model has no dependable internal signal that distinguishes the two cases. It does not store "Marie Curie, award received, Nobel Prize in Physics, 1903, source: Nobel Foundation" as a discrete record. It stores statistical tendencies from which such a sentence can usually be regenerated. Several consequences follow, each of which maps onto a feature of knowledge graphs discussed in earlier chapters. The model has no stable notion of identity. Two people with the same name, or a company and its similarly named subsidiary, blur together because the model never assigned them separate identifiers. A knowledge graph's first commitment is precisely to keep them apart. The model's knowledge is frozen at training time and decays. It cannot know about events after its training data was collected, and it cannot tell which of its beliefs have since become false. A graph can be updated continuously, and with temporal qualifiers can say not only what is true but since when. The model cannot cite. Even when it states a fact correctly, it cannot point to the document that supports it, because there is no record of which training documents contributed to which parameters. A graph built with the evidence trail described in Chapter 3 can return, for every fact, the passage it came from. And the model knows nothing about private data. An organisation's customers, contracts, products and internal decisions were not in the training data, and should not be. Whatever the model is to say about them must be supplied at the time of the question. Retrieval: documents, graphs, or both The dominant approach to grounding is retrieval-augmented generation, a term introduced in a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research. The idea is simple: when a question arrives, first retrieve relevant information from a trusted store, then give that information to the model along with the question, and instruct the model to answer from it. The model's role shifts from recalling facts to reading and synthesising supplied material. In its most common form, the trusted store is a collection of documents split into passages, each converted to a numerical vector (an embedding) that captures its meaning. Retrieval finds the passages whose vectors are closest to the question's. This vector retrieval works well for questions whose answer is contained in one or a few passages that use language similar to the question. It works less well for three kinds of question that are common in practice. The first is questions that require connecting facts spread across many documents: "Which of our suppliers share a parent company with a supplier that failed an audit last year?" No single passage contains the answer. Vector retrieval may surface passages about each supplier, but the joining has to be done by the model, unreliably, in its context window. The second is questions that require exhaustive or aggregate answers: "List every contract that includes a change-of-control clause." Vector retrieval returns the top few most similar passages, not all relevant ones, and the model has no way of knowing what it has missed. The third is questions where entity identity matters: "What did our company say about Jordan in its last filing?" Similarity in embedding space is a poor substitute for knowing which Jordan. Knowledge graphs offer different retrieval strategies that address these weaknesses. Table 3 compares the three most common patterns, which in practice are increasingly combined. Table 3. Three ways to retrieve facts for a language model. Pattern How retrieval works Best for Main risk Vector retrieval over documents Find passages similar to the question Questions answered in one or a few passages Misses connections, incomplete lists, wrong entity Graph query generation Model writes a SPARQL, Cypher or SQL query over the graph; results passed back Precise, exhaustive and multi-hop questions over structured data Wrong query silently returns wrong answer Graph-guided retrieval Link question entities to the graph, expand to neighbours and their source passages Questions about specific entities and their relationships Graph quality limits what is found In graph query generation, the model translates the user's question into a formal query against the graph, the query is executed by the database, and the results, which are exact, are given back to the model to phrase as an answer. This turns the model from a source of facts into a translator, a task at which it is much more reliable. The weakness is that a subtly wrong query returns a precise-looking wrong answer, and the user may never see the query. Systems in production therefore validate generated queries against the ontology, restrict them to templates for high-stakes questions, and show the query or a plain-language paraphrase of it for inspection. In graph-guided retrieval, entity linking is applied to the question itself. The entities it mentions are located in the graph, their neighbourhood is explored to a chosen depth, and the facts found, together with the source passages that support them, are given to the model. This combines the precision of identified entities with the richness of the original text. Many current systems combine all three patterns, routing each question to whichever strategy suits it, or using several and merging the results. A question traced through the system Following one question through a grounded system shows where each piece of the graph does its work. Suppose an analyst at a pharmaceutical company asks an internal assistant: "Which of our marketed drugs share a molecular target with compounds that failed for liver toxicity in the last five years?" First, the system parses the question and links its entities and concepts to the graph: "our marketed drugs" resolves to a class of products with a particular status and owner; "molecular target" to a defined relationship; "failed for liver toxicity" to trial outcomes with a particular reason code. If the ontology lacks a clear definition of what counts as a failure, the question cannot be answered precisely, and the system should say so rather than guess. Second, the system generates a graph query: find drugs with marketed status, follow their target relationships to proteins, follow those back to other compounds, filter to compounds whose development was terminated with hepatotoxicity as a reason within the date range. The query is validated against the ontology before it runs, which catches, for example, a reference to a relationship that does not exist. Third, the query runs and returns an exact list, with each step traceable: this drug targets this protein according to this curated database; this compound was terminated for this reason according to this internal trial record or this published paper. Fourth, the language model receives the results and the supporting passages, and writes a readable answer that names the drugs, the shared targets and the failed compounds, with citations. It may add context drawn from the passages, such as whether the toxicity was thought to be target-related or specific to the failed compound's chemistry. That distinction matters enormously and lives in the text, not the triples. Every stage depends on commitments made long before the question was asked. Entity linking needs a graph with resolved drugs and proteins. The query needs an ontology with clear meanings. The citations need provenance. And the reliability of the final answer is bounded by the accuracy of the target and termination data, which someone has to maintain. The same example raises an issue that document retrieval often neglects: permission. Some trial records are confidential to particular teams; some partnership data may be shared only under agreement. A grounded assistant must respect the same access rules as the underlying data, which means the graph needs to carry access controls at the level of facts or sources, and retrieval must apply them before anything reaches the model. A model cannot be trusted to withhold what it has been shown. Graphs with fine-grained provenance make this tractable, because access can follow source; graphs without it make it close to impossible. The evidence How much do graphs actually help? The honest answer is that it depends heavily on the question type and on the quality of the graph, and that published evidence, while encouraging, comes mostly from benchmarks and vendor studies rather than from independent evaluation of production systems. One frequently cited study, published in November 2023 by Juan Sequeda, Dean Allemang and Bryon Jacob at the data catalogue company data.world, tested GPT-4 on answering business questions over an insurance company's relational database, modelled on an industry-standard schema. Asked to generate SQL directly against the database's tables, the model answered correctly roughly 16 percent of the time. When the same questions were posed over a knowledge graph representation of the same data, with an ontology describing the business concepts and their mapping to the tables, accuracy rose to roughly 54 percent. A follow-up study in 2024 by Allemang and Sequeda reported further gains from using the ontology to check generated queries and repair them. The study is limited in scope, and its authors are advocates for the approach, but the mechanism it points to is persuasive: the model does far better when it is given a description of the business's concepts in business terms than when it is given only cryptic table and column names. That is the ontology doing its job of communication, as described in Chapter 2. A second influential line of work came from Microsoft Research, which in April 2024 published "From Local to Global: A Graph RAG Approach to Query-Focused Summarization" by Darren Edge and colleagues, and released its GraphRAG software as open source that summer. GraphRAG uses a language model to extract entities and relationships from a document collection, builds a graph, groups the graph into communities of closely connected entities using a clustering algorithm, and has the model write summaries of each community. Questions about the collection as a whole, such as "what are the main themes in these documents?", are then answered by drawing on the community summaries rather than on individual passages. The paper reported that this approach produced more comprehensive and diverse answers than vector retrieval for such global sense-making questions. GraphRAG triggered a wave of similar systems and considerable enthusiasm, followed by more measured assessment. Building the graph requires running a large model over the entire collection, which is expensive. Independent comparisons have generally found that graph-based retrieval helps on multi-hop and global questions but offers little or no advantage, sometimes a disadvantage, on simple factual questions answered within one passage, which vector retrieval handles well and more cheaply. And because the graph in GraphRAG is extracted by a language model without a curated ontology or a rigorous entity resolution stage, it inherits the extraction and identity errors discussed in Chapters 3 and 4. It is best understood as a powerful indexing and summarisation technique, rather than as a knowledge graph in the full sense this book has been describing. The broader research literature, surveyed for example in a 2024 roadmap by Shirui Pan and colleagues in IEEE Transactions on Knowledge and Data Engineering, sorts the combinations of graphs and models into three families: graphs used to enhance models (in training or at question time), models used to build and complete graphs, and systems in which the two work together iteratively. All three are active areas, and the boundaries among them are blurring. Circularity and its remedy There is a trap in combining language models with knowledge graphs, and many current systems fall into it. If a language model extracts the graph from documents, resolves its entities, and then answers questions from it, the graph contains the model's interpretations of the documents, not independently established facts. Grounding the model in such a graph is partly grounding it in itself. Errors in extraction become facts in the graph, and the model then repeats them with the added authority of having looked them up. The remedy is not to avoid language models in graph construction, which would discard their very real benefits, but to put independent checks at the points where the graph makes commitments. Facts extracted by a model should carry the source span and should be verified against it mechanically. Entity resolution decisions in the uncertain band should be reviewed by people, or by methods independent of the model that extracted the facts. Facts from authoritative structured sources, such as registries, regulated filings and curated databases, should be distinguished from facts extracted from prose, and given priority when they conflict. The ontology should be designed by people who understand the domain, not generated wholesale by a model from the documents. When these checks are in place, the graph is no longer an echo of the model. It is an independent structure, partly built with the model's help, against which the model's outputs can be tested. Graphs as checks, not just sources That last point suggests a role for knowledge graphs beyond retrieval. A graph can be used to verify what a model says after it has said it. Claims in a generated answer can be extracted, linked to entities, and compared with the graph. An answer stating that a company's chief executive is someone the graph records as having left two years ago can be flagged or corrected before it reaches the user. An answer that cites a regulation the graph does not contain can be marked as unsupported. This verification step uses exactly the same machinery as graph construction, extraction and linking, applied in reverse. It cannot catch every error, since it can only check claims that the graph covers, but in domains where a curated graph exists, such as drug interactions, product specifications, corporate ownership and regulatory requirements, it provides a safeguard that no amount of prompting can. The limits of grounding Knowledge graphs do not make language models trustworthy in general. They make them more trustworthy on the questions the graph covers, to the extent the graph is accurate. Several limits should be kept in view. Coverage is always partial. Most of what an organisation knows, and most of the nuance in any field, does not reduce to triples. A graph can say that a contract has a change-of-control clause; it cannot convey the clause's careful drafting. Systems must still retrieve and read text for questions that depend on such nuance, and should not pretend the graph is the whole of knowledge. The model can ignore what it is given. Even with correct facts in its context, a language model may contradict them, particularly when they conflict with strong patterns from its training. Instructions to answer only from supplied material help, and verification helps further, but neither is absolute. And graph quality is a moving target. A graph that was accurate at launch decays as the world changes, unless it is actively maintained. The final chapter takes up that problem. Chapter 8: Keeping the Graph True A knowledge graph begins to go out of date on the day it is loaded. Executives leave, companies are acquired, products are discontinued, drug labels are revised, borders and office holders change, and the documents from which facts were extracted are corrected or retracted. Meanwhile, the pipelines feeding the graph continue to add new facts, some of them wrong, and the entity resolution decisions made at launch quietly become wrong as new records arrive that do not fit the old clusters. A graph that is not actively maintained does not stay still; it drifts away from the world while continuing to answer questions with the same confidence. This is the least discussed and most important part of the knowledge graph lifecycle. The industry survey cited in Chapter 1 found that the organisations running the largest graphs in the world regarded quality, provenance and evolution, rather than scale, as their persistent difficulties. This chapter sets out what quality means for a graph, how it can be measured, what techniques help maintain it, and what kind of organisation it takes to keep a graph true over years. What quality means Quality in a knowledge graph is not one property but several, and they trade against one another. Table 4 summarises the main dimensions and how each can be checked in practice. Table 4. Dimensions of knowledge graph quality. Dimension Question it asks How it is usually checked Accuracy Are the stated facts true? Expert review of random samples; comparison with authoritative sources Coverage Are the facts that matter present? Competency questions; comparison with reference lists Consistency Do facts obey the ontology's rules? Automated constraint and shape validation Identity Is each node exactly one real thing? Review of merge decisions; duplicate detection Timeliness Are facts still current? Age of supporting evidence; change feeds from sources Traceability Can each fact be traced to a source? Share of facts with resolvable provenance Accuracy is what most people mean by quality: whether what the graph says is true. It cannot be measured exhaustively in a large graph, but it can be estimated well by sampling. Draw a random sample of facts, stratified by type and source, have people who understand the domain check each against reliable evidence, and compute the error rate with a confidence interval. A few hundred checked facts per important relationship type give a usable estimate. This is tedious, and it is the only way to know. Organisations that never sample their graph's accuracy do not know how accurate it is, whatever their dashboards suggest. Coverage, sometimes called completeness, asks whether the facts that ought to be there are. A graph of corporate ownership that contains every relationship it holds correctly but misses half of all ownership links is accurate and nearly useless for sanctions screening. Coverage is harder to measure than accuracy because measuring it requires knowing what is missing. Practical methods include checking against an independent reference list (for example, all companies listed on a given exchange), and running the competency questions from Chapter 2 against cases where the answer is known. Consistency asks whether the graph obeys its own rules. Every person has at most one date of birth; nothing is both a person and a company; every subsidiary relationship runs from a company to a company. Unlike accuracy and coverage, consistency can be checked automatically and completely, using the validation languages and constraints described in Chapter 2. Wikidata, for example, defines constraints on its properties and publishes regular reports of violations, which volunteers work through. Consistency violations are among the cheapest errors to find and often point to deeper problems in an extraction or resolution pipeline. Identity quality asks whether each node corresponds to exactly one real-world entity. It has two failure modes: duplicates, where one entity is split across several nodes, and conflations, where several entities have been merged into one. Conflations are usually worse, because they cause facts about one entity to be presented as facts about another. Both can be estimated by sampling and reviewing nodes, and some duplicates can be found automatically by rerunning entity resolution across the graph itself. Timeliness asks whether facts are still current. Its measurement depends on recording when each fact was last confirmed and on knowing, for each type of fact, how quickly it typically changes. A chief executive changes every few years; a share price every second; a person's date of birth never. A fact that was confirmed recently and changes slowly can be trusted; one that has not been confirmed for years and changes often should be flagged. Traceability asks whether each fact can be traced to the evidence for it. It is the precondition for checking all the others. A fact without provenance cannot be verified, cannot be withdrawn when its source is retracted, and cannot be weighed against a conflicting fact from a better source. Provenance and conflict The single most valuable maintenance decision is to treat every fact in the graph as a claim made by a source, rather than as a bare truth. When the graph records that a company was founded in 1998 according to its own website, in 2003 according to a commercial database, and in 1998 according to its registration in a national company register, it holds three claims about one property. Which value the graph presents as current is then a policy decision, made explicitly: perhaps official registries outrank commercial databases, which outrank self-description, which outranks extraction from news. This approach was developed at scale in Google's Knowledge Vault research, published in 2014 by Xin Luna Dong and colleagues, which combined facts extracted from the web by many different extractors with prior knowledge from existing curated graphs, and estimated for each candidate fact a probability that it was true. The underlying principle, known as knowledge fusion or truth discovery, is that agreement among independent sources is evidence, that sources have measurable reliability, and that reliability can be estimated from how often each source agrees with established facts. A fact asserted by a single unreliable source should not carry the same weight as one asserted independently by three reliable ones. Storing claims with provenance has costs: more storage, more complex queries, and the need to define resolution policies. But it has decisive benefits for maintenance. When a source is found to be unreliable, every claim it contributed can be found and reweighted. When a document is retracted, the facts extracted from it can be withdrawn. When a user reports an error, the investigator can see immediately where the fact came from and whether other facts from the same source or pipeline might share the problem. The open world and the meaning of silence A subtle but important principle governs what a missing fact means. Most databases operate under a closed-world assumption: if a customer table does not list a customer, that customer does not exist. Knowledge graphs, especially those built from many sources, usually operate under an open-world assumption: if the graph does not record that a company has a subsidiary in a given country, it may simply not know. Absence of a fact is not evidence that the fact is false. The distinction matters most when graphs are used for compliance and when they ground language models. A sanctions check that concludes "no sanctioned owner found" is only as good as the graph's coverage of ownership; it means "none known," not "none exists." A language model that answers from a graph should be prevented from turning the absence of a fact into a confident negative statement. Good systems state the scope of their knowledge explicitly, and some graphs record negative facts deliberately (this person has no children; this compound has no known interaction with that drug) where a source has positively established them, so that real negatives can be distinguished from gaps. Predicting missing facts Since coverage is always incomplete, a substantial research field has developed around predicting missing facts from the structure of the graph itself, a task known as knowledge graph completion or link prediction. The best-known techniques learn embeddings: numerical vectors for each entity and relationship, adjusted so that true triples score well and false ones poorly. In TransE, an influential model published by Antoine Bordes and colleagues in 2013, a relationship is represented as a translation in vector space, so that the vector for the subject plus the vector for the relationship should land near the vector for the object. Many more sophisticated models followed, and graph neural networks extended the idea by passing information along edges to compute each node's representation from its neighbourhood. These techniques are useful, but their outputs belong in a particular place in the maintenance process. A link prediction model can propose that a missing relationship probably holds, for example that a gene is probably associated with a disease because similar genes are, or that a person probably holds a nationality because of where they were born and worked. Such predictions are hypotheses, valuable for prioritising what to check or investigate. They should not be written into the graph as facts alongside verified statements, or if they are, they should be clearly marked with their provenance as model predictions. A graph that silently mixes predicted and observed facts loses the property that makes it valuable for grounding: the ability to say what is known, and how. Language models now play a similar role, proposing candidate facts to fill gaps, and the same discipline applies. A missing founding date filled in by a model from its training data may well be right; it is still a claim from a source of unknown reliability, and should be treated accordingly. Change over time Maintaining timeliness requires more than periodic reloading. Several practices help. Subscribing to change feeds from authoritative sources is the most effective. Company registries, regulatory databases, scientific databases and Wikidata itself publish updates continuously or regularly; consuming those updates keeps the corresponding parts of the graph current without re-extracting everything. Recording validity intervals, so that a fact has a start time and, when superseded, an end time, preserves history rather than overwriting it. This lets the graph answer questions as of a past date, which matters for audits, investigations and explaining past decisions, and avoids the confusion of a fact simply disappearing. Setting freshness expectations by fact type, and flagging facts that have exceeded them, focuses review effort where it is most likely to be needed. A board membership not confirmed in three years is suspicious; a chemical formula is not. And re-running entity resolution periodically, not only on new records but across existing clusters with the benefit of accumulated evidence, catches identity errors that were invisible at the time they were made. Anatomy of an error Tracing a single error through a graph shows why these practices reinforce one another. Suppose a user reports that the graph lists a well-known engineer as a board member of a company she has never been associated with. With provenance, the investigator finds immediately that the fact was extracted from a press release two years ago by the extraction pipeline then in use. Reading the source span, she sees that the press release named a different person with the same surname and first initial. The extraction was correct; the entity linking was wrong, attaching the mention to the more famous person because the context was thin and her prior probability was high. That diagnosis points to a class of errors, not a single one: mentions linked to famous people on the basis of initials and surnames in thin contexts. A query finds several hundred similar links, a sample is reviewed, and a fraction are wrong. The linker's handling of initials is changed, the affected links are re-run, and the corrected decisions are recorded with their reasons. Without provenance, the investigator would have deleted the one false fact and moved on, leaving the hundreds of others in place, and the same pipeline would have continued producing more. The difference between a graph that gets better over time and one that gets worse is largely the difference between treating errors as items to delete and treating them as evidence about the process. Personal data and the right to be forgotten Many knowledge graphs contain facts about people, and that brings legal obligations that shape maintenance. Under data protection regimes such as the European Union's General Data Protection Regulation, individuals have rights to access the data held about them, to correct it, and in some circumstances to have it erased. Organisations must also be able to show a lawful basis for holding each kind of personal data, and must not keep it longer than necessary. These obligations are much easier to meet in a graph that records provenance and identity well. Answering a request for access means finding every fact attached to a person's node, and every source from which those facts came; if the person has been split across several unresolved nodes, some of their data will be missed. Erasure means removing facts and, often, preventing them from being re-extracted from the same sources on the next run, which requires knowing which sources and pipelines produced them. Retention limits require knowing when each fact entered the graph. A graph built without these records may find that complying with a single erasure request requires a manual search with no guarantee of completeness. Here, as elsewhere, the investment in identity and provenance pays for itself in ways that are not obvious at design time. Governance: who keeps it true All of this requires people with defined responsibilities, which is why knowledge graph quality is ultimately an organisational matter rather than a technical one. The most durable public graph, Wikidata, offers one model. Its quality rests on a large community of volunteers, many of whom specialise in particular domains, supported by automated tools that detect vandalism, report constraint violations and suggest missing references. Its policies require references for many kinds of statements, and its property proposals are discussed and approved before new properties are created. The result is far from error-free, but it is self-correcting at a scale no single organisation could fund. Enterprise graphs need a smaller, more formal version of the same structure. Someone owns the ontology and approves changes to it. Someone owns each major domain of data and is accountable for its accuracy. Someone runs the quality measurements described above and reports them, so that the graph's accuracy is a known number rather than an assumption. And users have a simple way to report errors, with a visible process that fixes them at their source rather than only in the graph. These roles do not require large teams. They require clarity that the graph is a product with a lifecycle, not a project with an end date. The most common reason enterprise graphs fail after a successful launch is that the team that built them is disbanded or moved on, and the pipelines continue running unattended while the world changes around them. Quality as a feature There is a final reason to take maintenance seriously, which follows from the argument of the previous chapter. As knowledge graphs increasingly serve as the grounding for language models, their quality becomes the quality ceiling of every answer generated from them. A model grounded in a graph with a known, measured accuracy, explicit provenance and clear temporal scope can tell its users what it knows, how it knows it, and as of when. A model grounded in an unmaintained graph inherits all of its errors and presents them with fluent confidence. The care described in this chapter is what makes the difference. Conclusion For most of the history of knowledge graphs, the expensive part was getting facts in. Hand-written rules, annotated training data, bespoke classifiers for each relationship type: extraction consumed the budgets and defined the ambitions of graph projects. Large language models have changed that. A competent model, given an ontology and a document, will produce a plausible set of entities and relationships in seconds. Any organisation can now produce a graph from its documents in a week. The chapters of this book point to a conclusion that follows from that change and is not yet widely acted upon: when extraction becomes cheap, the value of a knowledge graph migrates almost entirely to the commitments that extraction cannot make for itself. Which node a mention refers to. Whether two records are one entity or two. What a relationship means and what it does not. Which source a fact came from, and when it was last true. These are the parts of the graph that a language model cannot reliably supply, because they are exactly what a language model lacks. They are also the parts that make the graph useful for grounding a language model in the first place. A graph that is cheap to build but has weak identity, loose meaning and no provenance adds little that the model could not have produced by reading the documents directly. What follows for practice Several practical consequences follow, and they cut against common habits. The first concerns where to invest. Organisations starting a knowledge graph project today should expect to spend less than in the past on extraction and more on entity resolution, ontology stewardship and provenance. A sensible allocation treats extraction as a commodity component, to be swapped as better models arrive, and treats the identity layer, with its stored match decisions, reviewed edge cases and stable identifiers, as the durable asset. If a graph's extraction were thrown away and regenerated with next year's model, the identity layer and the ontology should survive intact. If they would not, they were not designed as assets. The second concerns what to measure. Many graph projects report the number of nodes and edges, which measures effort rather than value. More useful numbers are the sampled accuracy of the most important relationship types, the estimated rate of conflated entities, the share of facts with traceable provenance, and the share of competency questions the graph answers correctly. These numbers are harder to produce and less flattering. They are the ones that predict whether a graph can safely ground a model. The third concerns the place of people. The decisions this book has identified as central, which entity is which, what a relationship means, which source to believe, are judgements. Models can propose them and evidence can inform them, but someone must be accountable for them, and some fraction must be reviewed by people who understand the domain. The temptation, now that models can do so much, is to remove humans from the loop entirely. Doing so does not eliminate the judgements; it delegates them to a system that cannot explain them and cannot be held responsible for them. Designing the human review stage well, small, targeted, and fed back into the automated pipeline, is a better use of effort than trying to eliminate it. The fourth concerns what to say to users. A system grounded in a knowledge graph can tell the truth about its own knowledge: this fact is from the company registry and was confirmed last month; this one was extracted from a news article and has not been verified; this question falls outside what the graph covers. Few systems do this today, because it complicates the interface and makes the system look less certain. But in high-stakes settings, the ability to distinguish what is known from what is generated is the most important thing a grounded system can offer, and it is available only if the graph recorded the distinction in the first place. What remains unsettled Several questions remain genuinely open. One is whether language models will eventually internalise the commitments that graphs make explicit. Research continues on models that represent entities more stably, update their knowledge without retraining, and attribute their claims to sources. If those efforts succeed, the external graph might become less necessary for some purposes. It seems unlikely to become unnecessary for all of them. An organisation needs to be able to inspect, correct and audit what its systems believe, and a structure of explicit, sourced statements is inspectable in a way that model parameters are not. Even a far better model would still need something like a knowledge graph to serve as the record against which it is checked. A second is the governance of public identity. As generated answers take a larger share of how people learn about the world, the entity layers of search engines and the public graphs that feed them gain influence over how people, organisations and ideas are represented. Volunteer communities such as Wikidata's carry a responsibility that has grown faster than their resources, and the incentives to manipulate public graphs grow with their influence. How that responsibility should be shared, and by whom it should be funded, has no settled answer. A third is the circularity described in Chapter 7. As more graphs are built by language models from text that was itself increasingly written by language models, the risk grows that graphs and models will reinforce each other's errors with no independent anchor. The defence is to keep authoritative sources, registries, regulated filings, curated scientific databases, clearly separated from extracted and generated material, and to preserve the provenance that lets the two be told apart. Whether that discipline holds across the industry is an open question with real consequences for the reliability of the information environment. The work that remains The phrase that launched the field's public life, "things, not strings," turns out to describe a continuing obligation rather than a solved problem. Strings are abundant and getting cheaper. Things, meaning identified entities with defined types, traced facts and known histories, are scarce and have to be made and maintained by deliberate effort. The knowledge graph is the place where that effort is recorded. Its value is not that it stores relationships, which many technologies can do, but that it records decisions about what exists, what is the same as what, and what is true according to whom, in a form that machines can use and people can inspect. Those decisions were always the hard part. In an age of fluent machines that make none of them on their own, they are also the most valuable part. Notes Chapter 1 1. Amit Singhal, "Introducing the Knowledge Graph: things, not strings," Google official blog, 16 May 2012. Source of the figures of 500 million objects and 3.5 billion facts. 2. Tim Berners-Lee, James Hendler and Ora Lassila, "The Semantic Web," Scientific American, May 2001. 3. ISO/IEC 39075:2024, Information technology, Database languages, GQL. Published April 2024. 4. Natasha Noy, Yuqing Gao, Anshu Jain, Anant Narayanan, Alan Patterson and Jamie Taylor, "Industry-Scale Knowledge Graphs: Lessons and Challenges," Communications of the ACM 62, no. 8 (2019). Chapter 2 1. Michael Grüninger and Mark S. Fox, "Methodology for the Design and Evaluation of Ontologies," Workshop on Basic Ontological Issues in Knowledge Sharing, IJCAI-95, 1995. 2. R. V. Guha, Dan Brickley and Steve Macbeth, "Schema.org: Evolution of Structured Data on the Web," Communications of the ACM 59, no. 2 (2016). Chapter 3 1. Marti A. Hearst, "Automatic Acquisition of Hyponyms from Large Text Corpora," Proceedings of COLING 1992. 2. Mike Mintz, Steven Bills, Rion Snow and Dan Jurafsky, "Distant Supervision for Relation Extraction without Labeled Data," Proceedings of ACL-IJCNLP 2009. 3. Michele Banko, Michael J. Cafarella, Stephen Soderland, Matt Broadhead and Oren Etzioni, "Open Information Extraction from the Web," Proceedings of IJCAI 2007. 4. Alexander Ratner and colleagues, "Snorkel: Rapid Training Data Creation with Weak Supervision," Proceedings of the VLDB Endowment 11, no. 3 (2017). Chapter 4 1. Halbert L. Dunn, "Record Linkage," American Journal of Public Health 36, no. 12 (1946). 2. Ivan P. Fellegi and Alan B. Sunter, "A Theory for Record Linkage," Journal of the American Statistical Association 64, no. 328 (1969). 3. Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel and Luke Zettlemoyer, "Scalable Zero-shot Entity Linking with Dense Entity Retrieval," Proceedings of EMNLP 2020 (the BLINK model). 4. ISO 17442, Legal Entity Identifier; Global Legal Entity Identifier Foundation (gleif.org). Chapter 5 1. Danny Sullivan, "A reintroduction to our Knowledge Graph and knowledge panels," Google The Keyword blog, May 2020. Source of the figures of 500 billion facts about five billion entities. 2. Web Data Commons project, "RDFa, Microdata, and Microformat Data Sets" (webdatacommons.org), annual analyses of structured data in the Common Crawl. Chapter 6 1. Peter Richardson and colleagues, "Baricitinib as potential treatment for 2019-nCoV acute respiratory disease," The Lancet 395, no. 10223 (February 2020). 2. EDM Council, Financial Industry Business Ontology (spec.edmcouncil.org/fibo). Chapter 7 1. Patrick Lewis and colleagues, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks," Advances in Neural Information Processing Systems 33 (2020). 2. Juan Sequeda, Dean Allemang and Bryon Jacob, "A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databases," arXiv:2311.07509 (November 2023). 3. Dean Allemang and Juan Sequeda, "Increasing the LLM Accuracy for Question Answering: Ontologies to the Rescue!," arXiv:2405.11706 (May 2024). 4. Darren Edge and colleagues, "From Local to Global: A Graph RAG Approach to Query-Focused Summarization," arXiv:2404.16130 (April 2024). 5. Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang and Xindong Wu, "Unifying Large Language Models and Knowledge Graphs: A Roadmap," IEEE Transactions on Knowledge and Data Engineering 36, no. 7 (2024). Chapter 8 1. Xin Luna Dong and colleagues, "Knowledge Vault: A Web-Scale Approach to Probabilistic Knowledge Fusion," Proceedings of KDD 2014. 2. Antoine Bordes, Nicolas Usunier, Alberto García-Durán, Jason Weston and Oksana Yakhnenko, "Translating Embeddings for Modeling Multi-relational Data," Advances in Neural Information Processing Systems 26 (2013). Further Reading Aidan Hogan and colleagues, "Knowledge Graphs," ACM Computing Surveys 54, no. 4 (2021). The most thorough single survey of the field, covering data models, schemas, identity, context, deduction, embeddings, construction and quality. Also published as a book by Morgan & Claypool in its Synthesis Lectures series and freely available online. Dean Allemang, James Hendler and Fabien Gandon, Semantic Web for the Working Ontologist, third edition (ACM Books, 2020). The clearest practical introduction to RDF, RDFS, OWL and SHACL, written for practitioners rather than logicians, with extensive attention to modelling patterns and their pitfalls. Natasha Noy and colleagues, "Industry-Scale Knowledge Graphs: Lessons and Challenges," Communications of the ACM (2019). A short, candid comparison of the graphs at Microsoft, Google, Facebook, eBay and IBM by people who built them. Essential for understanding where the real costs lie. Peter Christen, Data Matching: Concepts and Techniques for Record Linkage, Entity Resolution, and Duplicate Detection (Springer, 2012).** A comprehensive treatment of entity resolution, from standardisation and blocking to classification and evaluation. Its methods predate large language models but its framework remains the right one. Mayank Kejriwal, Craig Knoblock and Pedro Szekely, Knowledge Graphs: Fundamentals, Techniques, and Applications (MIT Press, 2021).** A textbook covering construction, completion and querying, with good chapters on information extraction and entity resolution. Shirui Pan and colleagues, "Unifying Large Language Models and Knowledge Graphs: A Roadmap," IEEE Transactions on Knowledge and Data Engineering (2024). A widely cited map of the ways graphs and language models are being combined, with extensive references to the research literature. Darren Edge and colleagues, "From Local to Global: A Graph RAG Approach to Query-Focused Summarization" (2024). The paper behind Microsoft's GraphRAG, worth reading in full for its description of the method and its evaluation design, and as a basis for judging later claims about graph-based retrieval. Wikidata documentation (wikidata.org). The help pages on the data model, property proposals, constraints and references are a practical education in how a large, long-lived graph handles identity, context and quality in public. Hashtags: #KnowledgeGraphs #ArtificialIntelligence #KnowledgeRepresentation #SemanticWeb #StructuredKnowledge #InformationArchitecture #GraphDatabases #OntologyEngineering #EntityResolution #EntityLinking #NamedEntityRecognition #InformationExtraction #RelationshipExtraction #ResourceDescriptionFramework #RDF #SPARQL #OWL #LinkedData #Wikidata #Neo4j #GraphQueryLanguages #KnowledgeGraphConstruction #KnowledgeGraphReasoning #SemanticSearch #EnterpriseKnowledgeGraphs #DataIntegration #DataProvenance #LargeLanguageModels #GraphRAG #ExplainableAI

Download the Book (PDF): Introduction A forty-tonne articulated truck leaving a distribution centre in the Rhine valley at five in the morning will, on a typical long-haul day, cover somewhere between 600 and 900 kilometres. A diesel version of that truck burns roughly a third of a litre of fuel for every kilometre and refuels in a few minutes at any of thousands of stations. Its engine is designed to run for well over a million kilometres. Nothing about this arrangement is glamorous, but it is extraordinarily well optimised, and it is the benchmark that every zero-emission alternative has to beat on cost, uptime and payload, not on aspiration. Heavy transport is where the decarbonisation of mobility gets hard. Trucks are a small share of vehicles on the road but a large share of road-transport emissions; in the European Union heavy-duty vehicles account for roughly a quarter of road transport's carbon dioxide. Ships move about four-fifths of world trade by volume and emit on the order of a billion tonnes of greenhouse gases a year. Both sectors run on energy-dense liquid fuels, both are cost-sensitive to the cent, and both operate assets for decades. Any replacement has to carry a great deal of energy, deliver it reliably for tens of thousands of hours, and be refuelled quickly in places that may be far from a strong electricity grid. Hydrogen, used in a proton exchange membrane fuel cell, was for many years the default answer to that problem. A fuel cell combines hydrogen with oxygen from the air to produce electricity, water and heat, with no combustion and no tailpipe pollution. Hydrogen carries about three times as much energy per kilogram as diesel. Refuelling a truck takes minutes, not hours. On paper, a hydrogen fuel cell truck behaves like a diesel truck that happens to be electric. The last few years have tested that picture severely. Battery-electric trucks have improved faster than almost anyone predicted, and in Europe in 2025 they outsold hydrogen trucks by a wide margin; hydrogen trucks took around a tenth of one percent of heavy commercial registrations. Nikola, once the most visible promoter of hydrogen trucking in the United States, filed for bankruptcy in February 2025. Hyzon and several European start-ups wound down. Retail hydrogen in California, the one large market with public stations, cost well over thirty dollars a kilogram in 2024, which is several times what a trucking fleet can afford. Daimler Truck, the most serious developer of liquid-hydrogen long-haul trucks, pushed series production from the late 2020s to after 2030 and will instead put a small series of one hundred trucks into customer hands from the end of 2026. At sea, the International Maritime Organization voted in October 2025 to delay adoption of its global Net-Zero Framework by a year, removing for now the carbon price that would have made clean fuels competitive. It would be easy to read these events as the end of the hydrogen truck and ship. That would be as wrong as the earlier enthusiasm. The engineering record of the same period is more interesting than the headlines. A fleet of 48 Hyundai XCIENT fuel cell trucks in Switzerland drove ten million kilometres in under four years. Daimler's liquid-hydrogen prototypes covered more than a thousand kilometres on a single fill and 225,000 kilometres in their first customer trial phase. A car ferry in a Norwegian fjord, the MF Hydra, has been running on liquid hydrogen since 2023. Toyota, the company with more fuel cell vehicles on the road than any other, released a third-generation system it says doubles durability, and agreed to take an equal stake in cellcentric, the fuel cell joint venture of Daimler Truck and Volvo. The technology works. The question is where it is worth using, and what has to be true for it to pay. The argument of this book The controlling claim of this book is simple to state: the proton exchange membrane fuel cell is no longer the main obstacle to zero-emission heavy transport. The stack has become a capable, manufacturable machine. What decides whether hydrogen succeeds in trucks and ships is the system around it, in three specific ways. First, durability under real heavy-duty duty cycles, which is a harder and different problem from the passenger car. Second, onboard storage, because hydrogen's poor volumetric energy density forces a choice between heavy high-pressure tanks and cryogenic liquid with its own penalties. Third, and most important, the delivered cost of the hydrogen itself, which is set by electricity prices, conversion losses and infrastructure utilisation rather than by anything inside the vehicle. Those three constraints sort transport applications into a clear pattern. Where batteries can carry the energy and the grid can deliver it, batteries win, because they waste far less of the original electricity. Where the energy required is very large, the duty is continuous, refuelling time matters and grid connection is weak or absent, hydrogen and its derivatives become the serious option. The engineering task is to know precisely where the boundary lies and to design for it, rather than to argue about whether hydrogen is good or bad in general. How the book is organised The chapters move from the inside of the cell outward. Chapter 1 explains how a fuel cell converts hydrogen into electricity: the half-reactions, the voltage that thermodynamics allows, and the losses that pull the real voltage down. Chapter 2 opens the stack itself, layer by layer, from the thin polymer membrane and its platinum catalyst to the bipolar plates, and explains why water and heat management are the daily preoccupation of every stack engineer. Chapter 3 is devoted to durability: what physically wears out in a fuel cell, why a truck that must run for 25,000 to 30,000 hours is so much harder than a car that runs for 5,000 to 8,000, and what designers do about it. Chapter 4 turns to the fuel tank. It compares compressed gas at 350 and 700 bar with liquid hydrogen at minus 253 degrees Celsius, explains the thermodynamics of boil-off and the energy cost of liquefaction, and examines the subcooled liquid refuelling process that Daimler and Linde developed. Chapter 5 puts the pieces together into a truck powertrain, sizing the fuel cell, battery and tanks against a real long-haul duty cycle and showing where heat rejection, parasitic loads and packaging constrain the design. Chapter 6 confronts the economics directly: the efficiency chain from electricity to wheel, the cost structure of delivered hydrogen, the business failures of the past few years and the comparison with battery-electric trucks that now dominates fleet decisions. Chapter 7 goes to sea, where the energy demands of shipping change the calculation, where liquid hydrogen competes with ammonia and methanol as carriers, and where the first commercial hydrogen ferries offer real operating experience. Chapter 8 deals with safety, codes and the industrial path to scale, including the properties of hydrogen that make it hazardous in some ways and forgiving in others, and the standards that govern tanks, refuelling and ships. The Conclusion sets out what follows from the whole argument for engineers, fleet operators and policymakers deciding where hydrogen belongs. A note on numbers Hydrogen is a field in which numbers are routinely quoted without their basis, and the confusion that results has cost investors a great deal of money. Throughout this book energy is given on the lower heating value basis unless stated otherwise: one kilogram of hydrogen contains about 33.3 kilowatt-hours of usable chemical energy, roughly 120 megajoules. Efficiencies are stated with their boundaries, because a fuel cell that is 60 percent efficient at the stack may be 50 percent efficient as a system and far less than that measured from the electricity that made the hydrogen. Where the book works through an example with assumed values, the assumptions are stated so that the reader can change them and see what happens. That habit, more than any single figure, is what separates sound hydrogen engineering from salesmanship. Chapter 1: The Electrochemical Engine Every heat engine, from a steam locomotive to a modern diesel, works by the same broad route. Fuel burns, the hot gases expand, and some fraction of that thermal energy is turned into mechanical work. The fraction is capped by the Carnot limit, which depends on the temperatures between which the engine operates, and real engines fall well short of even that cap. A heavy-duty diesel at its best operating point converts about 45 to 47 percent of its fuel energy to shaft work, an achievement of more than a century of refinement. A fuel cell takes a different route. It never burns the fuel. Instead it separates the chemical reaction between hydrogen and oxygen into two halves, carried out at two different electrodes, and forces the electrons that pass between them to travel through an external circuit where they can do useful work. There is no flame, no expansion stroke and no Carnot limit. The ceiling on efficiency is set by thermodynamics of a different kind: the ratio between the energy that the reaction can deliver as electrical work and the total energy it releases. Understanding that ratio, and the losses that keep real cells below it, is the foundation for everything that follows in this book. Splitting the reaction in two The overall reaction in a hydrogen fuel cell is the same one that happens when hydrogen burns: two molecules of hydrogen combine with one molecule of oxygen to form two molecules of water. In a flame the electrons are exchanged directly between colliding molecules, and the energy appears immediately as heat. In a fuel cell the exchange is arranged so that it can only happen indirectly. At the anode, hydrogen gas arrives at a catalyst surface, almost always platinum dispersed as tiny particles on carbon. Each hydrogen molecule splits into two protons and two electrons. This hydrogen oxidation reaction is fast on platinum; it is one of the easiest reactions in electrochemistry and needs very little catalyst to run at high rates. The protons and electrons then go different ways. The electrons cannot pass through the membrane that separates the two electrodes, because the membrane is an electrical insulator. They leave through the anode's electrically conductive layers, travel around the external circuit, through the truck's motor or the ship's power electronics, and return to the other electrode. The protons pass directly through the membrane, which is a solid polymer engineered to conduct protons and almost nothing else. That membrane gives the technology its name: proton exchange membrane, or equivalently polymer electrolyte membrane, both abbreviated PEM. At the cathode, oxygen from the air arrives at another platinum catalyst surface. There it meets the protons coming through the membrane and the electrons coming around the circuit, and the three combine to form water. This oxygen reduction reaction is the slow, difficult half of the process. Breaking the strong double bond of the oxygen molecule and passing four electrons to it in sequence is a multi-step process with sluggish kinetics, even on platinum, the best practical catalyst known. Most of the platinum in a fuel cell sits on the cathode for this reason, and much of the research of the past three decades has aimed at making this one reaction faster. The elegance of the arrangement is that it converts chemical energy into electrical energy in a single step. There are no moving parts in the cell itself. The only mechanical components in a fuel cell system are the pumps, compressors, fans and valves that supply reactants and remove heat and water. How much voltage thermodynamics allows The maximum electrical work a reaction can deliver is given by the change in Gibbs free energy. For the hydrogen-oxygen reaction producing liquid water at 25 degrees Celsius and atmospheric pressure, that maximum corresponds to a cell voltage of about 1.23 volts. This is the reversible or open-circuit potential: the voltage a perfect cell would show with no current flowing and no losses. The total energy released by the reaction, the enthalpy change, is larger. If all of it were turned into electricity, the cell would show about 1.48 volts when the product water is liquid, or about 1.25 volts when the water leaves as vapour. These are not voltages any cell achieves; they are useful bookkeeping numbers that convert voltages into efficiencies. The difference between the enthalpy and the free energy is the heat that even an ideal cell must reject, a consequence of the entropy change of the reaction. This yields the most useful rule of thumb in fuel cell engineering. The efficiency of a hydrogen fuel cell, on the lower heating value basis used throughout this book, is approximately its operating voltage divided by 1.25 volts, provided all the hydrogen supplied is consumed. A cell operating at 0.75 volts is about 60 percent efficient. A cell at 0.625 volts is about 50 percent efficient. A cell at 0.5 volts is about 40 percent efficient. Everything the cell does not deliver as electricity appears as heat, and that heat must be removed. The rule has a striking consequence. Unlike a diesel engine, which is least efficient at light load and most efficient somewhere near its rated torque, a fuel cell is most efficient at low current and loses efficiency as it is pushed harder. At very light load a cell can exceed 65 percent stack efficiency. At its rated power it may be at 45 percent or below. This behaviour shapes the entire design of fuel cell vehicles, because it means a designer can buy efficiency by installing more cell area than strictly needed and running each cell at a gentler point, at the cost of more platinum, more membrane and more money. The polarization curve and its three losses The relationship between cell voltage and current density is called the polarization curve, and it is the single most important performance description of a fuel cell. Current density is the current per unit of active area, usually expressed in amperes per square centimetre. A modern automotive cell might deliver 1.5 to 2.5 amperes per square centimetre at its rated point, with a voltage around 0.6 to 0.7 volts. Starting from about 1.23 volts in theory, the measured open-circuit voltage of a real cell is already lower, typically 0.95 to 1.0 volts, because a small amount of hydrogen permeates through the membrane and reacts directly with oxygen on the cathode, and because of side reactions on the platinum surface. From there, three distinct losses reduce the voltage as current rises. The first is activation loss, which dominates at low current. Every electrochemical reaction needs some driving voltage to proceed at a useful rate, and the sluggish oxygen reduction reaction needs a lot of it. Even at small current densities the cathode surrenders around 0.3 to 0.4 volts to activation. The loss grows only logarithmically with current, so it produces a sharp initial drop in the polarization curve followed by a much gentler slope. Activation loss is attacked by better catalysts, by more catalyst surface area, and by higher temperature, all of which speed the reaction. The second is ohmic loss, which dominates in the middle of the curve. The membrane resists proton flow, and the electrodes, plates and contacts resist electron flow. The voltage lost is proportional to current, so this region of the curve is nearly a straight line. The membrane usually contributes most of it, and its resistance depends heavily on how wet it is. A well-hydrated thin membrane has an area-specific resistance of a few tens of milliohm-square-centimetres; a dried-out one can be several times worse. This is why water management, the subject of much of the next chapter, matters so directly for performance. The third is mass transport loss, which dominates at high current. To react, oxygen has to travel from the air channel through the porous gas diffusion layer and the catalyst layer to the platinum surface, and the product water has to travel the other way. At high current the cell consumes oxygen faster than diffusion can supply it, and the local concentration at the catalyst falls. Liquid water accumulating in the pores makes this worse by blocking the gas pathways. The result is a steep fall in voltage at the end of the curve, which effectively sets the maximum useful current density. Much of the improvement in recent automotive stacks has come from pushing this wall outward, with thinner and more open electrode structures and with flow field designs that sweep water out more effectively. A worked example: from cell to stack A single cell produces well under one volt, which is too little for any practical purpose. Cells are therefore stacked in series, one after another, so that their voltages add. Consider a stack designed to deliver 150 kilowatts of gross electrical power, close to the size of each of the two modules in Daimler's GenH2 truck. Suppose the designer chooses a rated operating point of 0.65 volts at 1.6 amperes per square centimetre. Each square centimetre of active area then delivers about 1.04 watts. To produce 150 kilowatts requires about 144,000 square centimetres of active area in total, or 14.4 square metres. If each cell has an active area of 300 square centimetres, a typical size for automotive designs, the stack needs about 480 cells. At 0.65 volts each, the stack voltage at rated power is about 312 volts and the current about 480 amperes. The efficiency at that point is 0.65 divided by 1.25, or 52 percent at the stack. The heat produced is the hydrogen energy consumed minus the electricity delivered. At 52 percent efficiency, 150 kilowatts of electricity requires about 288 kilowatts of hydrogen energy, so roughly 138 kilowatts of heat must be removed, nearly as much heat as electricity. The hydrogen consumption at this point is 288 kilowatts divided by 33.3 kilowatt-hours per kilogram, or about 8.7 kilograms per hour. Now suppose the same stack is run at a lighter point of 0.75 volts and 0.6 amperes per square centimetre. Power falls to about 65 kilowatts, but efficiency rises to 60 percent and heat rejection falls to about 43 kilowatts. This illustrates why a truck that cruises on the motorway at 70 to 100 kilowatts of average traction power benefits greatly from a stack that is generously sized relative to that average: the cells spend most of their time on the efficient part of the curve. Several practical complications refine this picture. Not all the hydrogen supplied reacts; small amounts are purged periodically to remove nitrogen and water that diffuse across into the anode, and typical fuel utilisation is in the range of 97 to 99 percent. The air must be supplied in excess of the stoichiometric requirement, usually by a factor of about 1.5 to 2, so that oxygen concentration does not collapse near the air outlet. And the stack's gross power is not what the vehicle receives, because a compressor, pumps and fans consume a share of it. Those parasitic loads, discussed in Chapter 5, typically take 10 to 15 percent of gross stack power at rated load, reducing system efficiency correspondingly. Pressure, temperature and the case for running hot Two operating variables, pressure and temperature, shape the polarization curve in ways that matter directly for vehicle design, and both involve trade-offs between the cell and the machinery that serves it. Raising the pressure of the reactant gases raises the reversible voltage slightly, by an amount described by the Nernst equation. The effect on the thermodynamic voltage is small: doubling the oxygen partial pressure raises it by only about 5 millivolts at typical operating temperatures, and doubling the hydrogen pressure by about 10 millivolts. The larger benefit lies elsewhere. Higher oxygen partial pressure at the cathode speeds the sluggish oxygen reduction reaction, cutting activation losses, and it raises the concentration of oxygen that can be driven through the porous electrode, pushing the mass transport wall to higher current. A cell operated with air at 2.5 bar absolute rather than 1.5 bar can deliver substantially more current at the same voltage near its rated point. That is why heavy-duty systems use compressors rather than simple blowers. The cost is the work of compression. Compressing air from atmospheric pressure to 2.5 bar takes roughly 100 kilojoules per kilogram of air with a realistic compressor efficiency, and a 150-kilowatt stack at a stoichiometric ratio of 2 needs on the order of 0.15 to 0.2 kilograms of air per second at full power. The compressor therefore draws roughly 15 to 20 kilowatts at rated load unless part of the energy is recovered by an expander on the exhaust. The designer chooses an operating pressure that balances the gain in stack output against this parasitic draw, and the best choice differs between full load, where high pressure pays, and part load, where it does not. Modern systems vary pressure with load for this reason. Temperature presents a similar balance. Higher temperature speeds both electrode reactions, improves the conductivity of a properly humidified membrane and, most important in vehicles, makes heat easier to reject. But the thermodynamic voltage falls slightly with temperature, the membrane dries more easily, and degradation mechanisms accelerate. Low-temperature PEM cells for cars typically run with coolant around 65 to 80 degrees Celsius. Heavy-duty developers are pushing towards 90 to 95 degrees Celsius, and research programmes target membranes that tolerate operation above 100 degrees, because, as later chapters show, rejecting heat at a temperature not far above a summer afternoon is one of the hardest problems a fuel cell truck faces. These interactions illustrate a theme that runs through the rest of this book. The cell, viewed alone, is a simple device with well-understood physics. Its performance in a vehicle depends on an operating point negotiated between the cell and its supporting machinery, with each choice of pressure, temperature, humidity and current density shifting cost, efficiency and durability in different directions. Why PEM, and not another fuel cell Several families of fuel cell exist, distinguished chiefly by their electrolyte and operating temperature. Alkaline fuel cells powered the Apollo spacecraft. Phosphoric acid fuel cells have run in stationary power plants for decades. Molten carbonate and solid oxide fuel cells operate at 600 to 1,000 degrees Celsius and can use fuels other than pure hydrogen, including natural gas and ammonia. For road vehicles, the PEM cell won decisively, for reasons that follow directly from its physics. It operates at around 60 to 90 degrees Celsius, so it can start quickly and respond to changes in load within seconds. Its solid membrane can be made very thin, giving high power density; recent automotive stacks exceed four kilowatts per litre. It tolerates vibration and orientation changes. The price paid is a need for platinum catalysts, which the low temperature makes necessary, a requirement for very pure hydrogen, because carbon monoxide at even a few parts per million poisons the anode catalyst, and the difficulty of rejecting large amounts of heat at a low temperature, a problem that becomes serious in a truck. The PEM cell's history explains its current maturity. General Electric developed the first practical PEM cells in the early 1960s, and they flew on NASA's Gemini missions. DuPont's Nafion, a perfluorosulfonic acid polymer still at the heart of most membranes, appeared in the same decade. The Canadian company Ballard Power Systems revived the technology for vehicles in the 1980s and 1990s, and the automotive industry spent the following two decades driving down cost and platinum content. Toyota's Mirai, launched in 2014 and redesigned in 2020 with a 128-kilowatt stack, demonstrated that a PEM system could be mass-produced to automotive standards. The heavy-duty systems now entering trucks and ships draw directly on that lineage. At sea the competition is more open. Solid oxide fuel cells, with electrical efficiencies that can exceed 55 to 60 percent and an ability to run on ammonia or methanol, are being developed for ships where slow load changes and steady running suit them. High-temperature PEM variants using phosphoric-acid-doped membranes, running at 160 to 180 degrees Celsius, tolerate impurities better and reject heat more easily, at the cost of lower power density. These options matter in Chapter 7. For the trucks that form the main subject of this book, the low-temperature PEM cell is the technology, and its interior is the subject of the next chapter. Chapter 2: Inside the Stack Seen from outside, a fuel cell stack is an unremarkable metal box with pipes for hydrogen, air and coolant and two heavy electrical terminals. Inside, it is a repeating sandwich of layers, several hundred times over, each layer a feat of materials engineering measured in micrometres. The performance and life of a fuel cell vehicle depend on what happens inside a structure about a millimetre thick, repeated hundreds of times and compressed between end plates. This chapter takes that structure apart, from the centre outward, and then examines the two problems that dominate a stack engineer's working life: water and heat. The membrane electrode assembly At the centre of each cell is the membrane electrode assembly, usually abbreviated MEA. It contains the membrane, the two catalyst layers on either side of it, and, in most definitions, the two gas diffusion layers that sandwich the whole. The membrane is a sheet of polymer typically 8 to 20 micrometres thick in current automotive designs, thinner than a sheet of household cling film. Its standard chemistry is perfluorosulfonic acid, a polymer whose backbone resembles PTFE, the material of non-stick pans, with side chains ending in sulfonic acid groups. The backbone gives chemical and mechanical robustness. The side chains cluster together when the membrane absorbs water, forming a network of nanometre-scale hydrated channels through which protons hop from one water molecule to the next. The membrane conducts protons well only when it is wet, which is the central fact of PEM water management. Thinner membranes have lower resistance and allow water to move more easily between anode and cathode, helping keep both sides hydrated. They are also weaker, more prone to pinholes and more permeable to hydrogen crossing directly to the cathode. Modern membranes resolve this tension with reinforcement: a thin porous layer of expanded PTFE embedded within the ionomer, which carries mechanical loads and limits swelling. Manufacturers such as Gore, Chemours and several Japanese and Chinese suppliers produce reinforced membranes at industrial scale. Most heavy-duty membranes are made somewhat thicker than those in passenger cars, trading a little performance for life, because the truck must run several times longer. On each side of the membrane sits a catalyst layer about 5 to 15 micrometres thick. It is a porous composite of platinum nanoparticles, typically 2 to 5 nanometres across, supported on carbon particles and bound together with a thin film of the same proton-conducting ionomer as the membrane. The structure has to do three things at once. It must conduct electrons through the carbon, conduct protons through the ionomer, and let gas reach the platinum through the pores. A reaction can only occur where all three pathways meet at a platinum surface, the so-called triple-phase boundary. Designing catalyst layers is largely a matter of maximising those meeting points while keeping the pores open enough for oxygen to get in and water to get out. Platinum loading is the most watched number in the field because platinum is expensive and supply is concentrated, mainly in South Africa and Russia. Early fuel cells used several milligrams per square centimetre. Automotive stacks now use a few tenths of a milligram per square centimetre in total, with the U.S. Department of Energy's light-duty target at 0.125 milligrams. Heavy-duty designs typically carry more, around 0.3 to 0.5 milligrams per square centimetre, deliberately, because extra platinum provides a reserve against the gradual loss of active surface area over a long life. Using the stack example from Chapter 1, 14.4 square metres of active area at 0.4 milligrams per square centimetre amounts to about 58 grams of platinum per 150-kilowatt stack. A truck with two such stacks carries more than 100 grams, which explains why platinum thrift and platinum recycling are taken seriously in heavy-duty programmes. Research into catalysts has proceeded on two fronts. One is making platinum more active, by alloying it with cobalt or nickel to alter its surface electronic structure, by shaping it into particular crystal facets, or by building core-shell particles with platinum only on the outside where it can work. Toyota's second-generation Mirai stack used a platinum-cobalt alloy catalyst. The other front is eliminating platinum altogether, using iron and nitrogen embedded in carbon, which has reached impressive activities in laboratory cells but still falls short on durability for vehicle use. For the trucks being built this decade, platinum alloy catalysts on engineered carbon supports are the reality. Gas diffusion layers and bipolar plates Outside each catalyst layer is the gas diffusion layer, a sheet of carbon fibre paper or felt about 150 to 250 micrometres thick. It distributes gas from the channels evenly across the catalyst, conducts electrons and heat, and provides mechanical support. It is treated with PTFE to make it partly water-repellent, and usually coated on the catalyst side with a microporous layer of carbon black and PTFE. That microporous layer has an important job: it controls how liquid water leaves the catalyst layer, preventing it from flooding the pores and helping to keep the membrane side moist. The bipolar plates enclose each cell. They are called bipolar because one face serves as the anode of one cell and the other face as the cathode of the next, connecting the cells electrically in series. Each plate has fine channels machined, moulded or stamped into its surfaces to distribute hydrogen on one side and air on the other, and internal passages through which coolant flows. The plates conduct the full stack current, separate the gases, and carry away much of the heat. Two material families compete. Graphite composite plates are corrosion-resistant and durable but relatively thick and brittle. Stamped stainless steel or titanium plates, around 0.1 millimetres thick and formed in pairs that are welded together, are much thinner and allow higher volumetric power density; they require protective coatings to prevent corrosion in the acidic, oxidising environment of the cell and to keep electrical contact resistance low. Most passenger-car stacks have moved to coated metal plates. Heavy-duty developers have used both, and the choice embodies a recurring heavy-duty trade-off: metal plates are compact and suited to mass production, while graphite plates have a long record of surviving tens of thousands of hours in buses. The geometry of the flow field matters as much as the material. Straight parallel channels have low pressure drop but tend to let water collect. Serpentine channels force the gas along a single winding path and clear water well but cost compressor power. Toyota's first Mirai introduced a three-dimensional fine-mesh flow field on the cathode that pushes air towards the catalyst and lifts water away, one reason it could dispense with an external humidifier. Every stack maker treats its flow field geometry as a closely held asset. The cell pile is held together by end plates and tie rods or bands that apply a carefully controlled compressive load. Too little and contact resistance rises and gases leak; too much and the gas diffusion layers are crushed and the membranes damaged. Gaskets around each cell keep hydrogen, air and coolant apart. A stack of 400 cells has some 1,200 or more sealed interfaces, and seal failure is one of the more mundane but real causes of stack end-of-life. Water: too little, too much Water management is where fuel cell engineering becomes an art. The membrane needs water to conduct protons, so it must not dry out. Yet the cell generates water at the cathode, and if liquid water accumulates in the catalyst layer, the gas diffusion layer or the channels, oxygen cannot reach the platinum and performance collapses. The engineer has to hold the cell in a narrow window between drying and flooding, across a wide range of loads, temperatures and ambient conditions. Several mechanisms move water inside the cell. Protons passing through the membrane drag water molecules with them from anode to cathode, a phenomenon called electro-osmotic drag, which tends to dry the anode side at high current. Water produced at the cathode diffuses back towards the anode because of the concentration difference, partly counteracting the drag. Pressure differences between the anode and cathode push water one way or the other. And the gas streams carry water vapour out of the cell, at a rate that depends steeply on temperature: air at 80 degrees Celsius can hold roughly twice as much water vapour as air at 65 degrees. That temperature sensitivity is the crux of the problem. A designer would like to run the stack hot, because higher temperatures speed the reactions and make heat rejection easier. But at higher temperatures the incoming air can carry away more water than the cell produces, drying the membrane. Many systems therefore humidify the incoming air, usually with a passive membrane humidifier that transfers moisture from the wet exhaust air to the dry inlet air. Others recirculate hydrogen through the anode with a blower or ejector, which returns water to the anode side. Designs that do away with external humidification, like the Mirai's, depend on thin membranes and carefully engineered internal water transport to stay hydrated. A worked water balance shows how narrow the operating window is. Return to the 150-kilowatt stack of Chapter 1, with about 480 cells carrying 480 amperes. Each pair of electrons passing through a cell corresponds to one molecule of hydrogen consumed and one molecule of water produced. Across the whole stack this amounts to about 1.2 moles of water per second, or roughly 21 grams per second, some 77 kilograms per hour. A long-haul fuel cell truck produces several hundred litres of pure water on a typical day, which it discharges as vapour and droplets. Now consider how much water vapour the exhaust air can carry away. At a stoichiometric ratio of 2, the stack draws about 1.2 moles of oxygen per second, which arrives with about 4.5 moles of nitrogen, so roughly 5.1 moles of dry gas leave the cathode each second once half the oxygen has been consumed. Suppose the cathode outlet is at 2 bar absolute. At 80 degrees Celsius, the saturation vapour pressure of water is about 47 kilopascals, so saturated exhaust gas could carry about 1.6 moles of water per second, roughly 28 grams. That exceeds the 21 grams per second being produced: if the inlet air were dry, the cell would lose water faster than it makes it and the membrane would dry out. At 70 degrees Celsius, the saturation pressure falls to about 31 kilopascals, the exhaust can carry only about 17 grams per second, and the surplus of roughly 4 grams per second must leave as liquid, with the risk of flooding. In other words, a shift of ten degrees moves the cell from a drying regime to a flooding regime at constant pressure and air flow. The engineer's levers are the outlet pressure, which reduces the vapour-carrying capacity of the exhaust as it rises, the air stoichiometry, which increases it, the inlet humidity, and the temperature itself. A system controller continuously juggles these levers as load and ambient conditions change, which is why water management is described as a control problem as much as a materials problem. Liquid water has a further hazard: freezing. A truck parked overnight in a Scandinavian winter must start at minus 30 degrees Celsius. If liquid water remains in the catalyst layers or channels when the stack shuts down, it freezes, can block gas access on the next start and, through repeated cycles, damage the electrode structure. Practical systems purge water from the stack at shutdown when freezing is expected, and use carefully controlled start-up strategies in which the stack heats itself quickly by operating at low voltage, generating more heat per unit of hydrogen until the cells are above freezing. Modern automotive systems start reliably from minus 30 degrees Celsius; demonstrating that capability repeatedly over many years in a truck is part of what heavy-duty validation is about. Heat: the hidden design constraint The worked example in Chapter 1 showed that a stack delivering 150 kilowatts of electricity generates nearly as much heat. That is not unusual in itself; a diesel engine of comparable output produces more waste heat. The difference lies in where the heat goes and at what temperature. A diesel engine sends a large share of its waste heat, typically around a third of the fuel energy, out of the exhaust pipe as hot gas. Only part goes into the coolant and through the radiator. A PEM fuel cell's exhaust is warm, humid air at below 100 degrees Celsius, which carries away only a small share of the heat. Almost all the waste heat must go through the coolant to the radiator. Worse, the coolant temperature is limited to around 75 to 90 degrees Celsius, compared with 100 degrees or more for a diesel, so the temperature difference between coolant and ambient air, which drives heat transfer, is smaller. On a hot day with 40-degree ambient air, a fuel cell radiator has perhaps 40 to 50 degrees of driving temperature difference to work with, where a diesel's has 60 or more. Since radiator heat rejection scales roughly with area times temperature difference, a fuel cell truck needs a radiator two to three times the size of a diesel's, or more airflow, or both. The front of a European cab-over truck has limited room for radiators, and fans consume power. This is one of the most important practical constraints on heavy-duty fuel cell systems, and it explains why sustained full-power climbs on hot days, such as a loaded truck on a long alpine grade in summer, can force a fuel cell system to derate. It also explains why heavy-duty developers push hard for stacks that tolerate higher temperatures, towards 95 or even 105 degrees Celsius, and why the high-temperature membranes discussed in Chapter 7 attract interest in maritime applications. The coolant itself is unusual. Because it flows through passages in the bipolar plates, in electrical contact with cells at different voltages, it must be nearly non-conductive. Fuel cell systems use deionised water and glycol mixtures with ion-exchange filters that continuously remove dissolved ions; ordinary engine coolant would short-circuit the stack and corrode it. Making stacks at scale The final element of the stack story is manufacturing. A fuel cell stack is not a bespoke machine; it is a product with hundreds of nearly identical parts, and its cost depends on how fast and how consistently those parts can be made. Membranes and catalyst-coated membranes are produced by roll-to-roll coating. Bipolar plates are stamped, coated and laser-welded. Stacks are assembled by robots, and every cell must be leak-tested and electrically checked, because one weak cell limits the whole series string. Cost studies for the U.S. Department of Energy have repeatedly shown that stack cost falls steeply with production volume: membrane, catalyst application and plate manufacturing all have large fixed costs that are spread thinly at high volume. At a few thousand systems a year, a heavy-duty system can cost several times what it would at a hundred thousand. This creates the chicken-and-egg problem familiar from all new vehicle technologies, made harder for hydrogen trucks because the fuel infrastructure faces the same problem at the same time. The Department of Energy's targets for long-haul truck fuel cell systems, discussed in the next chapter, are therefore paired: a cost of 80 dollars per kilowatt and a durability of 25,000 hours by 2030, with ultimate goals of 60 dollars per kilowatt and 30,000 hours. Meeting either target alone is useless. A cheap stack that wears out is not cheap, and a durable stack that costs too much will not be bought. The industrial base for heavy-duty stacks is consolidating around a small number of players with automotive-scale ambitions: Toyota, Hyundai and Honda among car makers; cellcentric, owned by Daimler Truck and Volvo and with Toyota agreed in 2026 as a third equal partner; Ballard, the long-time leader in buses and the supplier for the MF Hydra; and a group of Chinese manufacturers supplying the largest fuel cell truck fleet in the world, concentrated in demonstration city clusters supported by national policy. The technical paths they follow are converging. The harder problem, which separates heavy-duty stacks from automotive ones, is how long they last. Chapter 3: Durability, or Why Trucks Are Harder Than Cars A passenger car is a lightly used machine. The average car in Europe or the United States spends more than 95 percent of its life parked, and over a fifteen-year life its engine runs for perhaps 5,000 to 8,000 hours. A fuel cell designed for such a car needs to survive that many hours with acceptable loss of performance, and automotive developers have reached that standard. Toyota and Hyundai have sold fuel cell cars for a decade without widespread stack failures. A long-haul truck lives a different life. It may run ten to twelve hours a day, five or six days a week, covering 100,000 to 150,000 kilometres a year, and the operator expects the powertrain to last a million kilometres or more. At an average speed of 60 kilometres per hour, a million kilometres is roughly 17,000 hours of driving, before idling, loading and auxiliary operation are counted. This is why the U.S. Department of Energy set a durability target of 25,000 hours for heavy-duty fuel cell systems by 2030 and an ultimate goal of 30,000 hours, measured to the point at which the system has lost 10 percent of its rated power. It is three to five times the passenger car requirement, demanded of a machine that also has to carry more load. The good news is that buses have already shown it can be done in principle. Several fuel cell bus fleets in California and Europe have logged individual stacks beyond 25,000 and even 30,000 operating hours. The caution is that city buses run at moderate power with predictable duty cycles and generous cooling, and many of those stacks carried more platinum than a truck economics model can afford. The job of this chapter is to explain what actually wears out in a PEM fuel cell, why truck operation is more demanding, and how designers fight back. What wears out A fuel cell has no pistons, rings or bearings in the stack. It does not wear by friction. It degrades by chemistry and by slow mechanical fatigue, and the degradation is driven less by how many hours it runs than by what conditions it sees during those hours. Voltage, temperature, humidity and their rates of change matter far more than the clock. A summary of the main mechanisms, the conditions that trigger them and the standard countermeasures is set out in Table 1, after which each is examined in turn. Table 1. Principal degradation mechanisms in PEM fuel cell stacks. Mechanism Main trigger Effect on cell Typical mitigation Platinum dissolution and growth Voltage cycling, high idle voltage Lost catalyst area, lower efficiency Voltage limits, alloy catalysts, extra loading Carbon support corrosion Start-stop, fuel starvation Electrode collapse, flooding Purge strategies, graphitised carbon, oxygen-evolution catalysts Membrane chemical attack Radicals at open circuit, low humidity Thinning, pinholes, hydrogen crossover Cerium radical scavengers, avoid idle at open circuit Membrane mechanical fatigue Humidity and temperature swings Cracks, crossover Reinforced membranes, gentle humidity control Contamination Impure hydrogen or polluted air Blocked catalyst sites Fuel specification, cathode air filters The first mechanism is loss of platinum surface area. The tiny platinum nanoparticles on the cathode are thermodynamically unstable. When the cathode voltage cycles up and down, as it does every time the vehicle accelerates and decelerates, platinum atoms dissolve at high voltage and redeposit at low voltage. Small particles shrink and large particles grow, a process akin to Ostwald ripening. Some dissolved platinum migrates into the membrane and precipitates there, where it is lost entirely. The total mass of platinum may change little, but the surface area available for reaction falls, sometimes by half over a life. Because activation losses depend on catalyst area, cell voltage falls and efficiency with it. The rate of loss rises steeply when the cathode spends time above about 0.85 volts, so the single most effective countermeasure is never to let cells sit at high voltage. Control systems clamp the minimum load on the stack, using the battery to absorb any surplus, so that cell voltages stay below a ceiling during idle and light cruising. The second mechanism, carbon corrosion, is more violent. The carbon that supports the platinum oxidises to carbon dioxide at high potentials, and those potentials occur in a specific circumstance: when hydrogen and air are both present in the anode, as happens briefly at every start-up and shut-down if no precautions are taken. A hydrogen-air front passing through the anode creates a reverse-current situation in which parts of the cathode are driven well above 1.4 volts, where carbon corrodes rapidly. A few thousand unmitigated start-stop events can destroy an electrode, collapsing its porous structure and leaving it prone to flooding. Local fuel starvation, where one cell runs short of hydrogen because of water blocking a channel, produces a similar effect on the anode, forcing the cell to oxidise its own carbon to sustain the current. The countermeasures include controlled purge sequences that minimise the duration of the hydrogen-air front, shorting resistors that pull cell voltages down at shutdown, more corrosion-resistant graphitised carbon supports, and catalysts on the anode that preferentially split water rather than consuming carbon when starvation occurs. The third mechanism is chemical attack on the membrane. Small amounts of hydrogen and oxygen permeate the membrane, and where they meet on the platinum they can form hydrogen peroxide. In the presence of trace metal ions, especially iron, peroxide decomposes into hydroxyl radicals, among the most aggressive oxidants in chemistry. These radicals attack the polymer, gradually thinning the membrane and releasing fluoride, which is measured in the product water as an indicator of degradation. Attack is worst at open-circuit or near open-circuit voltage, at high temperature and at low humidity. Modern membranes include cerium or manganese ions that scavenge radicals before they do harm, and this single addition has extended membrane chemical life by an order of magnitude. The fourth mechanism is mechanical fatigue of the membrane. A membrane swells by tens of percent in volume when it absorbs water and shrinks when it dries. Every cycle of humidity, which in a vehicle can mean every load change, puts the membrane through stress, particularly at the edges where it is clamped. Over thousands of cycles cracks and pinholes develop. Once a pinhole forms, hydrogen crosses directly to the cathode, reacts locally, creates a hot spot and enlarges the hole, a failure that can propagate quickly. Reinforced membranes and control strategies that limit humidity swings are the defences. The fifth is contamination. Carbon monoxide and hydrogen sulphide in the fuel, even at parts per million or less, adsorb on the platinum. The international fuel quality standard for vehicles, ISO 14687, sets carbon monoxide at no more than 0.2 parts per million and total sulphur compounds at 0.004 parts per million for this reason. The air side faces sulphur dioxide, nitrogen oxides and ammonia from traffic and agriculture; trucks running behind diesel traffic in tunnels breathe dirty air. Chemical air filters protect the cathode, and some contaminant damage is reversible by controlled operation, but the fuel specification makes hydrogen purity a cost item for every supplier. Why truck duty cycles are harsher Each of these mechanisms is triggered by conditions that a long-haul truck experiences more often or more intensely than a car. The first issue is sustained high power. A car rarely needs more than a quarter of its rated power, and spends most of its driving at a small fraction of it. A loaded 40-tonne truck needs around 90 to 120 kilowatts at the wheels and auxiliaries merely to hold 80 kilometres per hour on a level motorway, because air and rolling resistance on a heavy vehicle are large, and several hundred kilowatts on a long climb. The stack spends far more time at high current, where temperatures are higher and heat removal harder, and where humidity can swing as the cooling system struggles. The second issue is the economics of platinum and stack sizing. The obvious way to make cells last longer is to give them a gentler life: more cells, more area, more platinum, each running at lower current and higher efficiency. This works, but every additional square metre of active area costs money and space. A truck designer is caught between durability, efficiency and capital cost in a way a car designer is not, because the car can simply be over-designed relative to its light usage. The third issue is the number of transients. A truck in regional distribution may make dozens of stops a day, each with acceleration, braking and idling. Each acceleration pushes cell voltage down and each deceleration lets it rise; each idle raises it towards the dangerous zone above 0.85 volts. Over a 25,000-hour life the stack may see hundreds of thousands of load cycles. The fourth issue is the environment: dust, diesel soot from neighbouring traffic, winter road salt in the air intake and cold starts in northern climates. Each adds a little to the degradation budget. Fighting degradation by design The key insight of the past decade of heavy-duty development is that durability is largely a system and control problem rather than a materials problem. The same stack can last 10,000 hours or 30,000 hours depending on how it is operated, and the battery is the tool that makes gentle operation possible. In a fuel cell truck the battery buffers the stack from the road. When the driver accelerates, the battery supplies the sudden demand, and the stack ramps up its output at a controlled rate. When the driver lifts off, the battery absorbs regenerative braking energy and any surplus from the stack, which can continue at a moderate level instead of dropping to idle. On long descents, where a diesel uses its engine brake, the fuel cell truck uses its motors as generators and its battery as a sink; once the battery is full, it dumps the energy as heat through resistors in the cooling system. The stack's operating point thus moves slowly within a preferred window, avoiding both high voltage at light load and very high current at peak load. Operating strategies are refined further by several techniques. Stacks may be divided into two or more modules, as in Daimler's GenH2 with two 150-kilowatt systems and Hyundai's XCIENT with two 90-kilowatt systems, so that at light load one module can run at a good operating point while the other is shut down or idled, and so that operating hours can be shared between modules. Shut-down and start-up sequences are designed to purge the anode with hydrogen or nitrogen-rich gas, minimising carbon corrosion events, and the number of full shutdowns is limited. Cell voltage monitoring, at least of groups of cells, detects early signs of flooding or starvation so the controller can intervene. Recovery procedures form a less obvious part of the toolkit. Some of the voltage lost during operation is not permanent damage but reversible: platinum surfaces become covered with oxides at high voltage, and contaminants accumulate. Periodically running the cathode at low voltage, or briefly starving it of air, strips these oxides and recovers part of the lost performance. Truck control software can schedule such recovery cycles during natural pauses in operation. A durability budget, worked through Engineers make durability tractable by treating it as a budget. The end-of-life criterion is usually defined as a 10 percent loss of rated power. Because a fuel cell's power at a given current is proportional to its voltage, a 10 percent loss of power at rated current corresponds roughly to a 10 percent loss of cell voltage at that current. For a cell rated at 0.65 volts, the whole lifetime budget is therefore about 65 millivolts. Spread evenly over 25,000 hours, 65 millivolts amounts to an average decay rate of 2.6 microvolts per hour. Degradation in the fuel cell literature is routinely reported in microvolts per hour, and this conversion explains why: it allows a laboratory test or a fleet measurement to be compared directly with the lifetime target. A stack that loses voltage at 5 microvolts per hour in a given duty cycle will reach its end-of-life criterion in about 13,000 hours; to reach 25,000 hours, either its decay rate must be halved or its initial design must start with a larger margin. The budget can then be allocated to causes. Suppose, for illustration, that a long-haul truck over its life experiences 15,000 start-stop events, 500,000 significant load cycles, and 25,000 hours at a variety of steady operating points. If each start-stop, with good purging strategies, costs on average one microvolt, start-stops alone consume 15 millivolts, nearly a quarter of the budget. If load cycling consumes another 20 millivolts through platinum loss, the remaining 30 millivolts must cover all steady-state degradation, membrane thinning, contamination and the slow processes that continue whenever the cell is at temperature, an allowance of about 1.2 microvolts per hour. These numbers are illustrative rather than measured, but the method is how real programmes work. It shows designers which mechanisms are worth attacking and where controls can buy the most life. In this example, halving the cost of each start-stop or reducing the number of full shutdowns would free more budget than any plausible improvement in membrane chemistry. The budget also shows why designers add initial margin. A stack that begins life at 0.68 volts at rated current, instead of 0.65, because it has more active area or more platinum than strictly necessary, carries a reserve of 30 extra millivolts. Equivalently, a system can be specified so that the truck's rated power is available even after the stack has lost its 10 percent, by sizing the fuel cell 10 percent larger at the start. Both approaches convert capital cost into durability, and the right balance depends on the price of platinum, the cost of active area and the value of the truck's uptime. There is one further subtlety. The end of a fuel cell's life is not usually sudden. Unlike a diesel engine that seizes, a fuel cell stack loses performance gradually, and the vehicle keeps running with somewhat reduced peak power and somewhat higher fuel consumption. The economic end of life arrives when the extra hydrogen consumed and the lost performance cost more than replacing or refurbishing the stack. Refurbishment, replacing the membrane electrode assemblies while reusing plates and hardware, is likely to become an important service business once fleets mature, as it has for bus operators. Measuring durability honestly Durability claims in this field must be read carefully. Laboratory accelerated stress tests use standard protocols, such as those published by the U.S. Department of Energy, that cycle voltage rapidly to provoke catalyst loss or hold open circuit to provoke membrane attack. They are valuable for comparing materials, but they do not translate simply into vehicle hours. Bus fleet records are real but reflect a gentler duty. Truck data are only beginning to accumulate at scale. The most informative evidence so far comes from fleets that publish cumulative distance. Hyundai's XCIENT fleet in Switzerland, 48 trucks operating since 2020, had covered ten million kilometres by mid-2024. Daimler's five GenH2 trial trucks covered 225,000 kilometres in their first customer phase with consumption between 5.6 and 8.0 kilograms of hydrogen per 100 kilometres. These are encouraging figures, but they are fleet totals, not individual stack lives approaching 25,000 hours. Toyota describes its third-generation fuel cell system, announced in 2025 and aimed primarily at commercial vehicles, as offering twice the durability of the previous generation and a life comparable to diesel engines; independent long-term confirmation in trucks will take several more years to arrive. What can be said with confidence is that the remaining durability gap is now a matter of engineering refinement and validation, not of unknown physics. The mechanisms are understood, the countermeasures are known, and the trade-offs between platinum, cell area, cost and life can be quantified. That is a different situation from a decade ago. It also shifts attention to the part of the vehicle that has proved much harder to solve: the fuel tank, and the cost of filling it. Chapter 4: Carrying the Fuel Hydrogen has a split personality as a fuel. By mass it is superb: a kilogram holds about 33.3 kilowatt-hours of chemical energy, nearly three times as much as a kilogram of diesel. By volume it is dismal. At ordinary temperature and pressure a kilogram of hydrogen occupies about eleven cubic metres, the volume of a small room. Every hydrogen vehicle has to find a way to pack the gas densely enough to carry a useful amount, and every way of doing so costs energy, money, weight or space. For a long-haul truck that may need 70 to 100 kilograms of hydrogen to match the daily range of a diesel, and for a ship that may need tonnes, storage is not an accessory to the powertrain. It frequently decides whether the vehicle is feasible. There are two mainstream approaches, compressing the gas to very high pressure or cooling it into a liquid, and a hybrid of the two. Chemical carriers such as ammonia and methanol, which matter greatly at sea, are taken up in Chapter 7. The comparison in Table 2 sets out the basic physical facts that drive every design choice in this chapter. Table 2. Storage density of hydrogen options compared with diesel (approximate physical property values, NIST data; energy on lower heating value basis). Storage option Condition Density of fuel (kg/m³) Energy per litre of fuel (kWh) Key penalty Compressed hydrogen 350 bar, 15 °C about 24 about 0.8 Large tank volume Compressed hydrogen 700 bar, 15 °C about 40 about 1.3 Compression energy, tank cost Liquid hydrogen 1 bar, −253 °C about 71 about 2.4 Liquefaction energy, boil-off Diesel Ambient about 835 about 10 Carbon emissions The values in Table 2 refer to the fuel itself. Tanks add volume and mass, and for hydrogen they add a great deal. Even liquid hydrogen, the densest practical form of the element, stores only about a quarter of the energy of diesel in the same volume before the tank is counted. Compressed gas: the established path Almost all hydrogen vehicles on the road today store compressed gas. Passenger cars use 700 bar, roughly 700 times atmospheric pressure. Buses and many trucks, including Hyundai's XCIENT, use 350 bar, which is cheaper and simpler and acceptable where space for tanks is available on the roof or behind the cab. The tanks are marvels of composite engineering. The dominant design, known as Type IV, has a polymer liner, usually high-density polyethylene or polyamide, that acts as a gas barrier but carries almost no load. Around the liner is wound a thick shell of carbon fibre in epoxy resin, laid down by filament winding machines in helical and hoop patterns calculated to carry the pressure stresses. A metal boss at one or both ends accepts the valve. Regulations derived from the United Nations Global Technical Regulation on hydrogen vehicles require that the tank survive at least 2.25 times its nominal working pressure before bursting; a 700-bar tank must therefore hold beyond 1,575 bar. It must also survive thousands of pressure cycles, drop tests, bonfire tests in which a thermally activated relief device must vent the contents before the tank fails, and gunfire penetration tests. Carbon fibre is the expensive part. A 700-bar tank system for a passenger car carries five to six kilograms of hydrogen and weighs on the order of 80 to 100 kilograms; the carbon fibre accounts for most of the tank cost. Gravimetric capacity of current 700-bar systems is around 5 to 6 percent hydrogen by mass. For a heavy truck the numbers scale uncomfortably. Consider a truck that needs 80 kilograms of usable hydrogen, the capacity of Daimler's liquid-hydrogen GenH2. At 700 bar and about 40 kilograms per cubic metre, that hydrogen alone occupies 2,000 litres of internal tank volume. With tank walls and mountings the installed volume is substantially larger, and at roughly 5.5 percent gravimetric capacity the tank system weighs about 1,450 kilograms. The same energy in diesel, about 270 litres, would fit in a single side-mounted tank weighing perhaps 300 kilograms full. At 350 bar the hydrogen would need over 3,300 litres of internal volume, which is why 350-bar trucks generally accept shorter range: the XCIENT in Switzerland carries roughly 32 kilograms, sufficient for about 400 kilometres. Compression also costs energy. Raising hydrogen from a few bar to the 900 bar or so needed in station storage to fill a 700-bar vehicle tank takes typically 2 to 4 kilowatt-hours per kilogram in practice, depending on the inlet pressure and the compressor, or roughly 6 to 12 percent of the hydrogen's energy content. A further complication is heating during refuelling. When gas flows rapidly into a tank, the gas already inside is compressed and heats up; hydrogen, unusually, also warms slightly when it expands through a valve at room temperature. Composite tanks are rated to a maximum material temperature of 85 degrees Celsius. To fill quickly without exceeding that limit, stations precool the hydrogen to as low as minus 40 degrees Celsius before dispensing, following fuelling protocols such as SAE J2601 for light vehicles and newer high-flow protocols for heavy vehicles. Precooling adds equipment and energy at every station. The faster the fill, the more cooling is required, and heavy vehicles need flows of several kilograms per minute to refuel in a time comparable to diesel. Liquid hydrogen: density at a price Cooling hydrogen to minus 253 degrees Celsius, about 20 degrees above absolute zero, turns it into a clear liquid with a density of about 71 kilograms per cubic metre, nearly double the density of 700-bar gas, at essentially atmospheric pressure. The tank no longer needs thick carbon fibre walls. Instead it is a double-walled vessel with a vacuum space between inner and outer shells, filled with dozens of layers of reflective foil and spacer material, a design known as multilayer insulation. Such tanks are lighter per kilogram of hydrogen than high-pressure tanks and can be shaped to fit the space available on a truck chassis. The space industry has handled liquid hydrogen since the 1960s, and industrial gas companies have trucked it to customers for decades. Two physical facts make it difficult as a vehicle fuel. The first is the energy cost of liquefaction. The theoretical minimum work to liquefy hydrogen from room temperature is about 3.9 kilowatt-hours per kilogram. Real plants consume considerably more, typically in the range of 10 to 13 kilowatt-hours per kilogram for existing large liquefiers, roughly 30 to 40 percent of the hydrogen's own energy content. Part of the difficulty is peculiar to hydrogen. Hydrogen molecules exist in two forms, ortho and para, distinguished by the relative spin of their nuclei. At room temperature three-quarters of the molecules are ortho; at liquid temperatures nearly all should be para. The spontaneous conversion from ortho to para releases about 0.15 kilowatt-hours per kilogram of heat, more than the latent heat of vaporisation, which is about 0.12 kilowatt-hours per kilogram. If the liquid were stored without first converting it, the conversion heat would boil most of it away over days. Liquefiers therefore include catalysts that force the conversion during cooling, and they must remove that heat at cryogenic temperatures, where every unit of heat removed costs many units of work. New liquefier designs promise figures nearer 6 to 8 kilowatt-hours per kilogram at large scale, but these have yet to be demonstrated commercially. The second fact is boil-off. No insulation is perfect, and heat leaking into the tank evaporates liquid. In a sealed tank the pressure rises; when it reaches the relief setting, gas must be vented or consumed. The tiny latent heat of hydrogen makes this sensitive. A worked example shows why. Suppose a 40-kilogram truck tank has an effective heat leak of five watts, a reasonable figure for a well-insulated vessel of that size. Over a day that heat amounts to about 430 kilojoules. Divided by the latent heat of about 446 kilojoules per kilogram, it evaporates nearly one kilogram of hydrogen per day, or more than 2 percent of the tank. A truck in daily use consumes that vapour long before the pressure limit is reached, so the loss does not matter. A truck parked for a long weekend, or in a workshop for a week, gradually builds pressure and eventually vents. Large tanks fare far better, because the volume grows faster than the surface area: storage spheres at launch sites and on ships lose a fraction of a percent per day. Conventional liquid hydrogen refuelling also generated waste. Filling a warm tank flashes some liquid to gas, and the displaced vapour has to be returned to the station or vented. For trucks, which might refuel daily, this return gas complicated station design and reduced efficiency. Subcooled liquid and cryo-compressed storage Daimler Truck and Linde addressed these problems with a process they call subcooled liquid hydrogen, or sLH2, which they presented in early 2024 and opened to standardisation through ISO. The idea is to pump the liquid to a pressure somewhat above the vapour pressure corresponding to its temperature. Liquid in that state is subcooled: it can absorb some heat before it begins to boil. Delivered into the truck's tank in this condition, it fills without flashing and without the need to return vapour to the station, through a single nozzle. The pilot station at Wörth am Rhein, opened in 2024, dispenses more than 400 kilograms per hour, so the GenH2's two 40-kilogram tanks fill in roughly 10 to 15 minutes. Daimler's prototype completed a record run of 1,047 kilometres on a single fill in 2023, and its trial trucks have since worked in customer service. The process does not remove the energy cost of liquefaction, which is paid upstream, but it makes liquid storage practical at the vehicle and station. A third approach, cryo-compressed hydrogen, combines low temperature with high pressure. Hydrogen stored at around 30 to 40 kelvin and pressures of a few hundred bar reaches densities comparable to or slightly above liquid hydrogen, while the high-pressure tank can tolerate long periods of heat leak before venting, greatly improving dormancy. BMW pursued cryo-compressed storage for cars in the 2000s and 2010s, and developers including the German start-up Cryomotive have proposed it for heavy trucks. Its disadvantage is that it combines the costs of both worlds: a cryogenic vacuum-insulated vessel that is also a pressure vessel, and a supply chain built on liquid hydrogen. Refuelling a fleet: the station's side of the problem The storage decision on the truck fixes the design of the station that feeds it, and the station is where most of the infrastructure cost lies. Consider a depot serving 100 long-haul trucks, each refuelling with 60 kilograms of hydrogen per day. The station must dispense 6 tonnes per day, much of it concentrated in morning and evening peaks when the trucks return or set out. If the trucks use 700-bar compressed gas, the station receives hydrogen by pipeline, tube trailer or on-site electrolysis, compresses it into high-pressure buffer storage, cools it to minus 40 degrees Celsius and dispenses it. At around 3 kilowatt-hours per kilogram for compression and a further allowance for precooling, the station consumes on the order of 20 megawatt-hours of electricity a day, about the daily consumption of 2,000 average European homes. Its compressors must be sized for peak hours, not daily averages, and are among the least reliable components in the hydrogen chain: early stations for buses and cars suffered frequent compressor outages, and redundancy adds capital cost. Supplying 6 tonnes a day by compressed tube trailer would require many deliveries, since typical trailers carry a few hundred kilograms to around a tonne, so a station of this size would realistically need on-site production or a pipeline. If the trucks use liquid hydrogen, the station receives deliveries from a tanker carrying around four tonnes, so it needs about one and a half deliveries a day, stored in a vacuum-insulated tank of a few tens of cubic metres. The station's own energy use is small, because a cryogenic pump requires far less power than a high-pressure gas compressor. But upstream, at the liquefier, about 11 kilowatt-hours per kilogram has been spent, roughly 66 megawatt-hours for the day's 6 tonnes, more than three times the energy used by the compressed gas station. The liquid supply chain thus shifts energy use from many distributed stations to a few central plants. That may be acceptable, and even desirable, if liquefaction is done where electricity is cheap and low in carbon, and the logistics advantage is large, but it does not make the energy disappear. A further consideration is the market for liquid hydrogen. Liquefaction capacity in Europe is small, measured in tens of tonnes per day across a handful of plants, with Linde building a new liquefier at Leuna in Germany. North America has more capacity, built largely for the space and electronics industries. A truck fleet on any significant scale would require new liquefiers, each costing hundreds of millions of dollars and needing steady demand to pay off. The fleet's choice of storage technology is therefore tied to investments far beyond the depot fence. Choosing a storage path The choice between these options depends on the mission. Compressed gas at 350 bar suits regional distribution, buses and trucks that return to base each night, where a range of 300 to 500 kilometres is sufficient and tanks can be mounted behind the cab. Compressed gas at 700 bar extends range to perhaps 600 to 800 kilometres for long-haul tractors, at significant tank cost and with demanding station precooling; the Nikola Tre fuel cell truck, for example, carried about 70 kilograms at 700 bar. Liquid hydrogen offers the longest range on a conventional tractor package, beyond 1,000 kilometres, and the simplest vehicle tanks, at the price of liquefaction energy, boil-off management and a supply chain that currently relies on a handful of liquefiers worldwide. There is an important asymmetry in where the costs fall. With compressed gas, the energy-intensive step, compression, happens at the station and is moderate. With liquid hydrogen, the energy-intensive step, liquefaction, happens at a central plant and is large, but delivery by tanker is cheaper because a liquid tanker carries several times more hydrogen than a gas trailer, roughly four tonnes against one tonne or less for typical compressed tube trailers. For trucks served from a central production site hundreds of kilometres away, liquid can be the cheaper route despite the liquefaction penalty. For stations supplied by an on-site electrolyser, compressed gas usually wins. The storage choice also shapes the truck itself, its weight, its packaging and the volume left for cargo. That is where storage meets the rest of the powertrain, the subject of the next chapter. Chapter 5: Designing the Fuel Cell Truck Powertrain A fuel cell truck is a battery-electric truck with a power plant on board. The wheels are driven by electric motors, usually integrated into an electric axle; a high-voltage battery supplies and absorbs power peaks; and a fuel cell system, fed from hydrogen tanks, generates electricity to keep the battery charged and supply the steady load. Seen this way, the design questions are concrete. How big should the fuel cell be relative to the battery? How much hydrogen should the truck carry? Where does the heat go? What do the auxiliary machines consume? And how does the whole package fit within the length and weight limits that govern road freight? This chapter works through those questions for a European 40-tonne long-haul tractor, the hardest and most valuable case, using real vehicles to anchor the reasoning. The anatomy of the system Starting from the hydrogen tank, the fuel passes through a pressure regulator or, in a liquid system, through a heat exchanger that vaporises and warms it using waste heat from the coolant. It enters the anode of the stack at a controlled pressure, typically one to two and a half bar above atmospheric. Unused hydrogen leaving the anode is recirculated by a blower or an ejector back to the inlet, because only a fraction of the hydrogen reacts on a single pass; recirculation keeps the anode flow high, carries water and maintains uniform conditions. A purge valve periodically releases a small amount of anode gas to remove nitrogen that diffuses across from the air side. On the cathode side, an electrically driven air compressor draws air through a filter, compresses it to two or three bar, and sends it through an intercooler and often a humidifier into the stack. Higher pressure raises oxygen concentration and improves performance, especially at high power, but the compressor is the largest parasitic load in the system. Many heavy-duty systems use high-speed turbo-compressors on air bearings with an expander turbine on the exhaust side that recovers part of the compression work. The stack produces direct current at a voltage that varies with load, typically between 250 and 450 volts per module depending on cell count. A boost converter raises this to the vehicle's high-voltage bus, which in modern trucks is commonly around 600 to 800 volts. The battery connects to the same bus, and the inverters for the traction motors draw from it. The cooling circuit removes heat from the stack through deionised coolant, sends it to a large radiator at the front of the truck, and diverts some of it to warm the cab, the battery and, in a liquid hydrogen truck, the hydrogen vaporiser. A separate lower-temperature circuit cools the power electronics and motors. The system controller coordinates all of this. It decides, many times a second, how much power the stack should produce, how much the battery should provide or absorb, what air flow and pressure to command, and how to keep cell voltages, temperatures and humidity within their preferred windows. It is in the controller, as Chapter 3 showed, that much of a fuel cell's durability is won or lost. Three real trucks and a benchmark It helps to compare actual designs. Table 3 sets out the principal energy specifications of three zero-emission heavy trucks alongside the diesel benchmark they must displace. Table 3. Energy systems of representative heavy trucks (manufacturer data). Truck Energy stored on board Converter Traction battery Stated range Mercedes-Benz GenH2 (trial) 80 kg liquid hydrogen 2 × 150 kW fuel cell 70 kWh over 1,000 km Hyundai XCIENT Fuel Cell (Swiss) about 32 kg at 350 bar 2 × 90 kW fuel cell small buffer pack about 400 km Mercedes-Benz eActros 600 621 kWh battery none 621 kWh about 500 km Typical diesel long-haul tractor 600–1,000 litres diesel combustion engine none 2,000 km or more The three zero-emission designs embody different philosophies. The XCIENT, designed when the first deployments in Switzerland began in 2020, uses modest fuel cell power and 350-bar storage, accepting a short range suited to regional distribution within a small country with short distances. Its fuel cell rating of 180 kilowatts is enough for the steady load on Swiss roads but relies on the battery for peaks. Hyundai's later versions for North America raised hydrogen capacity to reach over 700 kilometres. The GenH2 is a long-haul tractor. Its 300 kilowatts of fuel cell power, together with the battery, give it performance comparable to a diesel of similar rating, and its liquid hydrogen tanks give it diesel-like daily range. Its battery is deliberately small because the fuel cell carries the sustained load; the battery handles peaks and regenerative braking. The eActros 600 shows how far battery trucks have come. With 621 kilowatt-hours of lithium iron phosphate cells, it covers about 500 kilometres on a charge and can be recharged during the 45-minute break that European drivers must take after four and a half hours of driving, using megawatt-class chargers. Its very existence has shifted the debate about hydrogen trucks, and it is the comparison that Chapter 6 takes up in depth. Sizing the fuel cell and battery The core sizing problem is to divide power between the fuel cell and the battery. At one extreme is the fuel-cell-dominant design, where the stack is sized for the truck's continuous power demand, including long climbs, and the battery is a small buffer. At the other is the range-extender design, where a large battery does most of the work and a small fuel cell trickles energy into it, extending range. Between lie many compromises. A worked example illustrates the trade-offs. Take a 40-tonne tractor-trailer cruising on a level motorway at 80 kilometres per hour. Rolling resistance with good tyres absorbs about 2,300 newtons; aerodynamic drag on a modern European tractor-trailer adds roughly 1,600 newtons. The total of about 3,900 newtons at 22.2 metres per second requires about 87 kilowatts at the wheels. Allowing for losses in the motors, inverters and gearing, and for auxiliary loads such as air compressors for the brakes, power steering and cab heating or cooling, the electrical demand on the bus is around 100 to 110 kilowatts. A net fuel cell output of 110 kilowatts at a system efficiency near 50 percent consumes about 6.6 kilograms of hydrogen per hour, or roughly 8 kilograms per 100 kilometres at 80 kilometres per hour, consistent with the upper end of the 5.6 to 8.0 kilograms per 100 kilometres Daimler reported from its GenH2 trials. Lighter loads, lower speeds and flatter routes account for the lower figures. Now consider a long climb, say 20 kilometres at a 5 percent gradient, gaining 1,000 metres. At 60 kilometres per hour, lifting 40 tonnes up that slope requires about 330 kilowatts at the wheels for gravity alone, plus around 55 kilowatts for rolling and air resistance, so roughly 385 kilowatts at the wheels and well over 400 kilowatts from the electrical bus. A fuel-cell-dominant truck with 300 kilowatts of gross stack power will deliver perhaps 260 to 270 kilowatts net after parasitic loads. The battery must supply the remaining 150 kilowatts or so for the 20 minutes of the climb, about 50 kilowatt-hours. A 70-kilowatt-hour battery can do that, but only if it starts the climb well charged, which means the controller must anticipate the climb using navigation data and charge the battery beforehand. Without predictive energy management, the truck would slow down on long grades, just as an underpowered diesel does. On the descent, the truck has 1,000 metres of potential energy to dispose of, about 109 kilowatt-hours. Some is consumed by rolling and air resistance, but most must be absorbed by braking. A 70-kilowatt-hour battery that has just discharged 50 kilowatt-hours on the climb can recover a good share. Once it is full, the remaining energy goes through electric brake resistors into the cooling system, adding load to a radiator already sized at the limit of the truck's frontal area. The example shows why stack and battery sizing is a negotiation between route, payload and cost. A larger battery handles longer climbs and recovers more braking energy, but adds weight and cost. A larger stack reduces dependence on the battery and runs at more efficient points, but costs more and produces more heat at peak. For routes through mountainous regions, designers lean towards more battery; for flat continental corridors, the fuel cell can be smaller relative to the truck's peak demand. Parasitic loads and system efficiency The difference between stack efficiency and system efficiency is a recurring source of confusion. The stack in the Chapter 1 example ran at 52 percent at rated power. The system around it consumes power to keep it running. The air compressor is the largest load, typically on the order of 10 percent of gross stack power at high load, falling at lighter load as the required air flow and pressure fall. The hydrogen recirculation blower, the coolant pump and the radiator fans add several percent more, and on a hot day the fans can draw tens of kilowatts. The boost converter loses one or two percent. As a result, the net system efficiency of a heavy-duty fuel cell system typically peaks somewhere in the mid-50s percent at moderate load and falls to the low or mid-40s at full rated power. That shape matters. A truck whose average net demand is around 100 kilowatts, served by a system rated at 270 kilowatts net, spends most of its time near the system's peak efficiency. This is the strongest argument for generous fuel cell sizing in long-haul trucks: the extra capital cost buys fuel savings on every kilometre. The Department of Energy's heavy-duty targets reflect this, calling for peak system efficiency of 68 percent by 2030 and 72 percent ultimately, values which would require better catalysts and lower parasitic losses than today's systems achieve. Packaging, weight and heat A European tractor is constrained by a maximum length for the whole articulated combination of 16.5 metres, and by axle load limits. Every centimetre given to the powertrain is a centimetre less for the trailer. Fuel cell trucks must fit stacks, compressors, radiators, battery packs and hydrogen tanks into a chassis designed around a diesel engine, gearbox and fuel tanks. Liquid hydrogen tanks have a packaging advantage here: two 40-kilogram tanks can sit on either side of the chassis rails where diesel tanks would go. Compressed tanks for comparable hydrogen mass require racks behind the cab, lengthening the tractor or shortening the trailer, which is acceptable in some markets but not where length rules bite. Weight is a cost. A fuel cell tractor is heavier than a diesel equivalent, with the hydrogen tanks, fuel cell system and battery together outweighing a diesel engine, transmission and fuel. In Europe, regulations allow zero-emission trucks additional gross weight, up to two tonnes, to compensate for heavier powertrains, preserving most of the payload. Still, for weight-limited loads such as bulk liquids, aggregates or timber, every kilogram of powertrain is lost revenue, and that is where liquid hydrogen's lighter tanks and the fuel cell's ability to carry energy more lightly than batteries make the clearest case. Heat rejection, discussed in Chapter 2, is the constraint that most often defines the upper limit of continuous power. The radiator area available at the front of a cab-over European truck is limited, and North American conventional-cab trucks with long noses have more room. The design response includes higher stack operating temperatures, larger and more efficient radiators, electrically driven fans that can be controlled precisely, and use of waste heat for cab heating and hydrogen vaporisation. On hot days, systems are allowed to derate gracefully rather than overheat, with the battery covering short peaks. Winter, refrigeration and the auxiliary loads The worked examples above concerned traction. A truck has other energy demands, and some of them favour the fuel cell in ways that are easy to overlook. The first is heating. A diesel truck heats its cab in winter with waste heat from the engine, at essentially no cost. A battery-electric truck must heat its cab, and often its battery, with electrical energy drawn from the battery, usually through a heat pump. On a cold night, with the driver sleeping in the cab, the energy used for heating is a significant fraction of the battery's capacity and must be replaced before the next day's driving. A fuel cell truck, like a diesel, has abundant waste heat whenever the stack is running, typically tens of kilowatts at the coolant temperature of 70 to 90 degrees Celsius, more than enough for the cab. Even while parked, a fuel cell system can be run at low power to provide both electricity and heat for an overnight rest, though this costs hydrogen and must be managed to avoid the high-voltage idling that Chapter 3 identified as damaging. The second is refrigeration. A refrigerated trailer, or reefer, typically carries its own diesel-powered refrigeration unit, consuming a few litres of diesel per hour. Zero-emission alternatives include battery-powered units charged from the tractor or from shore power, and units powered from the tractor's fuel cell. For long cold-chain journeys, the fuel cell's ability to provide continuous electrical power for both traction and refrigeration is attractive, adding perhaps 5 to 10 kilowatts of continuous load that the system can carry without the range penalty a battery would suffer. The third is cold-climate operation more generally. Lithium-ion batteries lose capacity and charging speed in severe cold unless warmed. A fuel cell system, once started, warms itself and its battery. That is one reason fuel cell trucks have been tested in Nordic countries and in Canada, and why cold-climate long-haul is often cited as a favourable niche. The countervailing issue is the freeze protection discussed in Chapter 2: a fuel cell truck parked at minus 30 degrees Celsius without power must have purged its stack thoroughly at shutdown and must manage its cold start carefully. The fourth is the truck's own pneumatic and hydraulic systems: compressed air for brakes and suspension, hydraulic power for tippers and cranes, and power take-off for special vehicles. In a fuel cell truck all of these are driven electrically, adding to the base load but without affecting the logic of the design. Taken together, the auxiliary loads add 5 to 15 percent to a long-haul truck's energy demand, more in winter or in refrigerated service. For a battery truck they come directly out of range. For a fuel cell truck, part is met by waste heat that would otherwise be thrown away, a modest but real advantage that becomes significant in cold regions and refrigerated logistics. Integration and control as competitive advantage The truck makers that have gone furthest share an important characteristic: they treat the fuel cell not as a bought-in engine substitute but as one element in an integrated energy system. Predictive control that uses map data to prepare for climbs, thermal management that routes waste heat to where it is useful, energy management that protects the stack from high voltages and shields the battery from deep cycles, and diagnostics that detect and recover reversible degradation are as decisive as stack chemistry. This software layer is where experience accumulates, which is why the trial fleets of the mid-2020s, small as they are, matter more than their numbers suggest. Yet all of this engineering serves a vehicle whose operating cost is dominated not by any of its components but by the price of the fuel it burns. A truck that drives 120,000 kilometres a year at 7 kilograms per 100 kilometres consumes 8,400 kilograms of hydrogen annually. At ten dollars per kilogram that is 84,000 dollars a year in fuel, more than the purchase price of many diesel tractors. The engineering of the vehicle is now good enough that its economics are decided almost entirely elsewhere, in the cost of making, moving and dispensing hydrogen. Chapter 6: The Economics of the Molecule For most of the 2010s, the case for hydrogen trucks rested on a comparison with diesel. Diesel trucks were cheap and refuelled quickly; battery trucks were assumed to be hopelessly heavy and slow to charge; hydrogen offered zero emissions with diesel-like operations. That framing no longer holds. Battery-electric trucks now operate long-haul routes in Europe, China and North America, megawatt charging is being standardised and installed, and fleet managers compare hydrogen against batteries first and diesel second. This chapter sets out the economics behind that shift, explains why the cost of delivered hydrogen dominates everything else, and identifies the conditions under which fuel cell trucks remain the better choice. The efficiency chain Begin with the physics, because it sets the floor under every cost calculation. A zero-emission truck ultimately runs on low-carbon electricity, whether that electricity is stored in a battery or first converted into hydrogen. The question is how much of the original electricity reaches the wheels. Table 4 traces the chain for a battery truck and for fuel cell trucks using compressed and liquid hydrogen, using typical values discussed in this book. Table 4. Electricity-to-wheel efficiency chain for zero-emission trucks (typical values; author's calculation from figures cited in this book). Step Battery truck Fuel cell, compressed hydrogen Fuel cell, liquid hydrogen Electricity to stored energy Charging and transmission about 92% Electrolysis about 64% Electrolysis about 64% Conditioning and delivery Included above Compression and dispensing about 90% Liquefaction and delivery about 72% Stored energy to bus Battery discharge about 97% Fuel cell system about 50% Fuel cell system about 50% Bus to wheels Motors and inverters about 90% Motors and inverters about 90% Motors and inverters about 90% Overall about 80% about 26% about 21% The individual figures can be argued at the margin. Electrolysers may reach 70 percent on a lower heating value basis; fuel cell systems may average 55 percent on a favourable route; liquefaction may improve. Yet the conclusion is robust to any reasonable adjustment. A fuel cell truck needs roughly three to four times as much primary electricity per kilometre as a battery truck. Where renewable electricity is scarce or expensive, that multiplier is a heavy burden. Where electricity is abundant and cheap at the point of hydrogen production but cannot be delivered to the vehicle as electricity, the comparison changes, and that is precisely the situation the remainder of this chapter explores. What hydrogen at the pump really costs The price a truck operator pays for hydrogen is the sum of three broad components: production, delivery and dispensing. Production by electrolysis is dominated by electricity. A modern electrolyser system consumes around 50 to 55 kilowatt-hours of electricity per kilogram of hydrogen. At an electricity price of 50 dollars per megawatt-hour, the electricity alone costs about 2.6 dollars per kilogram; at 100 dollars per megawatt-hour, about 5.2 dollars. To this must be added the capital cost of the electrolyser, its balance of plant and its maintenance, spread over its output. An electrolyser run only when solar power is cheap might operate 20 to 25 percent of the hours in a year, which spreads its capital over few kilograms; one run continuously on grid power has lower capital cost per kilogram but higher electricity cost and, depending on the grid, higher emissions. Most realistic assessments of renewable hydrogen production in Europe and North America in the mid-2020s place costs between about 5 and 8 dollars per kilogram at the plant gate, with lower figures in exceptional locations. The U.S. Department of Energy's Hydrogen Shot, announced in 2021, aims at one dollar per kilogram for clean hydrogen production within a decade, an ambition that frames the gap rather than describing the present. For context, the International Energy Agency's Global Hydrogen Review reports world hydrogen demand of around 100 million tonnes a year, almost all of it made from natural gas and coal for refineries and fertiliser, with low-emission production amounting to less than one percent of supply. Hydrogen made by steam reforming of natural gas without carbon capture costs much less, typically one to two dollars per kilogram at the plant where gas is cheap, but carries emissions of roughly 9 to 12 kilograms of carbon dioxide per kilogram of hydrogen. A truck running on such hydrogen emits, on a well-to-wheel basis, about as much carbon dioxide as a diesel truck. It is cleaner locally but not a climate solution. Delivery and dispensing frequently cost more than production. Compressing, storing, precooling and dispensing hydrogen at a heavy-duty station requires compressors, high-pressure storage banks, chillers and dispensers, all built to demanding safety standards. A large truck station may cost several million dollars, and its cost per kilogram depends overwhelmingly on how much hydrogen it sells. A station designed for several tonnes a day but selling a few hundred kilograms recovers its capital over very few kilograms and must charge accordingly, or lose money. The California experience shows what happens when that arithmetic goes wrong. California built a network of about fifty retail stations for fuel cell cars. With few cars and unreliable supply, prices rose steadily. By October 2024 the average retail price was about 34.55 dollars per kilogram according to S&P Global Commodity Insights, and some stations charged 36 dollars. Shell closed its California light-duty hydrogen stations in early 2024. Fuel cell car sales fell sharply as owners struggled to find working stations. None of this was caused by a failure of fuel cell technology. It was caused by an infrastructure that never reached the throughput needed to be economic. The diesel parity calculation A useful way to understand the economics is to calculate the hydrogen price at which a fuel cell truck's fuel cost per kilometre matches a diesel truck's. Take a European long-haul tractor that uses about 30 litres of diesel per 100 kilometres. At a pump price of about 1.6 euros per litre including fuel taxes, fuel costs 48 euros per 100 kilometres. A fuel cell truck using 7 kilograms of hydrogen per 100 kilometres reaches parity at about 6.9 euros per kilogram. If the truck uses 6 kilograms, parity rises to 8 euros. In the United States, with lower diesel taxes, a truck using about 35 litres per 100 kilometres at roughly one dollar per litre spends about 35 dollars per 100 kilometres; parity at 7 kilograms per 100 kilometres is about 5 dollars per kilogram. These are rough figures, and fuel prices fluctuate, but they show the target. Delivered clean hydrogen needs to cost something like 5 to 8 units of currency per kilogram at the pump to compete on fuel cost alone, before the higher purchase price of the fuel cell truck is considered. With plant-gate renewable hydrogen at 5 to 8 dollars and delivery and dispensing adding several more, the gap in the mid-2020s is typically a factor of 1.5 to 2 unless policy closes it. The same calculation for a battery truck is more favourable. At about 1.2 kilowatt-hours per kilometre from the battery, a battery truck charged at a depot for 0.20 euros per kilowatt-hour spends 24 euros per 100 kilometres, half the diesel figure. Charged at a public megawatt station for perhaps 0.45 euros per kilowatt-hour, it spends about 54 euros, close to diesel. The cost depends heavily on how much charging happens at the depot. That flexibility is one reason battery trucks have taken the lead: an operator can begin with depot charging on regional routes, where the economics are strong, and extend to long haul as public charging grows. Fuel dominates the total cost Fleet operators buy trucks on total cost of ownership: purchase price, financing, fuel or energy, maintenance, tolls, insurance and residual value over the period they own the vehicle. A simple worked comparison shows which of these matters most for long-haul hydrogen trucks. Take a tractor driven 120,000 kilometres a year for five years, a typical first-owner life in European long haul, and compare only the energy costs using the illustrative prices above. The diesel truck at 48 euros per 100 kilometres spends about 57,600 euros a year, or 288,000 euros over five years. A fuel cell truck at 7 kilograms per 100 kilometres and 10 euros per kilogram, a plausible delivered price for renewable hydrogen at a moderately used station today, spends 84,000 euros a year, or 420,000 euros over five years. A battery truck charging half its energy at a depot for 0.20 euros per kilowatt-hour and half at public stations for 0.45 euros, averaging 0.325 euros, at 1.2 kilowatt-hours per kilometre spends about 46,800 euros a year, or 234,000 euros over five years. The differences in energy cost over five years, around 130,000 euros between hydrogen and diesel and nearly 190,000 euros between hydrogen and battery, exceed any plausible difference in purchase price between the vehicles. They also show the leverage of hydrogen price. Every euro per kilogram of hydrogen price adds 8,400 euros a year to this truck's fuel bill. Bringing the price from 10 euros to 6 euros saves more than 33,000 euros a year and brings the fuel cell truck close to diesel fuel cost. No conceivable improvement in fuel cell efficiency could achieve as much: raising system efficiency by a tenth, from 50 to 55 percent, would save only about 7,600 euros a year at 10 euros per kilogram. Maintenance and uptime complete the picture. Electric drivetrains, whether battery or fuel cell, have fewer moving parts than diesels and need no oil changes, but fuel cell systems have components, such as air compressors, humidifiers, filters and hydrogen recirculation hardware, that require service, and stack replacement or refurbishment may be needed within the vehicle's life. The real maintenance costs of fuel cell trucks will become clear only when trial fleets have accumulated several years of operation. Uptime depends critically on station reliability; a truck that cannot refuel because the station's compressor has failed earns nothing that day, and operators of early hydrogen bus fleets have learned to treat fuel availability as the main operational risk. The conclusion is unambiguous. For heavy trucks, the price of delivered hydrogen is the variable that decides competitiveness. Engineering improvements to the vehicle are valuable, but they matter at the margin. The cost of electricity, the utilisation of electrolysers, the logistics of delivery and the throughput of stations decide the outcome. Policy, and its reversals Policy shapes these calculations heavily, and the policy landscape shifted repeatedly in 2024 and 2025. In Europe, the revised carbon dioxide standards for heavy-duty vehicles require manufacturers to reduce average fleet emissions of new trucks by 45 percent by 2030, 65 percent by 2035 and 90 percent by 2040, relative to 2019. Manufacturers cannot meet these targets without selling large numbers of zero-emission trucks, but the regulation does not choose between batteries and hydrogen. The Alternative Fuels Infrastructure Regulation requires hydrogen refuelling stations at intervals of no more than 200 kilometres along the core trans-European transport network and in every urban node by 2030, a mandate that at least guarantees a skeleton network. Germany exempted zero-emission trucks from its motorway toll until the end of 2025, and charges them a much reduced rate thereafter, a substantial subsidy for any truck with high mileage. Battery trucks have captured most of the benefit. In the United States, the Inflation Reduction Act of 2022 created a production tax credit for clean hydrogen of up to three dollars per kilogram for the lowest-emission production. Legislation passed in July 2025 shortened its availability to projects that begin construction before 2028, and ended the commercial clean vehicle credit for purchases after September 2025. Congress also overturned in 2025 the federal waivers that allowed California to enforce its zero-emission truck sales rules, an action California is contesting in court. The net effect has been to reduce both the demand pull and the supply push for hydrogen trucks in the American market. Lessons from the failures The business collapses of 2024 and 2025 are instructive. Nikola filed for Chapter 11 bankruptcy in February 2025 after years of losses, a costly recall of its battery trucks and the difficulty of selling fuel cell trucks without affordable hydrogen; it had also tried to build its own hydrogen supply business, adding capital burden. Hyzon, which converted trucks to fuel cell power, wound down. Several European start-ups that planned fuel cell trucks or conversions entered insolvency. Even established players adjusted. Daimler Truck deferred series production of its fuel cell truck to after 2030, while planning a limited series of 100 trucks for customers from the end of 2026. The fuel cell partners at cellcentric delayed their planned large factory and, in 2026, agreed to bring Toyota in as a third equal shareholder to share costs and risks; the cellcentric partners now emphasise buses, rail and stationary power alongside trucks. The common lesson is that a hydrogen truck is not a product that can succeed by itself. It is half of a system whose other half, affordable hydrogen at high-throughput stations, has to be built at the same time and at matching scale. Companies that sold trucks without securing fuel at a viable price found that their customers could not operate them economically. Companies that tried to build both the trucks and the fuel supply lacked the capital to do either well. Where fuel cell trucks still make sense None of this means hydrogen trucks have no future. It means their future lies where their particular strengths outweigh their efficiency penalty. Several conditions point that way. The first is very high daily energy demand combined with limited time to recharge. A truck run in two or three shifts a day by different drivers, covering 1,000 kilometres or more daily with heavy loads, cannot easily stop long enough to recharge a battery repeatedly, and carrying enough battery to avoid recharging would cost too much payload. Fuel cell trucks refuelled in fifteen minutes fit this pattern. The second is weak grid access. Megawatt charging requires grid connections of several megawatts per site, which can take years to build where grid capacity is limited. A hydrogen station supplied by liquid tanker from a distant production plant needs far less local grid capacity. In regions where grid reinforcement is slow, hydrogen can arrive before charging capacity does. The third is heavy payload in weight-limited applications. Bulk haulage, tankers and timber transport lose revenue with every kilogram of battery. Liquid hydrogen systems can carry long range more lightly. The fourth is industrial clusters with existing hydrogen supply. Ports, refineries, chemical plants and steelworks that produce or use hydrogen for other purposes can supply trucks at marginal cost, with high station utilisation from captive fleets. Swiss operations of the XCIENT fleet, built around hydrogen produced at a hydroelectric plant and a closed network of customers, follow this logic. The fifth is off-road heavy equipment, where energy demands dwarf those of road trucks. Anglo American demonstrated a hybrid fuel cell and battery mining haul truck at its Mogalakwena platinum mine in South Africa in 2022, with about two megawatts of power, as a step towards decarbonising mine fleets that consume enormous quantities of diesel. Mining vehicles return to the same fuelling point repeatedly, which solves the infrastructure problem by construction. These niches are not small, but they are not the whole of trucking. A realistic view in late 2026 is that battery-electric trucks will take most of the regional and much of the long-haul market where charging can be provided, while fuel cell trucks compete for the most demanding duty cycles and in places where electricity cannot reach the vehicle cheaply. That division mirrors the logic that applies even more strongly at sea, where the energy requirements become too large for batteries on most voyages. Chapter 7: Hydrogen at Sea Shipping is the backbone of world trade and one of its least visible polluters. Ships carry around four-fifths of global trade by volume, and the International Maritime Organization's fourth greenhouse gas study estimated that shipping emitted just over a billion tonnes of greenhouse gases in 2018, close to 3 percent of the world total. Almost all of it comes from burning heavy fuel oil and marine diesel in large, slow, remarkably efficient engines. The largest two-stroke marine diesels convert more than half of their fuel energy into shaft power, a figure few other heat engines match. Decarbonising this fleet differs from decarbonising trucks in three important ways. The energy quantities are vastly larger: a single large container ship can burn on the order of 100 tonnes of fuel a day on long voyages. The vessels last 25 years or more, so ships ordered today will still be sailing in 2050. And ships cannot recharge at sea, so energy for an entire voyage must be carried on board. Batteries, which are winning on the road, are limited at sea to short routes. For anything beyond a few hours of sailing, chemical fuels remain necessary, and the question becomes which ones. The energy density problem, magnified The volume problem identified for trucks in Chapter 4 becomes severe at sea, where fuel tanks displace cargo. Table 5 sets out the energy density of the main candidate marine fuels. Table 5. Candidate marine fuels compared (standard physical property values; energy on lower heating value basis). Fuel Storage condition Energy per kg (MJ) Energy per m³ of fuel (GJ) Main converters Fuel oil Ambient about 41 about 40 Diesel engines Liquefied natural gas −162 °C about 50 about 22 Dual-fuel engines Methanol Ambient about 20 about 16 Dual-fuel engines, fuel cells via reforming Ammonia −33 °C or about 10 bar about 18.6 about 12.7 Ammonia engines, solid oxide fuel cells Liquid hydrogen −253 °C about 120 about 8.5 PEM fuel cells, hydrogen engines The comparison shows why hydrogen faces obstacles at sea. Liquid hydrogen carries about a fifth of the energy of fuel oil per unit volume, before the bulky cryogenic tanks are counted. A worked example makes the consequence concrete. Suppose a large container ship burns 100 tonnes of fuel oil per day, about 4,100 gigajoules of fuel energy. If a hydrogen fuel cell system could match a marine diesel's efficiency, the ship would need about 34 tonnes of liquid hydrogen per day, occupying about 480 cubic metres. A 30-day voyage would need over 14,000 cubic metres of liquid hydrogen, against about 3,000 cubic metres of fuel oil, and the insulated hydrogen tanks would occupy considerably more space than that. Large ocean-going ships could carry this volume only by sacrificing substantial cargo capacity or by bunkering more often on shorter legs. Ammonia and methanol offer a middle path. Both are hydrogen carriers in chemical form: ammonia contains no carbon at all, and methanol, when made from captured carbon dioxide or biomass and green hydrogen, can be close to carbon neutral. Their energy per unit volume is better than liquid hydrogen, and they can be stored at modest conditions: methanol as an ordinary liquid, ammonia at minus 33 degrees Celsius or under moderate pressure, as it already is in a global trade of about 18 to 20 million tonnes a year. Both can be burned in modified marine engines, and both can feed fuel cells. Their cost is that converting hydrogen into them consumes energy, and they bring their own hazards: ammonia is acutely toxic, and its combustion can produce nitrous oxide, a potent greenhouse gas. This is why large shipping lines have moved first to methanol and are preparing for ammonia, rather than to liquid hydrogen. Maersk took delivery of the first methanol-fuelled container ship, the Laura Maersk, in 2023, and dozens of methanol dual-fuel ships have been ordered since. Engine makers including MAN Energy Solutions and WinGD have developed two-stroke engines for ammonia, with the first vessels entering service in the mid-2020s. Where hydrogen fits at sea Pure hydrogen and PEM fuel cells find their maritime place in the same kind of niche they occupy on land: fixed routes, frequent bunkering at a single home port, moderate power, and operations too long for batteries alone. Ferries and harbour craft fit this profile well. Norway has been the testing ground. Its ferry network, operated under public contracts with environmental requirements, has electrified rapidly; the battery ferry Ampere entered service in 2015, and the country now operates dozens of battery ferries on short crossings. For longer crossings and open-sea routes, batteries alone are insufficient, and the Norwegian Public Roads Administration has used its procurement to push hydrogen. The MF Hydra is the first result. Built by Westcon and operated by Norled on a short route in Rogaland between Hjelmeland, Skipavik and Nesvik, the 82-metre ferry carries up to 300 passengers and 80 cars. It entered service on liquid hydrogen in March 2023. Its fuel is stored in an 80-cubic-metre vacuum-insulated tank on deck, holding on the order of five tonnes of liquid hydrogen at minus 253 degrees Celsius, enough for up to about twelve days of service. Two 200-kilowatt PEM fuel cell modules from Ballard Power Systems supply electricity, with a battery of about 1.36 megawatt-hours from Corvus Energy buffering the load. Electric thrusters drive the vessel. The Hydra is technically a success and economically a demonstration. Its liquid hydrogen came from Linde's liquefier at Leuna in Germany, transported more than a thousand kilometres by road, and analyses of the project have pointed out that its operating costs are several times those of a diesel ferry and that the upstream emissions of liquefaction and long-distance trucking eroded the climate benefit. That outcome follows directly from the efficiency chain in Chapter 6: hydrogen made far away, liquefied with grid electricity and trucked over a long distance, carries heavy energy and emissions penalties. The ferry proved the vessel; it did not prove the supply chain. The larger test is on the route from Bodø to the Lofoten Islands, a crossing of about 100 kilometres of open Arctic sea that is Norway's longest domestic ferry connection. Torghatten Nord won a 15-year contract to operate it from 2025, with a requirement that at least 85 percent of the energy consumed by the two main vessels come from hydrogen. The two ferries, named Røst and Moskenes, are about 117 to 120 metres long, carry 599 passengers and 120 cars, and have been built with hulls from a Turkish yard and outfitting at Myklebust Verft in Norway. They use compressed hydrogen supplied by GreenH from a production plant near Bodø, with MAN Cryo responsible for the hydrogen fuel system. Each ferry is expected to consume five to six tonnes of hydrogen per day. In energy terms, five and a half tonnes of hydrogen per day is about 183 megawatt-hours, roughly the energy content of fifteen tonnes of diesel. The ferries are scheduled to enter hydrogen service in 2026, and their operation will provide the first large-scale evidence of hydrogen fuel cell ships running every day in hard conditions with local supply. Why one ferry matters more than a hundred trucks The Lofoten project illustrates a point about hydrogen infrastructure that is easy to miss. The most difficult problem for hydrogen trucks, as Chapter 6 showed, is station utilisation: a station built for tonnes per day that sells only a few hundred kilograms cannot price its hydrogen competitively. A ferry solves this problem by its sheer size and regularity. A single Lofoten ferry consuming about 5.5 tonnes of hydrogen per day uses roughly 2,000 tonnes a year. A long-haul fuel cell truck driving 120,000 kilometres a year at 7 kilograms per 100 kilometres uses about 8.4 tonnes. One ferry therefore represents the annual hydrogen demand of about 240 long-haul trucks, and two ferries nearly 500. That demand is concentrated at a single bunkering point, predictable to the hour, and contracted for fifteen years by a public authority. For a hydrogen producer, this is the ideal customer. The production plant can be sized for a known load, financed against a long contract and run at high utilisation, which drives down the cost per kilogram. The scale of the production plant follows from simple arithmetic. Producing 5.5 tonnes per day by electrolysis at about 52 kilowatt-hours per kilogram requires about 286 megawatt-hours of electricity per day, equivalent to a continuous electrolyser load of about 12 megawatts for one ferry, or 24 megawatts for two. Northern Norway has abundant hydroelectric power, and a plant of that size near Bodø can be connected to it. GreenH's planned capacity of six to ten tonnes per day would supply both ferries with some surplus, which could in turn serve trucks, buses or other vessels in the region at a marginal cost, since the plant's base load is already paid for. This is the anchor-customer model, and it reverses the usual chicken-and-egg problem. Instead of building stations and hoping vehicles will arrive, a large and certain consumer underwrites the production plant, and smaller consumers join later. Ports with ferry routes, harbour craft and port equipment, mines with heavy vehicle fleets, and industrial sites that already use hydrogen are natural anchors. Many of the hydrogen hubs planned in Europe and North America are built around this idea, and its success or failure on the Lofoten route will be watched closely. The model also clarifies why the MF Hydra's economics were so poor. Its hydrogen was a small, irregular purchase from a distant plant serving other customers, carried a long distance by road. The vessel was right; the supply arrangement was a demonstration. The Lofoten project, by building production near the point of use for a contracted load, tests the arrangement that would make hydrogen ferries economic. Fuel cells for ships The fuel cell technologies suited to ships differ from those for trucks because the operating conditions differ. Ships run for long periods at steady load, with slow changes, and have space for larger and heavier equipment than a truck chassis allows. They also need megawatts rather than hundreds of kilowatts. PEM fuel cells remain the natural choice for pure hydrogen, especially where load varies or quick start is needed. Marine PEM systems are built from modules of a few hundred kilowatts, combined to provide megawatts. Their advantages at sea are maturity and responsiveness. Their disadvantages are the same as on land: the need for very pure hydrogen and the rejection of large amounts of low-temperature heat. On a ship, heat rejection is easier because seawater provides an unlimited cold sink, a significant advantage over a truck with a limited radiator. Solid oxide fuel cells, running at 600 to 850 degrees Celsius, offer higher electrical efficiencies, often above 55 percent and potentially more than 60 percent, and can consume ammonia or methanol directly or after simple internal processing, because their high temperature cracks ammonia and reforms methanol inside or close to the cell. Their high-grade waste heat can be used for further power generation or for ship heating. Their drawbacks are slow start-up, sensitivity to thermal cycling and lower power density. For ships that run steadily for days, those drawbacks matter less. Projects such as the European ShipFC programme have aimed to install a two-megawatt ammonia-fuelled solid oxide fuel cell on the offshore supply vessel Viking Energy, testing whether the combination can work at sea. High-temperature PEM fuel cells, running at 160 to 180 degrees Celsius with phosphoric-acid-doped membranes, occupy a middle ground. They tolerate carbon monoxide at much higher concentrations than low-temperature PEM, which allows them to run on hydrogen produced from methanol by an onboard reformer, and their heat is easier to use. Several maritime developers have built systems that pair methanol reformers with high-temperature PEM stacks for vessels where methanol storage is convenient. Rules, ports and the global framework Ships carrying low-flashpoint fuels are governed by the IMO's International Code of Safety for Ships using Gases or other Low-flashpoint Fuels, the IGF Code, which was written primarily around liquefied natural gas. Hydrogen is not yet covered by detailed prescriptive requirements. Hydrogen ships have so far been approved through the alternative design route, in which the designer must demonstrate through risk assessment that the ship is at least as safe as one built to the prescriptive rules. The IMO adopted interim guidelines for the safety of ships using fuel cell power installations in 2022, and classification societies such as DNV have published rules for fuel cell installations. The Hydra was approved this way by the Norwegian Maritime Authority, an extended process that every first-of-a-kind vessel has faced. Over time, experience from these vessels will be codified into prescriptive rules, reducing the cost and time of approval. Ports and bunkering are the other constraint. Liquid hydrogen bunkering infrastructure is almost non-existent outside demonstrations. Kawasaki Heavy Industries' Suiso Frontier, a small liquid hydrogen carrier with a 1,250-cubic-metre tank, made the first international shipment of liquid hydrogen from Australia to Japan in early 2022, demonstrating that the cargo could be moved by sea, but large-scale hydrogen trade by ship remains years away. The global regulatory framework has stalled. In April 2025 the IMO's Marine Environment Protection Committee approved a draft Net-Zero Framework that combined a global fuel standard with a pricing mechanism that would charge ships for emissions above set thresholds and reward those using zero or near-zero fuels. At an extraordinary session on 17 October 2025, after strong opposition led by the United States and Saudi Arabia, member states voted by 57 to 49 to adjourn the adoption for one year. Without a global carbon price, zero-emission fuels face a large cost disadvantage on international routes. The European Union has gone ahead with regional rules. Since 2024, shipping emissions on voyages to and from EU ports have been included in the EU Emissions Trading System in phases. Since January 2025, the FuelEU Maritime regulation has required ships calling at EU ports to reduce the greenhouse gas intensity of the energy they use by 2 percent from 2020 levels, rising to 6 percent in 2030 and 80 percent in 2050, with extra credit for renewable fuels of non-biological origin such as green hydrogen and its derivatives. These measures give European ferry and short-sea operators a reason to pay for clean fuels that deep-sea shipping does not yet have. The maritime picture thus reinforces the argument of the earlier chapters. Hydrogen fuel cells work at sea, and the first ferries prove it. Their best opportunities lie on fixed routes too long for batteries where hydrogen can be produced locally, while the large ocean-going fleet moves towards hydrogen-derived fuels such as methanol and ammonia. In every case, the decisive factors are the cost of the fuel and the safety frameworks that allow it to be handled, which brings the book to its final technical subject. Chapter 8: Safety, Standards and the Path to Scale Public anxiety about hydrogen tends to begin with the Hindenburg and end with a vague sense that the gas is uniquely explosive. Engineers who work with it hold a more nuanced view. Hydrogen is hazardous in ways that differ from petrol, diesel and natural gas, sometimes worse and sometimes better. It can be handled safely, as the chemical industry has done at enormous scale for a century, but only with systems designed around its particular behaviour, and with codes that encode that knowledge. As hydrogen moves from refineries into vehicles, depots, ports and ships operated by people who are not chemical engineers, those codes and the culture behind them become decisive. This chapter explains the hazards, the standards that manage them and the industrial conditions that must be met before fuel cell trucks and ships can move from fleets of dozens to fleets of tens of thousands. How hydrogen behaves when things go wrong Hydrogen is the smallest molecule, and it leaks through gaps and seals that would contain larger molecules. It diffuses quickly through air, and it is very buoyant, about one-fourteenth the density of air. These properties combine to an important practical advantage: outdoors, released hydrogen rises and disperses far faster than petrol vapour or propane, which pool at ground level. A leaking hydrogen tank in the open usually creates a flammable cloud that is small and short-lived. Against this, hydrogen is flammable across an unusually wide range of concentrations in air, from about 4 percent to about 75 percent by volume, compared with roughly 1 to 8 percent for petrol vapour. Its minimum ignition energy, around 0.02 millijoules at the most reactive mixture, is roughly a tenth of that of petrol vapour; a static spark from clothing can ignite it. In confined spaces, where hydrogen cannot escape upward, accumulated gas can reach concentrations that support explosions, and in sufficiently large and congested volumes the flame can accelerate towards detonation. The lesson is that hydrogen's danger depends overwhelmingly on confinement. Garages, workshops, ship compartments, tunnels and enclosures around equipment are where hydrogen must be managed most carefully. Hydrogen flames are nearly invisible in daylight and radiate less heat than hydrocarbon flames. A person standing near a burning hydrocarbon spill is endangered by radiant heat; a person near a hydrogen jet fire is relatively safe until they walk into it, which they may do because they cannot see it. Thermal cameras and flame detectors that work in ultraviolet or infrared bands are standard at hydrogen facilities. Hydrogen is not toxic, and burning it produces no smoke, which matters in fires in enclosed spaces where smoke usually kills more people than flames. Hydrogen also attacks some materials. Atoms of hydrogen diffuse into certain steels, particularly high-strength steels, and make them brittle, a phenomenon called hydrogen embrittlement. Piping, valves and tank components must be made from compatible materials, such as particular austenitic stainless steels and aluminium alloys, and qualified by testing. Natural gas can be odorised so that leaks are smelled, but the sulphur compounds used as odorants would poison a fuel cell, so hydrogen systems rely instead on electronic leak detection. Liquid hydrogen adds its own hazards. Contact with the liquid or uninsulated cold surfaces causes severe cold burns. Air in contact with surfaces at liquid hydrogen temperatures condenses, and because oxygen condenses at a higher temperature than nitrogen, the liquid that drips off can be enriched in oxygen, creating a fire hazard near combustible materials. A liquid tank whose relief systems fail can build pressure until it ruptures. These risks are well understood from decades of space launch and industrial practice, but they are new to truck depots and ferry terminals. Real incidents illustrate these points. In June 2019 a hydrogen refuelling station at Sandvika near Oslo exploded and caught fire. The investigation found that a plug in a high-pressure storage unit had been incorrectly assembled, allowing hydrogen to leak and ignite. No one was seriously injured, but the station operator temporarily closed its stations across Norway. In May 2019 an explosion at hydrogen storage tanks at a research facility in Gangneung, South Korea, killed two people; investigators traced it to oxygen contamination in hydrogen produced by an electrolyser, which formed an explosive mixture within the tanks. Both accidents arose from failures of assembly, design or process control rather than from the inherent impossibility of handling hydrogen safely. Both also had wide effects on public confidence, a reminder that in a nascent industry each accident costs more than its direct damage. The standards that make hydrogen vehicles possible A dense web of standards governs every stage of a hydrogen vehicle's life, and understanding its structure helps explain both why hydrogen vehicles are safe and why they are slow to develop. At the vehicle, the foundation is the United Nations Global Technical Regulation No. 13 on hydrogen and fuel cell vehicles, implemented in Europe through UN Regulation No. 134 and in other jurisdictions through national equivalents. It sets requirements for the integrity of the hydrogen storage system, including burst pressure, cycle life, fire exposure and permeation, and for the fuel system and electrical safety after a crash. Tanks must carry thermally activated pressure relief devices that vent the contents in a fire before the composite weakens enough to rupture. The vent is directed upward or away from occupants. Industry standards such as SAE J2579 cover the same ground with test details. At the station, ISO 19880-1 sets general safety requirements for hydrogen fuelling stations, and fuelling protocols such as SAE J2601 define how quickly a tank may be filled, at what temperature, and with what communication between vehicle and dispenser, to prevent overheating or overfilling. Heavy-duty fuelling protocols for high-flow 700-bar dispensing have been developed more recently to match the larger tanks of trucks. The subcooled liquid process developed by Daimler Truck and Linde has been taken through an ISO process so that any manufacturer can use it. Fuel quality is governed by ISO 14687, whose limits on impurities protect the fuel cell, as Chapter 3 explained. In the United States, NFPA 2, the Hydrogen Technologies Code, governs installations from production to dispensing. At sea, the IGF Code and the interim fuel cell guidelines discussed in Chapter 7 apply, supplemented by classification society rules and national maritime authorities' approval of alternative designs. Beyond these formal standards lie the many practical rules that shape operations: whether hydrogen trucks may use particular tunnels, how workshops must be ventilated and equipped with detectors, how much hydrogen may be stored on a depot site before stricter industrial safety regimes apply, what training technicians need. Each of these can delay a deployment by months if it is not settled in advance. A fleet operator introducing fuel cell trucks has to modify its maintenance workshop, typically with ceiling-level ventilation, hydrogen detection, explosion-protected electrical equipment near the roof and procedures for defuelling. Experience from bus depots, which have managed hydrogen fleets for years, has made these modifications routine, but they are still a cost to be counted. What buses have already taught Fuel cell buses are the closest precedent for heavy-duty hydrogen vehicles in commercial service, and their history over the past fifteen years contains lessons that truck and ship operators can use. Europe has run the largest coordinated programmes. Projects funded by the European Union's Fuel Cells and Hydrogen Joint Undertaking, now the Clean Hydrogen Partnership, including the JIVE programmes, put several hundred fuel cell buses into service in cities such as Aberdeen, Cologne, Wuppertal and London. California's transit agencies, notably AC Transit in the San Francisco Bay Area and SunLine Transit in the Coachella Valley, have operated fuel cell buses for years. These fleets demonstrated that fuel cell buses can run full shifts in regular service, refuel in minutes and keep stacks running for tens of thousands of hours. They also revealed the weaknesses. Availability, the share of days on which a bus was ready for service, was frequently lower than for diesel buses in early years, often because of hydrogen supply problems or station outages rather than faults in the buses themselves. Maintenance required specialised skills and parts that took time to obtain. And the cost of hydrogen, typically well above diesel parity, burdened transit budgets once demonstration funding ended. Some cities drew sharp conclusions. In early 2022, Montpellier in France cancelled a planned purchase of hydrogen buses after finding that their operating cost would be several times that of battery-electric buses, and chose battery buses instead. Other cities have continued with hydrogen, particularly where routes are long, where depots lack the grid capacity for large-scale charging, or where local hydrogen production was available. The split mirrors exactly the logic set out in Chapter 6: where batteries can do the job, they are generally cheaper; where they cannot, hydrogen remains the alternative. The most transferable lessons concern operations rather than technology. Successful bus fleets secured long-term hydrogen supply contracts before buying vehicles, built station redundancy so that a single compressor failure did not ground the fleet, trained their own technicians rather than relying entirely on manufacturers, and worked with fire services and local authorities from the beginning to settle safety approvals for depots. Truck fleets and ferry operators who follow these practices avoid the most common causes of disappointment. Materials and the industrial base Scaling up fuel cell trucks and ships to a material share of the fleet would put demands on materials and manufacturing that deserve a sober look. Platinum is the most frequently cited constraint. Global mine supply of platinum is on the order of 170 to 190 tonnes a year, concentrated heavily in South Africa, with Russia and Zimbabwe the other main producers. A heavy truck with about 100 grams of platinum in its stacks, as estimated in Chapter 2, implies that 100,000 trucks a year would need about 10 tonnes, roughly 5 to 6 percent of mine supply. That is manageable, particularly as platinum use in diesel catalytic converters declines with the diesel fleet and as recycling recovers most of the platinum from end-of-life stacks. A million fuel cell trucks a year at current loadings would be a different matter, and this is one reason the industry continues to pursue lower loadings. Iridium, used in the anodes of PEM electrolysers that produce hydrogen, is far scarcer, with annual production of only several tonnes, and is a genuine constraint on scaling PEM electrolysis. Alkaline electrolysers, which use nickel catalysts, and new anode designs with drastically reduced iridium are the responses. Carbon fibre for high-pressure tanks is available from several producers in Japan, the United States, Europe and China, but heavy-duty demand would require major capacity expansion, and its cost and energy intensity make liquid hydrogen tanks attractive for that reason as well. Manufacturing capacity for stacks is currently far ahead of demand rather than behind it. Several manufacturers have built plants with capacities of thousands to tens of thousands of systems a year that are running well below capacity. The binding constraint on fuel cell trucks in the mid-2020s is not the ability to make them but the willingness of customers to buy them, which depends on fuel cost and availability. A decision framework for engineers and operators The arguments of this book can be condensed into a sequence of questions that any engineer or operator considering hydrogen for a heavy transport application should answer, in order. Can the energy requirement be met by batteries with acceptable payload, charging time and grid connection? If so, batteries will almost certainly be cheaper to operate, because they use about a third of the electricity per kilometre. If not, where will the hydrogen come from, at what delivered price, and with what emissions? A credible answer names the production plant, the delivery method and the station, with a price at or near the diesel parity calculated in Chapter 6. What storage form suits the mission? Compressed gas for shorter routes and on-site production; liquid hydrogen for longest range and centralised supply; chemical carriers for most of deep-sea shipping. What durability will the duty cycle demand, and does the system's control strategy protect the stack from high voltage, start-stop damage and humidity cycling? Can the heat be rejected at the worst ambient conditions on the route? Are the safety approvals, tunnel rules, workshop modifications and training settled? Who carries the risk if the hydrogen supply fails or its price rises? Most past failures in this industry trace to that question being left unanswered. Applications that pass all seven questions exist today: captive fleets near industrial hydrogen, high-utilisation long-distance trucking on corridors with committed stations, ferries on routes too long for batteries with local production, mining and port equipment. Applications that fail the first question should not use hydrogen at all. The difficult middle, where hydrogen might win if its cost falls, is where the next decade's engineering and policy will be decided. Conclusion The proton exchange membrane fuel cell has passed the test that matters most for any energy technology: it works, repeatedly and predictably, in real service. A fleet of trucks in Switzerland has driven ten million kilometres. Liquid hydrogen prototypes have crossed a thousand kilometres on a single fill and worked in customer fleets. A ferry has carried cars across a Norwegian fjord on liquid hydrogen for more than three years. The physics of the cell is understood in detail, its degradation mechanisms are catalogued and countered, its manufacture is industrialised. The questions that dominated the field twenty years ago, whether a stack could be made cheaply enough, compactly enough and durably enough for a vehicle, have largely been answered for cars and are being answered for trucks. What follows from this is uncomfortable for both the advocates and the critics of hydrogen. For the advocates, it means that the remaining barriers cannot be solved by better stacks. A fuel cell truck that needs three to four times as much primary electricity per kilometre as a battery truck cannot overcome that ratio by engineering inside the vehicle. A station that sells a fraction of its design throughput will charge high prices regardless of the truck's efficiency. A liquid hydrogen supply chain that trucks fuel a thousand kilometres from the liquefier will carry the energy and emissions costs of that journey whatever the ferry at the end of it does. The future of hydrogen transport is therefore decided by the economics of producing, moving and dispensing the molecule, and by the patience of the institutions that must build that system at the same time as the vehicles. For the critics, it means that dismissing hydrogen in heavy transport is premature, because the cases where it is the right answer are real and identifiable. Where energy demands per vehicle are very large, where vehicles run continuously, where grid connections are weak or slow to build, where payload is precious and where hydrogen can be produced near its use, the fuel cell's advantages are decisive. Ships on routes too long for batteries, trucks in multi-shift long-distance operation on corridors with committed supply, and heavy equipment in mines and ports all fall into this category. In deep-sea shipping the derivatives of hydrogen, ammonia and methanol, are likely to carry most of the load, but they begin with the same electrolysers and the same economics. Three practical conclusions follow for those who must make decisions. The first is for engineers. The most valuable work in heavy-duty fuel cell systems is now in the integration layer: control strategies that extend stack life by keeping cells within benign voltage and humidity windows, thermal designs that reject heat at the worst ambient conditions without derating, storage systems that balance density, dormancy and cost, and energy management that uses route knowledge to size and use batteries intelligently. Stack chemistry still matters, especially reductions in platinum and improvements in high-temperature tolerance, but the competitive advantage of the next decade lies in systems. The second is for fleet operators. A fuel cell truck should never be bought separately from a hydrogen supply contract that fixes price, volume and reliability for the life of the vehicle. The operators who succeed with hydrogen will be those who control or contract their fuel supply, run vehicles at high utilisation on predictable routes and share stations with other fleets to keep throughput high. Those who buy fuel cell trucks in the expectation that public infrastructure will appear and prices will fall are taking a risk that has already bankrupted their suppliers. The third is for policymakers. Hydrogen transport requires coordinated support for vehicles, production and stations, sustained over a decade, in specific corridors and clusters, not scattered across a country. The European approach, combining binding fleet emission standards, an infrastructure mandate on core corridors and carbon pricing that extends to shipping, gives a framework within which hydrogen can find its place. The American retreat from clean hydrogen incentives and the delay of the global maritime framework show how quickly the economics can turn when policy changes. Above all, support should follow the efficiency logic: subsidise hydrogen where batteries cannot do the job, not where they can. The next five years will supply evidence that settles many of the arguments in this book, and it is worth knowing which signals matter. The first is the delivered price of clean hydrogen at heavy-duty stations with real throughput: if contracted prices at corridor and port stations fall towards 6 to 8 euros or dollars per kilogram, the truck economics of Chapter 6 change materially; if they remain in double digits, fuel cell trucks will stay in niches. The second is measured stack life in trucks, not fleet kilometres but hours on individual stacks with their decay rates, which Daimler's small series, Hyundai's fleets and Toyota's third-generation systems will begin to reveal. The third is the operating record of the Lofoten ferries, which will show whether a large hydrogen vessel with local production can run reliably every day of an Arctic year. The fourth is policy at the IMO, where the adjourned Net-Zero Framework will return, and in the major truck markets, where the durability of fleet emission standards will decide how much manufacturers invest. None of these signals depends on a scientific breakthrough. All depend on execution. The image with which this book began, a 40-tonne truck setting off before dawn on a long day's work, will look different in twenty years. Most likely it will be electric, and in many cases it will carry its energy in a battery charged during its drivers' rest periods. In some cases, on the longest routes, the heaviest loads, the hardest climates or the places where the grid cannot reach, it will carry hydrogen and convert it into electricity through a stack of thin polymer membranes coated with platinum, quietly turning a chemical reaction into motion. At sea, ferries running on hydrogen will be joined by larger ships burning or electrochemically converting fuels made from it. The engineering to make that happen exists. Whether it is used depends on whether hydrogen can be made, moved and sold at a price the work will bear, and on whether those deciding where to use it are honest about where it belongs. Notes The following sources underpin the specific figures and events cited in the text. Physical property values for hydrogen, diesel and marine fuels are standard reference data, such as those compiled by the U.S. National Institute of Standards and Technology. U.S. Department of Energy, Hydrogen and Fuel Cell Technologies Office, "Hydrogen Class 8 Long Haul Truck Targets," Program Record 19006 (2019), and subsequent Annual Merit Review presentations of the Fuel Cell Technologies programme. Source of the heavy-duty targets of 80 dollars per kilowatt, 25,000 hours and 68 percent peak efficiency by 2030, and 60 dollars per kilowatt, 30,000 hours and 72 percent ultimately. Daimler Truck, press releases on the GenH2 Truck, including the 1,047-kilometre record run (2023), the sLH2 refuelling technology developed with Linde and the Wörth am Rhein pilot station (February 2024), and the second customer trial phase (November 2025), reported by electrive.com on 14 November 2025. Source of the 225,000 kilometres, 5.6 to 8.0 kilograms per 100 kilometres, 2 × 150 kilowatt fuel cell, 70 kilowatt-hour battery, 80-kilogram tank capacity, the 100-unit small series from late 2026 and series production after 2030. Hyundai Motor Company, announcement that the XCIENT Fuel Cell fleet in Switzerland had surpassed 10 million kilometres, June 2024, reported by Korea JoongAng Daily. Source of the 48-truck fleet and 2 × 90 kilowatt fuel cell configuration. Trucking Dive and Manufacturing Dive, reports on Nikola Corporation's Chapter 11 filing, February 2025. Cleanthinking.de and electrive.com reports on cellcentric, including the 2024 delay of the Weilheim factory and the 2026 agreement to bring Toyota in as a third equal shareholder; source of the figure that hydrogen trucks took about 0.1 percent of European heavy commercial registrations in 2025. Toyota Motor Corporation, announcement of its third-generation fuel cell system, February 2025. S&P Global Commodity Insights, "California hydrogen pump prices for light duty vehicles reach new highs," 1 October 2024. Source of the 34.55 dollars per kilogram average and station availability figures. International Energy Agency, Global Hydrogen Review (annual editions 2024 and 2025). Source of global hydrogen demand and low-emission production shares. International Maritime Organization, Fourth IMO Greenhouse Gas Study (2020), for shipping's emissions in 2018; and reports by Seatrade and others on the adjournment of the Net-Zero Framework at the extraordinary session of the Marine Environment Protection Committee, 17 October 2025. Norled and Riviera Maritime Media reports on MF Hydra, and its public technical record, for the 80-cubic-metre liquid hydrogen tank, 2 × 200 kilowatt Ballard fuel cells, 1.36 megawatt-hour battery and March 2023 entry into service. Statens vegvesen (Norwegian Public Roads Administration), "Hydrogen ferries to Lofoten"; H2 International and Maritime Executive reports on Torghatten Nord, GreenH and MAN Cryo. Source of the route, the 85 percent hydrogen requirement, vessel dimensions and capacity, and the five to six tonnes per day consumption. Regulation (EU) 2024/1610 amending CO2 emission standards for new heavy-duty vehicles; Regulation (EU) 2023/1804 on the deployment of alternative fuels infrastructure; Regulation (EU) 2023/1805 (FuelEU Maritime). ISO 14687, Hydrogen fuel quality, product specification; United Nations Global Technical Regulation No. 13 on hydrogen and fuel cell vehicles; SAE J2601 fuelling protocols; ISO 19880-1 on hydrogen fuelling stations; IMO MSC.1/Circ.1647, Interim guidelines for the safety of ships using fuel cell power installations (2022). Norwegian investigation of the Sandvika (Kjørbo) hydrogen station explosion of June 2019, as reported by the station operator and equipment supplier; Korean government investigation of the Gangneung hydrogen tank explosion of May 2019. Anglo American, announcement of the nuGen hydrogen-powered mining haul truck demonstration at Mogalakwena, May 2022. Glossary Balance of plant. The pumps, compressors, blowers, humidifiers, valves, sensors and power electronics that supply a fuel cell stack and remove its products; their power consumption separates stack efficiency from system efficiency. Bipolar plate. The thin conductive plate between adjacent cells that distributes hydrogen on one face and air on the other, carries coolant and connects the cells electrically in series. Boil-off. Evaporation of a cryogenic liquid caused by heat leaking into its tank, which raises pressure and eventually forces venting or consumption of the gas. Catalyst layer. The porous layer of platinum nanoparticles on carbon, bound with ionomer, in which the electrochemical reactions take place. Electrolyser. A device that uses electricity to split water into hydrogen and oxygen, the reverse of a fuel cell. Gas diffusion layer. The carbon fibre sheet that spreads reactant gas across the catalyst layer, conducts electrons and heat, and controls the escape of liquid water. Lower heating value. The heat released by burning a fuel when the water formed leaves as vapour; for hydrogen about 33.3 kilowatt-hours per kilogram, the basis for energy figures in this book. Membrane electrode assembly. The core of each cell: the proton-conducting membrane with its two catalyst layers and, usually, the two gas diffusion layers. Polarization curve. The plot of cell voltage against current density that summarises a fuel cell's performance and its activation, ohmic and mass transport losses. Subcooled liquid hydrogen. Liquid hydrogen pumped to a pressure above its vapour pressure so that it can absorb heat without boiling, allowing refuelling without return gas. Type IV tank. A high-pressure gas cylinder with a non-load-bearing polymer liner and a carbon fibre composite shell that carries the pressure. Further Reading Ryan O'Hayre, Suk-Won Cha, Whitney Colella and Fritz B. Prinz, Fuel Cell Fundamentals (Wiley, 3rd edition, 2016).** The clearest university-level introduction to fuel cell thermodynamics, kinetics and transport, with worked problems. The natural next step after Chapters 1 and 2. Frano Barbir, PEM Fuel Cells: Theory and Practice (Academic Press, 2nd edition, 2013).** An engineer's treatment of PEM stacks and systems, strong on water and heat management, system design and the practical behaviour of real hardware. Andrew L. Dicks and David A. J. Rand, Fuel Cell Systems Explained (Wiley, 3rd edition, 2018).** A broad, readable survey of all major fuel cell types and their applications, useful for placing PEM technology alongside solid oxide and other cells relevant to ships. Rod Borup and colleagues, "Scientific aspects of polymer electrolyte fuel cell durability and degradation," Chemical Reviews 107 (2007).** The classic review of degradation mechanisms, still the foundation for understanding the subject of Chapter 3. David A. Cullen and colleagues, "New roads and challenges for fuel cells in heavy-duty transportation," Nature Energy 6 (2021).** A concise statement, by researchers from the U.S. national laboratories, of why heavy-duty requirements differ from light-duty ones and what research they demand. Kui Jiao and colleagues, "Designing the next generation of proton-exchange membrane fuel cells," Nature 595 (2021).** A forward-looking review of the materials and design advances needed to raise power density and durability. U.S. Department of Energy, Hydrogen Program Record 19006, "Hydrogen Class 8 Long Haul Truck Targets" (2019). The short document that set the quantitative targets for cost, durability and efficiency quoted throughout this book, with the reasoning behind them. International Energy Agency, Global Hydrogen Review (published annually).** The most comprehensive yearly survey of hydrogen production, demand, policy and costs worldwide, essential for tracking whether the economics described in Chapter 6 are changing. International Council on Clean Transportation, studies of zero-emission trucks. The ICCT's comparative analyses of battery-electric and fuel cell trucks, including total cost of ownership studies for Europe, the United States and China, offer transparent assumptions that readers can adjust. DNV, Maritime Forecast to 2050 (published annually).** A classification society's assessment of alternative fuels for shipping, regulatory developments and fleet uptake, with clear discussion of hydrogen, ammonia and methanol. Hashtags: #HydrogenFuelCellEngineering #HydrogenFuelCells #PEMFuelCells #ProtonExchangeMembrane #FuelCellStacks #FuelCellTechnology #HydrogenEngineering #ElectrochemicalEnergyConversion #MembraneElectrodeAssembly #PlatinumCatalysts #BipolarPlates #FuelCellDurability #FuelCellEfficiency #WaterManagement #ThermalManagement #HydrogenStorage #CompressedHydrogen #LiquidHydrogen #CryogenicHydrogen #HydrogenRefueling #HeavyDutyTransportation #HydrogenTrucks #FuelCellElectricVehicles #ZeroEmissionTrucks #SustainableTransportation #GreenHydrogen #HydrogenInfrastructure #HydrogenSafety #MaritimeDecarbonization #CleanEnergyTransition

Download the Book (PDF): Introduction Every autumn, somewhere around the second or third week of October, a certain kind of student begins to lose the thread. Nothing dramatic happens. The alarm goes off and the room is still dark, so the snooze button gets pressed twice instead of once. The eight o'clock lecture becomes a recording to watch later. Lunch turns into an afternoon of cereal and bread. A friend texts about a movie night and the reply is "maybe next time," typed from under a blanket at six in the evening, when it already feels like midnight. By early December the student has fallen behind in two classes, feels heavy and slow, sleeps nine or ten hours without waking rested, and has started to wonder whether something is wrong with them as a person. Then, in April, the fog lifts. Energy returns, friends reappear, and the whole winter feels like something that happened to somebody else. Until October comes round again. That pattern has a name. Seasonal affective disorder, usually shortened to SAD, is a form of depression in which episodes arrive and depart with the seasons, most often beginning in autumn or early winter and clearing in spring. It was described as a clinical syndrome in 1984 by Norman Rosenthal and colleagues at the National Institute of Mental Health, who reported on 29 patients whose depressions recurred at the same time each year and who seemed to improve when their days were artificially lengthened with bright light. In the four decades since, it has become one of the better understood forms of depression, with a reasonably clear biological story and several treatments that have been tested head to head in randomized trials. For college students it has a particular shape, because the academic year and the solar year run on top of each other. The fall semester begins in late summer light and ends in the darkest weeks of the year, just as exams, papers and final projects arrive. The spring semester opens in January, at the bottom of the light curve, after a winter break that often dissolves whatever routine a student had built. Many students are living for the first time far from home, sometimes hundreds of miles farther north than where they grew up. Many keep late hours that push their internal clocks later still, so they wake in darkness and spend their mornings indoors. If you set out to design conditions that would bring out a latent tendency toward winter depression, you could do worse than a northern campus between November and February. What this book argues The central claim of this book is simple, and it runs against the way most people handle winter mood. Seasonal depression is one of the most predictable forms of mental illness there is. It tends to arrive at the same point in the year, in the same person, with the same early signs. That predictability is usually experienced as a curse: the dread of knowing it is coming. It is better understood as an opportunity. A problem you can see coming is a problem you can plan for, and the students who do best with SAD are generally not the ones who endure winter most bravely but the ones who treat it as a season to prepare for, starting while they still feel well. The preparation rests on three supports, which correspond to the three pillars in this book's description. The first is light: specifically, bright light delivered at the right time of day, which acts on the brain's clock and on mood and which has the strongest and quickest evidence behind it. The second is the body's chemistry more broadly, including vitamin D, a subject where popular enthusiasm has run far ahead of the evidence and where a clear-eyed account can save students money and false hope. The third is behavior and connection: the daily structure, the activity and the social contact that winter quietly erodes, and that a specific psychological treatment for SAD, developed and tested over many years by the psychologist Kelly Rohan, rebuilds deliberately. That last pillar turns out to matter more than people expect, because the best long-term trial evidence suggests that skills learned in therapy protect people in later winters in a way that light alone may not. Who this book is for The primary reader is a college or university student who suspects that winter hits them harder than it should. You may have a diagnosis already, or you may simply have noticed that your grades, your sleep and your friendships sag in the same months every year. The book is also written for the people around such students: parents who notice that phone calls in January sound different from phone calls in September, residence life staff and academic advisers who see the same students struggle every winter, and friends and roommates who want to help without being intrusive. Graduate students, who often work in windowless labs through long northern winters with little built-in social structure, will find that much of it applies to them with extra force. The examples in these pages are illustrative composites, not real people or case reports. They are drawn to show how the research plays out in ordinary campus life: the sophomore who cannot get out of bed for a morning seminar, the international student from a tropical country facing her first Canadian or Scandinavian winter, the athlete whose training schedule collapses over break. Where the book reports numbers, they come from published studies, which are listed chapter by chapter in the Notes. How the book is organized The first chapter draws the line between ordinary winter blues and clinical seasonal depression, using the diagnostic criteria clinicians actually apply and explaining why the difference matters for what you do next. The second chapter explains why darkness affects mood at all, walking through the roles of latitude, the circadian clock, melatonin and serotonin, and what the research does and does not yet show. The third chapter lays the academic calendar over the solar calendar and identifies the specific weeks and transitions where students are most vulnerable. The middle of the book turns to treatment. Chapter 4 covers bright light therapy in practical detail: how strong the light should be, when to use it, how long it takes to work, how to fit it into a dorm-room morning, and who should not use it without medical supervision. Chapter 5 examines vitamin D, separating what is known about deficiency from what is known about supplements and mood, which are not the same thing. Chapter 6 explains cognitive-behavioral therapy tailored for SAD, including the evidence that it may protect against future winters better than light therapy does. Chapter 7 addresses social life in winter, the gradual withdrawal that SAD encourages and the practical ways students keep themselves connected. Chapter 8 covers professional help on campus, medication, academic accommodations and what to do in a crisis. The conclusion draws these threads into a single argument about planning for winter before it arrives. A note on safety and scope This book is written to inform, not to diagnose or treat. It cannot replace an assessment by a physician, psychologist, psychiatrist or other qualified clinician who knows your history, and several of the treatments discussed, including light therapy and medication, can have real side effects for some people, especially those with bipolar disorder or certain eye conditions. If you recognize yourself in these pages, the most useful next step is a conversation with your campus health or counseling service. Seasonal depression is still depression, and depression can be dangerous. If you are having thoughts of suicide or of harming yourself, or you are worried about someone else, do not wait for spring. In the United States you can call or text 988, the Suicide and Crisis Lifeline, at any hour, or chat online at 988lifeline.org. If someone is in immediate danger, call 911 or go to the nearest emergency department. Chapter 8 goes into more detail about crisis steps and how to use them. With those points made once, the rest of the book gets on with the practical business of understanding winter depression and managing it through an academic year. Chapter 1: Winter Blues or Winter Depression? Almost everyone feels some difference between July and January. Days are shorter, the weather keeps people indoors, and the novelty of a new semester has long worn off. It would be strange if mood did not shift at all. The useful question for a student is not whether winter affects them, but how much, and whether the change has crossed from an ordinary seasonal dip into an illness that deserves treatment. That line matters because the two call for different responses. Winter blues respond to common sense. Seasonal affective disorder often does not, and treating it as a character flaw or a bad attitude wastes months that could have been spent getting better. A spectrum, not a switch The best early evidence on how common seasonal mood change is came from a telephone survey of randomly selected households in Montgomery County, Maryland, published by Siegfried Kasper and colleagues in 1989. They used the Seasonal Pattern Assessment Questionnaire, a standard screening tool that asks people how much their sleep, social activity, mood, weight, appetite and energy vary across the year. The findings were striking. Ninety-two percent of respondents noticed some seasonal change in mood and behavior. For 27 percent, those changes were a problem. And between 4.3 and 10 percent, depending on how strictly the researchers defined a case, reported seasonal impairment on the level of patients diagnosed with SAD. The authors concluded that seasonal affective disorder sits at the far end of a continuum of seasonality that runs through most of the population. More recent work supports that picture. A 2025 systematic review and meta-analysis in the Journal of Affective Disorders, pooling 24 studies with nearly 33,000 participants from around the world, estimated the prevalence of SAD at about 5 percent and of subsyndromal SAD, a milder form that causes real distress without meeting full diagnostic criteria, at a little over 9 percent. Summer-pattern SAD, in which depression arrives in the warm months, was far rarer, at around half of one percent. Those numbers suggest a practical framework. Roughly, people fall into three groups. Most notice that they sleep a bit more and feel a bit flatter in winter but function normally; this is the territory usually called the winter blues. A substantial minority experience subsyndromal SAD, in which the winter change is large enough to hurt grades, relationships and wellbeing, but falls short of a major depressive episode. A smaller group experience full seasonal major depression. The categories blur into one another, and a student can move between them from year to year depending on stress, sleep, geography and how well they manage the season. That fluidity is one reason the strategies in this book are useful across the whole range: the same tools that treat full SAD can keep a milder pattern from getting worse. In the formal diagnostic system used in the United States, the fifth edition of the American Psychiatric Association's Diagnostic and Statistical Manual of Mental Disorders (DSM-5, and its 2022 text revision), seasonal affective disorder is not a separate diagnosis. It is a specifier, a label added to a diagnosis of recurrent major depressive disorder or of bipolar I or bipolar II disorder. The diagnosis reads something like "major depressive disorder, recurrent, with seasonal pattern." The specifier has four main requirements. First, there has to be a regular relationship between the onset of depressive episodes and a particular time of year, most commonly autumn or winter. Second, full remission, or a switch from depression into mania or hypomania, has to occur at a characteristic time of year as well, usually spring. Third, over the past two years the person must have had at least two major depressive episodes that show this seasonal relationship, and no episodes outside the pattern during that period. Fourth, across the person's lifetime, seasonal episodes must substantially outnumber non-seasonal ones. Each of these requirements carries a lesson. The first two mean that SAD is defined by repetition, not by one bad winter. A first-year student who has a hard first January away from home does not yet meet the pattern, even if the episode is severe, and should be assessed and treated for depression in the ordinary way. The third and fourth mean that a clinician will want to know about previous years, which is why keeping some kind of record, even an informal one, is valuable. The manual adds one more qualification that is especially relevant on campus. The seasonal pattern should not be explained by obvious seasonal psychosocial stressors. The textbook example is someone who is regularly unemployed every winter. For students, the obvious candidate is the academic calendar itself. If you feel terrible every December, is that the darkness, or is it finals? If January is bleak, is that the light, or the loneliness of returning to campus after break? A thoughtful clinician will try to tease these apart, and Chapter 3 returns to this question in detail. In practice, the answer is often both, and the treatments that work for SAD tend to help with seasonal stress as well. But students should know that the question is a fair one and not a sign they are being dismissed. What winter depression looks like The symptoms of a major depressive episode are the same whether or not it has a seasonal pattern: persistent low mood or loss of interest and pleasure, along with changes in sleep, appetite, energy, concentration and self-worth, lasting at least two weeks and causing real impairment. Winter depression, though, has a characteristic flavor that differs from the stereotype of depression as insomnia and loss of appetite. In Rosenthal's original 1984 description, the depressions were generally marked by hypersomnia, overeating and carbohydrate craving. Those features have held up. People with winter SAD tend to sleep more, not less, often nine or ten hours a night, and still wake unrefreshed. They crave starches and sweets, eat more, and often gain weight over the winter. They feel physically heavy and slow, a sensation sometimes described as having lead in the limbs. They withdraw socially, not always because they feel sad, but because everything takes too much energy. Clinicians sometimes describe the syndrome as a kind of hibernation. For a student, these symptoms show up in recognizable ways. Morning classes become impossible. Assignments that once took two hours take five, because concentration keeps sliding off the page. The student naps in the afternoon and still goes to bed early. Takeout and snacks replace meals. Invitations are declined, and after a while they stop coming. Because hypersomnia and overeating do not match the popular image of depression, students often do not recognize what is happening to them. They conclude instead that they are lazy, undisciplined or not cut out for college, and that interpretation itself deepens the low mood. It is also worth naming what winter depression is not. It is not simply disliking cold weather, or preferring summer, or feeling low on a gray week. It is also not the same as the understandable sadness of homesickness, a breakup or a family loss that happens to fall in winter, though those experiences can trigger or worsen an episode. The distinguishing marks are the recurrence, the characteristic cluster of symptoms, and the degree of impairment. Table 1 sets out the main differences between ordinary winter blues, seasonal affective disorder and non-seasonal depression. It is a guide to the general pattern, not a diagnostic instrument. Table 1. Winter blues, seasonal depression and non-seasonal depression compared. Feature Winter blues Seasonal affective disorder Non-seasonal depression Timing Mild dip in darker months Recurs in autumn or winter, remits in spring Can begin at any time of year Sleep Slightly longer Often much longer, unrefreshing Often insomnia, sometimes excess Appetite Little change Increased, with carbohydrate craving Often reduced, sometimes increased Functioning Largely intact Clearly impaired for weeks Clearly impaired for weeks First response Self-care and routine Clinical assessment, light therapy, CBT Clinical assessment and treatment One feature of seasonal mood disorders deserves special attention, because it affects treatment safety. A meaningful share of people with a seasonal pattern have bipolar disorder rather than unipolar depression. In the 1984 NIMH series, most of the patients had a bipolar disorder, particularly bipolar II, in which depressions alternate with periods of hypomania: elevated or irritable mood, reduced need for sleep, racing thoughts, increased activity and sometimes risky behavior that is noticeable to others but not severe enough to require hospitalization. Later community samples have found that most people with winter depression do not have bipolar disorder, but the link is real, and the seasonal pattern in bipolar disorder can include springtime or summertime highs as well as winter lows. This matters for three reasons. First, a student who feels unusually energetic, talkative and sleepless every spring, and flat every winter, may be describing a bipolar pattern rather than a simple winter depression, and should say so to a clinician. Second, bright light therapy and antidepressant medication can, in susceptible people, trigger hypomania or mania. Both are commonly used in bipolar depression, but they should be used with medical supervision, often alongside a mood stabilizer. Third, spring and early summer are themselves a risky period for some people with seasonal mood disorders, and the relief of feeling better can mask the early signs of an episode going too far in the other direction. Chapter 4 returns to this in the context of light therapy. In the Maryland survey, the people with the highest seasonality scores tended to be younger women who felt worse on short days and who reported that seasonal changes were a problem for them. That pattern has been repeated often enough that clinicians regard youth and female sex as risk factors, along with a family history of depression or of seasonal mood problems, and living far from the equator. College students, in other words, include a high concentration of the people most likely to be affected. A student who also has a personal history of depression, anxiety or an eating disorder, or who is prone to sleeping late, has additional reasons to pay attention. Consider a composite example. A sophomore at a university in the upper Midwest grew up in Georgia. Her first autumn on campus was hard, but she put it down to adjustment. In her second year, the same slide began in late October: she slept through two morning labs in a row, stopped going to the climbing club she had loved in September, and found herself eating dinner alone in her room most nights. By Thanksgiving she was behind in organic chemistry and convinced she had made a mistake choosing a science major. When a residence adviser mentioned that the counseling center ran a workshop on winter mood, she recognized herself in the description of the second winter, and the first. She had not been failing at college. She had been meeting a northern winter for the first time with a body clock set for Atlanta and no plan. Look-alikes and honest self-screening Several medical and lifestyle conditions produce fatigue, low mood and sleepiness in winter, and a sensible assessment rules them out. An underactive thyroid can cause tiredness, weight gain, low mood and sensitivity to cold. Anemia, including iron deficiency, which is common among young women and among students who have cut back on meat, causes fatigue and poor concentration. Infectious mononucleosis and the long tail of other viral illnesses can leave students exhausted for weeks. Sleep disorders, including obstructive sleep apnea and delayed sleep phase, in which the body clock runs so late that the student cannot fall asleep until the small hours and cannot function in the morning, can mimic or compound seasonal depression. Heavy alcohol or cannabis use produces low mood, poor sleep and low motivation. So does chronic sleep deprivation driven by academic workload. None of these is exotic, and most can be checked with a clinical interview and a few routine blood tests. A student visiting campus health about winter fatigue should expect questions about sleep, diet, substances, menstrual cycles and family history, and possibly a blood test for thyroid function and blood count. These are not signs that the clinician doubts the student's mood; they are the ordinary diligence that makes treatment work. There is no blood test for SAD, and diagnosis rests on a clinical interview. But a student can do some useful groundwork alone, and that groundwork often makes the conversation with a clinician much more productive. The first step is a look backward. Think about the last two or three years, one month at a time. In which months were you at your best, sleeping well, doing good work, seeing people? In which months were you at your worst? Did the worst months cluster in the same part of the year? Did anything happen in those months, such as exams, a breakup or a move, that might explain them? Students often find it easier to reconstruct this from external evidence than from memory. Old calendars, the timestamps on social media posts and messages, grade reports, and bank statements showing food delivery orders can all reveal patterns that memory smooths over. The second step is a look forward. Starting in early autumn, keep a brief daily log. It need not be elaborate. A line in a phone note is enough: hours slept, time you got up, a mood rating from one to ten, an energy rating, and whether you spent any time outdoors in daylight. After a few weeks, trends become visible that would be invisible day to day. Many students discover that their mood tracks their wake-up time and their daylight exposure more closely than they expected, which is itself useful information for the chapters that follow. The third step is to take seriously any symptom that goes beyond fatigue and flatness. Persistent hopelessness, a sense that others would be better off without you, thoughts of death or suicide, or any urge to harm yourself mean that you need to talk to someone now, not after a few weeks of self-monitoring. The same goes for symptoms that interfere seriously with your ability to attend class, eat, or look after yourself. Why the label matters Some students resist the idea of a diagnosis, worrying that it will follow them, that it pathologizes a normal reaction to winter, or that it will lead to medication they do not want. These are reasonable concerns, and a good clinician will discuss them openly. But there are strong practical reasons to take the question of diagnosis seriously. The first is that SAD has specific, well-tested treatments, and some of them, notably light therapy and the version of cognitive-behavioral therapy discussed in Chapter 6, were designed for seasonal depression in particular. A student who believes they are simply lazy will not seek them out. The second is that a documented condition can open the door to academic accommodations, such as extended deadlines or reduced course loads, through a campus disability services office, which can make the difference between a hard semester and a failed one. The third is that SAD is recurrent by definition. Understanding the pattern now pays dividends every winter for years to come, because the most effective approach, as later chapters show, is to act before an episode is fully underway. The final reason is less tangible but perhaps the most important. Many students with seasonal depression spend years explaining their winters to themselves in moral terms: as failures of willpower, discipline or character. Learning that there is a biological and psychological mechanism behind the pattern, and that the mechanism is well understood and treatable, often brings real relief. It does not remove responsibility for taking action. It redirects that responsibility toward actions that work. Chapter 2: Why Darkness Gets Into Your Head It is one thing to know that some people feel worse in winter. It is another to understand why, and the why matters for students in a concrete way. Each of the treatments in this book works through a particular mechanism. Light therapy works partly by shifting the body clock, which is why its timing matters so much. Behavioral strategies work partly by restoring the reward and structure that winter strips away. Vitamin D was proposed because of a plausible link between sunlight and brain chemistry. A student who understands the mechanisms can use the treatments intelligently, adjust them when they are not working, and avoid wasting effort on remedies that do not touch the problem. The honest summary is that no single cause explains seasonal affective disorder. Researchers have assembled strong evidence for several contributing processes, some better established than others, and the leading view is that different people have different mixtures. What unites them is light: specifically, the shortening of the day and the reduction in the amount of bright light reaching the eyes during the months around the winter solstice. Latitude and the length of the day The farther you are from the equator, the more day length swings across the year. Near the equator, days are about twelve hours long in every season. At the latitude of Atlanta, roughly 34 degrees north, the shortest day of the year is still close to ten hours long. At Boston, around 42 degrees north, it is a little over nine. At Minneapolis, about 45 degrees north, it is under nine. At Edinburgh, Stockholm or Anchorage, it falls to around seven hours or considerably less, and the sun that does rise stays low in the sky. Those differences are not trivial for a student who spends daylight hours in lecture halls and libraries. Day length is only half of the equation. The other half is how much bright light actually reaches a person's eyes, and here modern campus life makes winter much darker than the calendar alone would suggest. Light intensity is measured in lux. Ordinary indoor lighting in classrooms, dorm rooms and libraries typically provides a few hundred lux at eye level. Outdoors, even an overcast winter day usually delivers several thousand lux, and direct sunlight can exceed 50,000. The gap between indoors and outdoors is far larger than the eye perceives, because the visual system adjusts so smoothly to changes in brightness that a well-lit room feels bright. A student who walks to class in the dark, spends the day indoors and walks home in the dark may go days in winter without receiving a single meaningful dose of bright light, even on a campus where the sun technically shines for nine hours. It makes intuitive sense that SAD should be more common at higher latitudes, and in North America the evidence supports that. In 1990, Leora Rosen and colleagues mailed the Seasonal Pattern Assessment Questionnaire to randomly selected residents of four places at different latitudes on the US East Coast: Nashua, New Hampshire; New York City; Montgomery County, Maryland; and Sarasota, Florida. Rates of winter SAD and of milder subsyndromal SAD were significantly higher in the northern locations, with SAD estimated at 9.7 percent in New Hampshire compared with 1.4 percent in Florida. The 2025 global meta-analysis mentioned in Chapter 1 likewise found a significant positive association between latitude and the prevalence of both SAD and subsyndromal SAD, and no such association for summer-pattern SAD, which suggests that the winter and summer forms have different causes. The relationship is not as neat as the North American numbers imply. A widely cited 1999 review by Peter Mersch and colleagues found that the correlation between latitude and prevalence was clearer in North America than in Europe, and that latitude explained only a modest share of the variation between studies. Climate, cultural adaptation, genetics and how much time people spend indoors all seem to play a part. Populations with long histories at high latitudes, such as Icelanders, have sometimes shown lower rates than would be predicted from their latitude alone. The practical lesson for students is that latitude is a risk factor, not destiny, but that a move northward, especially a large one, deserves to be taken seriously. The body clock and melatonin Every cell in the human body keeps time, and those cellular clocks are coordinated by a master clock in a small region of the brain called the suprachiasmatic nucleus, which sits just above the point where the optic nerves cross. Left to itself, this clock runs on a cycle of roughly, but not exactly, twenty-four hours; in most people it runs slightly long. It stays aligned with the solar day because light entering the eyes resets it each day. Special light-sensitive cells in the retina, which are most responsive to bright light and particularly to the blue portion of the spectrum, send signals directly to the clock. The timing of light is crucial. Light in the morning, after the body's internal night has ended, pushes the clock earlier, which is called a phase advance. Light in the late evening and at night pushes it later, which is called a phase delay. The same amount of light can therefore have opposite effects depending on when it arrives. This principle is the foundation of light therapy, and it explains why a light box used at the wrong time of day can do little good, or can disturb sleep. One of the clock's most important outputs is the hormone melatonin, secreted by the pineal gland during the biological night. Melatonin levels begin to rise in the evening, a couple of hours before a person's natural sleep time, and fall in the morning. Researchers use the time at which melatonin starts rising in dim light, known as the dim light melatonin onset, as a marker of where a person's internal clock is set. In winter, dawn comes later. Without strong morning light to anchor it, the clock of many people drifts later relative to their sleep and work schedules. Alfred Lewy and colleagues at Oregon Health and Science University developed what became known as the phase-shift hypothesis: that many people with winter depression have body clocks that are out of alignment with their sleep schedules, most commonly delayed, so that they are trying to wake and function while their bodies are still in biological night. In a 2006 study published in the Proceedings of the National Academy of Sciences, the team gave patients with SAD low doses of melatonin at different times of day, deliberately shifting their clocks in one direction or the other without making them sleepy. Depression ratings tracked the degree of misalignment between the melatonin rhythm and sleep: the closer patients came to an optimal alignment, the better they felt. Most patients in the study were phase-delayed, and for them morning light and correctly timed afternoon melatonin both moved them in the right direction. The authors noted that SAD may be the first psychiatric disorder in which a physiological marker correlates with symptom severity both before and during treatment. This finding has particular force for college students. Adolescents and young adults naturally have later body clocks than children or older adults. In a very large survey of sleep timing published in 2004, the chronobiologist Till Roenneberg and colleagues found that people’s preferred sleep times grow steadily later through adolescence and reach their latest point at around age 20, after which they begin to move earlier again; the authors proposed this turning point as a biological marker for the end of adolescence. The typical undergraduate is therefore at the lifetime peak of lateness. Students also tend to keep irregular schedules, stay up late with screens and bright rooms, and sleep in on weekends, all of which push the clock later. The result is a population whose clocks are already late before winter makes them later still. A student who feels fine going to bed at 2 a.m. and waking at 10 a.m. in September may find that the same schedule in December means almost no morning daylight at all, and that a 9 a.m. class requires getting up in what the body considers the middle of the night. The winter clock problem has a second half that students control more than they realize: light at night. Because evening light delays the clock, a student who sits under bright overhead lights and a glowing laptop until one in the morning is sending the clock exactly the wrong signal at exactly the wrong time. The morning signal is weak because the mornings are dark; the evening signal is strong because the rooms are bright. The clock slides later, waking becomes harder, and the student gets even less morning light. Research on the precise effect of screens specifically is less settled than headlines suggest, and the brightness of room lighting and the overall timing of light exposure matter at least as much as the device. But the underlying principle, that the body clock integrates all the light it receives and weighs it by time of day, is well established. Dimmer, warmer light in the last couple of hours before bed and bright light soon after waking pull in the same direction. Melatonin is also a seasonal signal in its own right. In many animals, the length of the nightly melatonin pulse tells the body what time of year it is and triggers seasonal changes in breeding, fur growth and hibernation. Early research on SAD proposed that humans might retain a version of this signal, and that a longer winter melatonin pulse might drive the hibernation-like symptoms. The evidence for this in modern humans, who live under artificial light, is mixed, and it is no longer considered the main explanation. But it contributes to the picture of SAD as a disorder of the body's timing systems. Serotonin and the winter brain Serotonin is a chemical messenger involved in mood, appetite and sleep, and the target of the most commonly prescribed antidepressants. There have long been reasons to suspect a seasonal serotonin link in SAD. The carbohydrate craving characteristic of winter depression fits, because carbohydrate-rich meals can influence the brain's supply of tryptophan, the building block of serotonin. And antidepressants that act on serotonin help many people with SAD. A clearer picture emerged from brain imaging. The serotonin transporter is a protein that clears serotonin from the gaps between nerve cells; more transporter activity means serotonin is removed faster and its signal is weaker. In 2016, Brenda Mc Mahon and colleagues in Copenhagen published the first study to scan the same people in both summer and winter using positron emission tomography, measuring serotonin transporter levels in 17 patients with SAD and 23 healthy volunteers with low seasonality, across 80 scans. In summer, the two groups looked similar. In winter, the SAD patients had higher serotonin transporter binding than the healthy volunteers. Healthy people adjusted their transporter levels across the seasons, while the patients changed much less. And among patients, the size of the seasonal change in transporter binding was linked to the size of the change in their depression scores. The authors suggested that winter depression may involve a failure to adjust serotonin transport appropriately to the environmental stress of winter, particularly in people already predisposed to mood disorders. This is a single study with a modest number of participants, and it would be a mistake to treat it as the whole story. But it fits neatly with the clinical observation that serotonin-acting antidepressants help, and with the idea that light affects serotonin systems as well as the clock. It also illustrates a theme that runs through SAD research: the problem appears to lie less in the winter environment itself, which everyone at a given latitude shares, than in how well a particular person's brain adapts to it. Behavior, thought and the winter spiral Biology is only part of the explanation. The fact that most people in a northern city do not develop SAD, even though they experience the same short days, suggests that psychological factors shape who becomes ill and how severely. Kelly Rohan, whose work is discussed at length in Chapter 6, has argued for an integrative model in which biological vulnerability to reduced light interacts with psychological vulnerability, particularly with habits of thinking and behaving that amplify low mood. The behavioral part of the story is easy to see on campus. Short, cold days make it harder to get outside, exercise and see friends. Low energy and hypersomnia reduce activity further. With fewer rewarding experiences, mood drops, which reduces motivation, which reduces activity again. This is the familiar downward spiral of depression, and winter provides an unusually strong initial push. A student whose social life depended on outdoor activities, spontaneous meetups on the quad or walking across campus between classes may find that most of those contacts simply disappear in December without any conscious decision to withdraw. The cognitive part is subtler. Rohan's research has found that people vulnerable to SAD often have a distinctive style of thinking about the seasons: negative beliefs about winter and about their ability to cope with it, and a tendency to ruminate on how bad they feel and how bad the darkness is. In a study published in 2013, members of her team found that participants with higher levels of rigid, dysfunctional attitudes and frequent negative automatic thoughts at the start of treatment had worse outcomes the following winter if they were treated with light therapy alone than if they received cognitive-behavioral therapy. In other words, for some people, the thoughts that surround winter are part of the mechanism, not just a consequence of it. For students, winter-related thinking often mixes with academic and social worries. "I always fall apart in winter" becomes "I'm going to fail this semester." "It's dark at four o'clock" becomes "There's no point going out." "Everyone else seems fine" becomes "Something is wrong with me." These thoughts are understandable, and they are sometimes accurate reflections of past experience. But they also shape behavior. A student convinced that winter is unbearable is less likely to try the things that make it bearable. Much remains uncertain. Researchers still do not know exactly why some people's clocks and serotonin systems adapt poorly to winter while others' adapt well, how much of the vulnerability is inherited, or why a minority of people have the opposite pattern and become depressed in summer. Twin and family studies suggest that seasonality runs in families, but no single gene explains it. The phase-shift model fits many patients well and others poorly; some appear phase-advanced rather than delayed, and some show little misalignment at all. None of this undermines the treatments, which have been tested directly rather than inferred from theory. It is simply a reason for humility, and for paying attention to what actually works for an individual student rather than assuming that one mechanism explains every case. The various mechanisms are best thought of as layers rather than competing theories. Short days and weak winter light reduce the signal that anchors the body clock. In people whose clocks tend to run late, which includes many young adults, the clock drifts out of alignment with their sleep and waking schedule. In people whose serotonin systems adapt poorly to the season, winter brings a drop in the brain's effective serotonin signaling. The resulting fatigue, oversleeping and low mood reduce activity and social contact, which remove sources of reward and push mood lower. Negative thoughts about winter and about oneself deepen the slide and make it harder to break. This layered view has a hopeful implication. Because several processes are involved, there are several places to intervene, and interventions at different layers can reinforce one another. Morning bright light acts on the clock and probably on serotonin systems. Consistent wake times reinforce the clock. Scheduled activity and social contact rebuild reward. Cognitive work weakens the thinking patterns that keep people stuck. Antidepressants act on serotonin directly. The most robust approach for many students is not to find the one right remedy but to build a winter routine that addresses more than one layer at once. The view also explains a practical puzzle that students often raise: why a sunny spring break trip can produce dramatic, if temporary, improvement, and why a gray week in March can bring symptoms back. Mood in SAD responds to light exposure over days and weeks, not over a season as a whole. That responsiveness, which can feel frustrating, is exactly what makes light-based treatment work, and it is the subject of Chapter 4. First, though, it is worth looking closely at how the academic calendar interacts with these mechanisms, because the weeks of greatest risk for students are not always the ones they expect. Chapter 3: The Academic Year Meets the Solar Year Most writing about seasonal depression assumes a person whose life has roughly the same shape in every month: a job, a home, a stable set of relationships. Students do not live that way. Their year is divided into terms, breaks, exam periods and move-in days, and those divisions do not line up neatly with the seasons. For a student vulnerable to winter depression, the interaction between the academic calendar and the solar calendar produces a series of predictable pressure points. Knowing where they fall is the first step to planning for them. The pattern described here is for a typical semester system at a university in the northern United States or Canada, with a fall term from late August to mid-December, a winter break of three to five weeks, and a spring term from mid-January to early May. Quarter systems, trimester systems and universities in the United Kingdom, Europe or the southern hemisphere shift the dates but not the logic. The question to ask of any calendar is the same: where do the heaviest academic demands coincide with the least light, the weakest routine and the thinnest social support? The long slide of the fall term The fall semester begins at the best possible time. Days are still long, the weather is mild, and everyone is making new friends and joining clubs. Many students with a seasonal pattern feel their best in September. That is part of the trap. The routines built in early fall, such as late nights, spontaneous plans and outdoor socializing, often depend on conditions that will not last. The autumn equinox, around September 22, is the point after which nights are longer than days in the northern hemisphere. Day length then shortens fastest in the weeks around the equinox, losing several minutes each day at northern latitudes. Students often notice nothing for a few weeks. Then, typically in mid to late October, the first signs appear: harder mornings, more napping, a stronger pull toward food and bed. This coincides with midterm examinations, the first period of serious academic pressure, and with the end of the social honeymoon of the early semester. A student who attributes the change to midterm stress will expect it to lift once midterms end. It does not. The first Sunday of November brings a sharp, man-made shift in the United States and Canada, when daylight saving time ends and clocks fall back an hour. The change gives brighter mornings for a while, which can help, but it moves sunset abruptly earlier. On many northern campuses, the sun now sets before five o'clock, and before half past four by December. Students leaving an afternoon lab or a library session now walk out into full darkness. Many describe this weekend as the moment winter truly arrives, and the change in the evening light makes it psychologically harder to go out again after dinner. November and early December are the most dangerous stretch of the fall term. Light is declining toward its minimum at the winter solstice, around December 21. Coursework is piling up toward final projects and examinations. Thanksgiving in the United States provides a brief break that can help or hurt, depending on how it is spent. Students who are already sliding often fall behind in these weeks, and falling behind generates anxiety and guilt that deepen the low mood. The typical coping responses of stressed students, which include sleeping less during the week, sleeping more on weekends, living on caffeine and sugar, cancelling social plans and staying indoors to work, are almost exactly the behaviors that worsen seasonal depression. Consider a composite junior majoring in engineering at a university in upstate New York. Every December for three years, his grades have dropped by a letter compared with his autumn midterms. He has always blamed final exams, which are heavier in his major than in most. When he finally keeps a simple log of sleep, mood and time outdoors, he discovers that his decline starts weeks before finals, in the first week of November, and that his worst days are the ones on which he does not leave his building before dark. The exams are real, but they are landing on a student already running at half strength. Winter break and the January return The end of the fall term brings relief, but the winter break that follows is a mixed blessing. On the one hand, academic pressure lifts and many students go home to family, familiar surroundings and, for some, a lower latitude with more daylight. On the other hand, winter break often dissolves structure. There are no classes to get up for, no scheduled meals, no built-in social contact. Sleep schedules drift later, sometimes dramatically, with students going to bed at three or four in the morning and waking in the early afternoon, when little daylight remains. For a student vulnerable to SAD, a break spent in a dark bedroom in a northern hometown can be one of the worst periods of the year. Break can also be socially difficult. Some students return to strained family situations, to hometowns where their friends have scattered, or to the loneliness of staying on a nearly empty campus because travel is too expensive or home is not a safe place. International students who cannot go home for a few weeks are particularly exposed: residence halls close or thin out, dining halls shorten their hours, and the social network of the semester disappears exactly at the darkest time of year. The return to campus in January is, for many students with SAD, the lowest point of the academic year. The days are only just beginning to lengthen after the solstice, and at northern latitudes the increase is small at first. The body clock may have drifted many hours later over break, so the first week of early classes feels like jet lag. The social momentum of the fall has been lost. Weather is often at its coldest. The spring semester typically starts with a rush of new courses, syllabi and deadlines, and students who ended the fall behind may be carrying incomplete grades or academic probation into the new term. This jet-lag effect is largely preventable, and the prevention is simple enough to describe in a paragraph. The body clock moves earlier only gradually, by something in the range of half an hour to an hour a day under favorable conditions, and morning light is what moves it. A student whose clock has drifted to a four o'clock bedtime over break cannot expect to wake comfortably at seven on the first day of term. The practical answer is to begin moving the wake time earlier during the last week or so of break, by perhaps half an hour every day or two, and to get bright light, outdoors or from a light box, soon after each earlier waking. Bedtime tends to follow wake time once the clock begins to shift. It helps to keep evenings dimmer as the week goes on. A student who arrives back on campus with the clock already reset avoids the worst of the January crash and starts the semester attending classes rather than recovering from them. Consider a composite graduate student from Singapore beginning a doctoral program in Montreal. Her first autumn is full of novelty and she barely notices the shortening days. Over the December break, most of her lab colleagues leave the city, the building empties, and she spends most days in her apartment writing code, going to bed later and later. By the second week of January she is sleeping until noon, has not spoken to anyone in person for several days, and feels a flatness she has never experienced. Nothing in her upbringing has prepared her to read this as a seasonal problem; in Singapore, day length barely changes across the year. What would have helped her is not unusual: a warning from her department in November, a plan to keep a regular morning schedule and see people during the break, and knowledge that the counseling center stays open between terms. Each of those is a matter of foresight rather than willpower. January and February are also, as a rule, the months in which winter depression reaches its greatest severity, and the months in which many clinical trials of SAD treatments have been conducted and assessed, precisely because symptoms are reliably present then. The long follow-up study of cognitive-behavioral therapy and light therapy discussed in Chapter 6, for instance, scheduled its follow-up visits in January or February of each year. Spring, and the students most at risk The first real relief usually comes in March. Day length is now increasing at its fastest rate, and by the spring equinox, around March 20, days and nights are equal again. The change in daylight saving time in early March moves light into the evening, which many students find uplifting. Spring break can provide a burst of improvement, especially for students who travel somewhere sunny. By April, most people with winter SAD are feeling significantly better, and by May, as the semester ends, many are close to their best. The spring has hazards of its own. As mentioned in Chapter 1, some people with seasonal mood disorders, particularly those with bipolar disorder, can swing past normal into hypomania in spring. Students may also face the consequences of winter in April: the unfinished assignments, the damaged grades and the neglected friendships, which can generate a second wave of stress just as mood improves. It is worth knowing, too, that the spring months are associated in many countries with a seasonal rise in suicide rates, a pattern that has been observed for more than a century and is not fully understood. Feeling better is welcome. It is not a reason to stop paying attention, and students who are recovering from a hard winter should not drop their supports all at once. Several groups of students deserve particular mention because the calendar hits them harder. International students who come from tropical or subtropical countries to northern universities often face their first real winter with no previous experience of short days and no cultural knowledge about coping with them. Students from warmer US states attending northern universities are in a similar position. First-year students lack the established routines and friendships that buffer later winters. Graduate students, especially those in laboratory sciences, often work long hours in windowless rooms with little built-in social contact, and lack the calendar structure that undergraduates have. Student athletes may experience winter as a season of either intense scheduled training, which can be protective, or of off-season drift, which can be the opposite. Students living alone, particularly those in off-campus apartments, lose the incidental contact that residence halls provide. Table 2 lays out the main pressure points of a typical northern semester calendar alongside the approaches that tend to help at each stage. The purpose is not to prescribe a timetable but to make the pattern visible. Table 2. Pressure points in a northern academic year and matching countermeasures. Period What changes Why it is risky Countermeasures Late September to October Rapid loss of daylight; midterms Early symptoms mistaken for stress Start mood log; fix wake time; plan light therapy November Clocks fall back; sunset before five Evening darkness cuts social life Begin or increase light therapy; schedule standing social plans Early to mid December Finals near the solstice Workload peaks as light bottoms out Protect sleep; daylight breaks; use counseling early Winter break Structure disappears Sleep drifts late; isolation Keep wake time; daily outdoor light; plan contact January to February Return at the darkest point Jet-lag effect; lost momentum Reset clock before term; restart routines in week one March to April Rapid return of light Relief masks fallout; possible highs Taper supports gradually; watch for elevated mood Using the calendar as a planning tool The value of seeing the academic year this way is that it turns a vague dread of winter into a set of specific, manageable transitions. Each transition has an obvious question attached to it. What will I do when the clocks change? What is my plan for the week before finals? How will I keep my wake time during break? What will I do in the first week of January to get my clock back? Students who answer those questions in September are in a much stronger position than students who try to answer them in the middle of December while already depressed. There is also a useful lesson about the diagnostic question raised in Chapter 1: is the pattern the darkness, or the stress? Mapping the calendar shows why the answer is so often both, and why separating them is less important than it first appears. Academic stress and short days peak at the same time, in December, and they compound each other. Light therapy helps with the part driven by the body clock and brain chemistry; better planning, sleep protection and support help with the part driven by workload. Both parts respond to the same basic strategy of anticipating the hard weeks rather than reacting to them. One particular planning decision deserves early attention: the shape of the course schedule itself. Students who know they struggle in winter can sometimes arrange their spring schedule to reduce risk. A schedule with a consistent daily start time, rather than a mix of eight o'clock classes some days and none on others, helps anchor the body clock. A schedule that includes at least one course or commitment requiring regular attendance in the morning can provide a reason to get up at a steady hour. Some students find it helpful to take a slightly lighter load in the spring semester, or to front-load demanding courses in the fall, when they are at their strongest. These choices are not always possible, especially in structured programs, but they are worth discussing with an academic adviser. A student need not disclose a diagnosis to ask for scheduling advice, though advisers can often help more when they understand the reason. Parents can use the calendar as well. The student who sounded cheerful on the phone in September may sound very different in late November or January, and parents who know the pattern can check in at the times that matter most, without hovering. A short, regular call in the first week of the spring semester can be worth more than a long visit in October. Parents can also help by supporting structure over winter break rather than encouraging a student to sleep off the fall semester, and by being alert to the possibility that a student who seems withdrawn and exhausted at home is not simply recovering from exams. Residence life staff, academic advisers and faculty can do something similar at an institutional level. Resident advisers who know that November and January are hard months can plan programming, such as group outings, communal meals, or brightly lit evening study sessions, for those periods rather than front-loading everything in September. Faculty who notice a student fading in the weeks after the clocks change can mention the counseling center, and can reasonably ask whether something beyond ordinary busyness is going on. Some campuses have begun to offer loanable light boxes, light therapy stations in libraries or counseling centers, and winter wellness programs timed to these pressure points. Where such resources exist, the calendar tells students when to use them. The next three chapters turn to the specific tools that address the mechanisms described in Chapter 2. The first, and the one with the quickest effect, is bright light. Chapter 4: Bright Light Therapy Of all the treatments for seasonal depression, bright light therapy is the one most closely tied to the condition's history. The 1984 paper that first described SAD also reported preliminary findings that extending the day with bright artificial light had an antidepressant effect. In the four decades since, light therapy has been tested in dozens of controlled trials, compared directly with antidepressant medication and with psychotherapy, and refined in its timing and dose. It is not a cure, and it does not work for everyone. But for many students with winter depression it is the fastest-acting option available, it can be used without a prescription, and its side effects are usually mild. Used well, it can turn a lost winter into a manageable one. Used badly, it does very little. The most common reasons light therapy fails are not that the treatment is ineffective but that the light is too dim, the session too short, the timing wrong, or the routine abandoned after a week. This chapter is therefore as much about practice as about evidence. What the evidence shows The first rigorous synthesis of the evidence was a meta-analysis led by Robert Golden and published in the American Journal of Psychiatry in 2005. The authors searched nearly three decades of research and found that only a small fraction of studies met their standards for rigor, a reminder of how much early work in the field was poorly controlled. But the trials that did meet the standard showed a clear effect. Bright light treatment for SAD reduced depression severity with an effect size of 0.84, across eight studies, and dawn simulation, in which a bedside light gradually brightens during the last part of sleep, also showed a significant effect, of 0.73, across five studies. The authors noted that these effect sizes were comparable to those seen in most trials of antidepressant medication. A later meta-analysis, by Edda Pjrek and colleagues in Vienna, published in 2020, was more conservative. Pooling 19 randomized trials that compared bright light with a dim light or sham device, it found that bright light was clearly superior to placebo, but with a more modest standardized effect of 0.37, and that people receiving bright light were about 1.4 times as likely to respond. The authors also noted moderate differences between studies and a moderate-to-high risk of bias in many of them. Their conclusion was that bright light therapy can be regarded as effective for SAD, but that larger, higher-quality trials are still needed. The difference between those two estimates is worth understanding rather than smoothing over. Light therapy is hard to test. It is impossible to blind people completely to whether they are sitting in front of a very bright light or a dim one, and people who expect bright light to help may report more improvement. Researchers use dim red lights, deactivated negative ion generators and other devices as placebos, and these vary in how convincing they are. Different trials also use different doses, timings and durations. The true effect is probably somewhere between the two estimates, and it varies from person to person. What is not in doubt is that it is a real effect and a clinically useful one. The most useful single trial for students weighing their options is the Canadian study known as Can-SAD, led by Raymond Lam and published in 2006. It compared light therapy directly with the antidepressant fluoxetine, better known by the brand name Prozac, using a double-blind design in which every participant received both a light box and a capsule. One group received 10,000 lux of light for 30 minutes each morning along with a placebo capsule; the other received a dim, 100 lux light along with 20 milligrams of fluoxetine daily. Over eight weeks, 96 patients with winter SAD improved substantially in both groups, with identical response rates of 67 percent and similar remission rates of 50 percent for light and 54 percent for fluoxetine. The light group improved faster, showing a greater benefit at the one-week mark, and fluoxetine was associated with more side effects such as agitation, sleep disturbance and palpitations. The authors concluded that both treatments were effective and well tolerated, and that patient preference and other clinical factors should guide the choice. Two further points about the evidence are important for students. First, light therapy tends to work quickly, often within one to two weeks, which is faster than most antidepressants and much faster than psychotherapy. That makes it valuable for a student in the middle of a term who cannot afford to wait six weeks to feel better. Second, its effect usually lasts only as long as it is used. Symptoms commonly return within days to a couple of weeks if light therapy is stopped in the middle of winter. It treats the season; it does not change the underlying vulnerability. That limitation becomes central in Chapter 6. Getting the dose and timing right The standard regimen established in the research is 10,000 lux of white light for about 30 minutes a day, used soon after waking. Those numbers need some unpacking. Lux is a measure of light intensity at a particular distance. A light box rated at 10,000 lux delivers that intensity only at the distance specified by the manufacturer, which is often somewhere between 30 and 60 centimeters, or roughly one to two feet. Intensity falls off quickly with distance, so a box used at arm's length on the far side of a desk may deliver only a fraction of its rated output. Lower intensities can work but need longer sessions; clinical references commonly cite 2,500 lux for two hours or 5,000 lux for one hour as roughly equivalent to 10,000 lux for 30 minutes. For a student with a busy morning, the shorter session at higher intensity is usually more practical. Morning timing matters because of the clock mechanism described in Chapter 2. For most people with winter SAD, whose clocks are delayed, morning light advances the clock back into alignment. Evening light would delay it further. A study led by Jiuan Su Terman and Michael Terman at Columbia University, published in 2001, demonstrated this in detail. Patients with winter depression received 10,000 lux of light for 30 minutes in the morning or the evening, and the researchers measured the timing of each patient's melatonin rhythm before and after treatment. Morning light advanced the clock and evening light delayed it, and the earlier the morning light fell in each patient's own circadian day, the larger the advance. Among patients treated in the morning, those whose clocks shifted most improved most. In a larger combined sample, early morning light, in circadian terms, outperformed both late morning and evening light. The authors estimated the optimal time as about eight and a half hours after the evening rise in melatonin, or roughly two and a half hours after the midpoint of a person's sleep. Light at the same clock time could therefore be well timed for one patient and poorly timed for another, depending on where their internal clocks were set. For students, this has two practical implications. The first is that the best time is tied to a person's own clock, not to a universal hour. A student who sleeps from one in the morning until nine has a sleep midpoint of five, which by the Terman estimate puts the ideal session around seven-thirty, somewhat before that student's natural wake time. A student who sleeps from eleven to seven would aim earlier still. For many students, this means using the light box soon after waking and, over a week or two, nudging the wake time and the session a little earlier, rather than suddenly setting an alarm for five-thirty, which simply causes sleep deprivation. The second implication is that consistency matters as much as precision. A session at roughly the same early time every day, weekends included, does far more than a perfectly timed session three days a week. Some clinicians use short questionnaires about sleep preferences, such as morningness-eveningness questionnaires, to estimate timing more precisely. If light causes very early waking, the session can be moved slightly later; if it seems to have little effect, a slightly earlier or longer session may help. Such adjustments are best made with a clinician's guidance. Dawn simulation is an alternative or addition that some students find easier. A dawn simulator is a bedside lamp that gradually brightens over the last half hour or so of sleep, mimicking a spring sunrise. It uses much lower intensities than a light box, typically a few hundred lux at most, but acts during a sensitive window of the clock's cycle. The evidence base is smaller than for light boxes, but the Golden meta-analysis found a meaningful effect, and dawn simulators have a practical advantage for students who struggle to get up at all: they make waking easier, and waking is the first step in every other part of a winter routine. Table 3 summarizes the core practical parameters for light box use. It is a summary of common clinical practice, not a prescription, and individual plans should be discussed with a clinician. Table 3. Core parameters for bright light therapy in winter depression. Parameter Common practice Notes for students Intensity 10,000 lux at eye level Only at the distance the maker specifies Duration About 30 minutes daily Lower intensity needs longer sessions Timing Soon after waking Keep wake time steady, weekends included Position Slightly above, off to one side Eyes open; do not stare into it Season Autumn or early winter until spring Stopping early often brings symptoms back Response Often within one to two weeks Review with a clinician if none by week three or four Choosing and using a light box in student life Light boxes are widely sold online and in stores, and quality varies considerably. In the United States, light boxes are not approved by the Food and Drug Administration as treatments for SAD, so marketing claims have not been independently verified. A few features are worth looking for. The box should deliver 10,000 lux at a stated, reasonable distance, and the manufacturer should say what that distance is. The light should be white, filtered to block ultraviolet rays, which are not needed for the therapeutic effect and can harm the skin and eyes. The illuminated surface should be reasonably large, because small, bright devices must be held very close to reach the rated intensity and are more likely to cause glare. Devices marketed with vague claims, such as unspecified "full spectrum" or "natural daylight" bulbs without lux ratings at a given distance, are poor substitutes. Before buying, students should check whether their campus counseling or health center lends light boxes or has light therapy stations; a number of campuses do. The box should be placed so that the light falls on the eyes from slightly above and to the side, as sunlight does, rather than straight ahead. The eyes need to be open, because the effect works through the retina, but there is no need to look directly into the light, which can be uncomfortable. Most people read, eat breakfast, check messages or review notes during their sessions, glancing up occasionally. That is one of the practical virtues of light therapy for students: half an hour at a desk in the morning can double as time for email or a first look at the day's readings. The obstacles in student life are usually logistical. Shared rooms can be a problem if a roommate is sleeping when the light is needed; placing the box on a desk angled away from the other bed, or using it in a common area, usually solves this. Irregular class schedules make consistent wake times harder, which is another reason to anchor the day with a fixed wake time independent of the first class. Travel home over break means either bringing the light box or planning outdoor morning light as a substitute. And the biggest obstacle is simply keeping it up. Light therapy requires daily use for months, and students often stop once they feel better, only to relapse two weeks later. Treating the morning session as a fixed habit, rather than a remedy used when needed, is what makes it work. Outdoor light is a valuable complement and, for milder symptoms, sometimes a partial substitute. Even on a gray winter day, outdoor light is often brighter than a typical room by an order of magnitude or more. A morning walk to class, a twenty-minute walk outdoors in the late morning or around midday, or studying next to a large window in the morning all increase light exposure. For students with full SAD, outdoor exposure alone may not be enough, because northern winter mornings can be very dark and the most useful early-morning window may fall before sunrise. But it adds to the effect of a light box and has other benefits, including exercise and, often, social contact. Safety, side effects and who should be cautious Light therapy is generally safe, which is one of its main advantages over medication for students who want to avoid drugs. The common side effects are mild and usually temporary: headache, eye strain, a feeling of being wired or jittery, nausea and, when light is used too late in the day, difficulty falling asleep. Most of these can be managed by sitting slightly farther from the box, shortening the session, or moving it earlier. If side effects persist, a clinician can help adjust the plan. Some groups need medical input before starting, and students should take these cautions seriously. The most important concerns bipolar disorder. As noted in Chapter 1, a meaningful proportion of people with seasonal mood patterns have bipolar disorder, and bright light, like antidepressant medication, can trigger hypomania or mania in susceptible people. Signs to watch for include a marked decrease in the need for sleep, racing thoughts, unusual irritability, grandiosity and impulsive spending or risk-taking. Anyone with known or suspected bipolar disorder should use light therapy only under the supervision of a clinician, often in combination with mood-stabilizing medication, and sometimes with shorter sessions or lower intensity to begin with. A student who notices these signs after starting light therapy should stop and contact a clinician promptly. People with eye conditions also need care. Clinical references recommend an eye examination before starting light therapy for people with pre-existing eye disease, particularly retinal conditions, and for people at higher risk of eye problems, such as those with diabetes. The same applies to people taking medications that increase sensitivity to light. Several drugs have this effect, including certain antibiotics, some acne treatments and the herbal remedy St. John's wort. A pharmacist or physician can check a student's medications quickly. A history of migraine that is triggered by bright light is another reason to start cautiously. Finally, light therapy is a treatment for depression, not a replacement for assessment. A student whose symptoms are severe, who has thoughts of suicide, or who does not improve after a few weeks of consistent use needs clinical care. Light therapy can be combined with psychotherapy and with medication, and many people with SAD use it that way. Starting a light box on one's own is reasonable for a student with a clear, mild-to-moderate seasonal pattern and none of the cautions above. For anything more, it is better used as part of a plan made with a professional. One last practical note concerns timing across the year. Because light therapy works quickly but wears off quickly, the question of when to start matters. Many clinicians suggest that people with a well-established winter pattern begin light therapy in early autumn, when the first symptoms appear or even slightly before, rather than waiting until they are fully depressed, and continue until spring light is strong enough to take over. The direct evidence for this preventive use is limited: a Cochrane review published in 2019 found only one small eligible trial of light therapy for preventing SAD in people who were still well, too small to allow firm conclusions. Starting at the first signs, rather than months later, is supported by clinical experience and by the simple logic that it is easier to stay well than to climb out of an episode. It is a sensible plan for most students, provided it is discussed with a clinician and paired with attention to the thinking and behavior patterns covered in the chapters that follow. Chapter 5: Vitamin D and the Limits of a Simple Story Vitamin D has a story that almost tells itself. The skin makes vitamin D when exposed to sunlight. In winter there is less sunlight, especially at northern latitudes, and vitamin D levels fall. Winter is also when seasonal depression strikes. Vitamin D receptors are found in the brain, including in regions involved in mood. And surveys have repeatedly found that people with depression tend to have lower vitamin D levels than people without it. It seems to follow that SAD might be, at least in part, a vitamin D deficiency, and that a daily supplement might prevent or treat it. That story has made vitamin D one of the most popular remedies for winter mood. It is inexpensive, available without a prescription, and widely recommended in wellness articles and by friends. Many students take it. The difficulty is that when the story has been tested directly in randomized trials, the results have been far less impressive than the logic suggested. This chapter walks through what is known, what is not, and what a sensible student should do. Why the idea is plausible The first part of the story is well established. When ultraviolet B radiation from the sun strikes the skin, it converts a precursor molecule into vitamin D, which the liver and kidneys then convert into its active forms. In winter at higher latitudes, the sun is so low in the sky that most ultraviolet B is filtered out by the atmosphere before it reaches the ground. A classic study by Ann Webb, Lorrie Kline and Michael Holick, published in 1988, measured vitamin D production in skin samples exposed to sunlight at different times of year. In Boston, at about 42 degrees north, there was essentially no production from November through February. In Edmonton, at about 52 degrees north, the period without production extended from October through March. At these latitudes, the body relies in winter on stores built up over the summer and on dietary sources, which are limited: oily fish, egg yolks, and fortified foods such as milk and some cereals. It is also well established that many people have lower vitamin D levels in late winter than in late summer, and that low levels are more common in people with darker skin, which produces vitamin D less efficiently from a given amount of sunlight; in people who spend little time outdoors; in people who cover most of their skin; in people with obesity; and in people with certain digestive conditions that impair absorption. Students with several of these characteristics living at a northern university may well have low levels by February. The link with depression is supported by many observational studies, which have found that people with lower blood levels of vitamin D are somewhat more likely to be depressed or to become depressed. But observational links are notoriously difficult to interpret. People who are depressed tend to spend less time outdoors, exercise less and eat differently, all of which would lower their vitamin D levels. Vitamin D levels also reflect general health, weight and lifestyle. The association could mean that low vitamin D contributes to depression, that depression leads to low vitamin D, or that both are driven by something else. The only way to find out whether raising vitamin D levels improves mood is to do it, in randomized controlled trials. What the trials found The largest and most rigorous trial of vitamin D and depression is VITAL-DEP, a part of the broader Vitamin D and Omega-3 Trial in the United States. Its results, led by Olivia Okereke and published in JAMA in 2020, involved 18,353 adults aged 50 or older, randomly assigned to take either 2,000 international units of vitamin D3 a day or a placebo, for a median of 5.3 years. The researchers looked at whether the supplement reduced the risk of developing depression, or of depression returning in people who had experienced it before, and at changes in mood scores over time. It did neither. The rate of depression or clinically significant depressive symptoms was almost identical in the two groups, and mood scores did not differ. The authors concluded that the results did not support using vitamin D3 in adults to prevent depression. VITAL-DEP has limitations as a guide for students. Its participants were older adults, most of whom were not deficient in vitamin D at the start, and it tested prevention of depression in general rather than treatment of seasonal depression. It is possible that vitamin D matters more for people who are genuinely deficient, or for younger people, or for people who already have depressive symptoms. But it is a large, long, well-conducted trial, and it sets an important boundary on the enthusiasm: for adults who are not deficient, taking vitamin D to prevent depression does not appear to work. Trials focused specifically on seasonal depression are far smaller. One example is a Danish trial published in 2014 by Tenna Bloch Frandsen and colleagues, which randomly assigned healthcare workers with seasonal symptoms to 70 micrograms, or 2,800 international units, of vitamin D a day or placebo for three months over the winter. Only 34 participants completed the study. There was no difference between the groups in seasonal depression scores or wellbeing, although both groups improved as the season progressed, a reminder that winter symptoms tend to lift with time regardless of treatment, which makes uncontrolled testimonials unreliable. The authors noted that their study was underpowered and could not assess whether vitamin D helped people with low levels. Other small trials of vitamin D for SAD have reported mixed findings, and no large, definitive trial has settled the question. Meta-analyses, which pool results from many trials, add further nuance. A 2022 review by Tuomas Mikola and colleagues combined 41 randomized trials involving more than 53,000 participants and found an overall positive effect of vitamin D on depressive symptoms, with the authors suggesting that doses of at least 2,000 international units a day might reduce symptoms. But the same review rated the certainty of the evidence as very low, found very high variation between studies, and noted concerning risk of bias in most of them. It excluded studies involving light therapy and people with bipolar disorder, and it did not focus on seasonal depression. Its conclusion called for further research into whether vitamin D might be useful as an addition to standard treatments in clinical depression. That is a long way from demonstrating that vitamin D treats SAD. Table 4 summarizes these key studies. It shows why careful reviewers describe the evidence for vitamin D in depression as mixed, and why the evidence for SAD in particular remains thin. Table 4. Selected studies of vitamin D supplementation and mood. Study Design and participants Daily dose Main finding Frandsen et al., 2014 Small trial; 34 adults with seasonal symptoms 2,800 IU for 3 months No difference from placebo Okereke et al., 2020 (VITAL-DEP) Large trial; 18,353 adults aged 50+ 2,000 IU, median 5.3 years No reduction in depression risk Mikola et al., 2022 Meta-analysis of 41 trials Various Small benefit; very low certainty Sources: Frandsen et al., BMC Research Notes, 2014; Okereke et al., JAMA, 2020; Mikola et al., Critical Reviews in Food Science and Nutrition, 2022. Melatonin, other supplements and winter eating Vitamin D is not the only product that students encounter in the search for a winter remedy, and a few others deserve brief comment. Melatonin is sold over the counter in the United States as a sleep aid, often in doses of several milligrams, and it is easy to assume that because SAD involves the melatonin system, taking melatonin must help. The reality is more complicated. In the research of Alfred Lewy and colleagues described in Chapter 2, melatonin was used in very low doses, carefully timed in the afternoon or evening to shift the body clock of patients whose clocks were delayed. That is a precise chronobiological intervention, not a sleeping pill, and taken at the wrong time melatonin can shift the clock in the wrong direction. A Cochrane review published in 2019 looked for trials of melatonin to prevent SAD and found none that met its criteria; it found only one trial of a related prescription drug, agomelatine, which gave an uncertain result. Students who want to use melatonin to help reset a delayed clock should discuss dose and timing with a clinician rather than taking a large dose at bedtime. Melatonin products are sold as dietary supplements in the United States, and independent testing has found that the actual content of some products differs substantially from the label. St. John's wort, an herbal remedy used for mild depression, raises two separate concerns. It interacts with a wide range of prescription medications, including hormonal contraceptives, whose effectiveness it can reduce, and many antidepressants. It also increases the skin's and eyes' sensitivity to light, which is a particular problem for anyone combining it with light therapy, as noted in Chapter 4. Students should not take it without telling a clinician and pharmacist. Omega-3 fatty acid supplements have been studied in depression with mixed results; in the same VITAL trial that tested vitamin D, fish oil was also tested and did not show a clear benefit for preventing depression. They are not an established treatment for SAD. Food itself deserves a word, because the carbohydrate craving of winter depression is real and can be distressing. Students often respond to winter weight gain with guilt and restrictive dieting, which tends to worsen mood and energy. A more useful approach is to regularize eating rather than restrict it: three meals at roughly consistent times, including breakfast soon after waking, which also helps anchor the body clock. Including protein and vegetables at meals, and keeping some easy, satisfying options on hand for the late afternoon slump, reduces the pull toward grazing on sweets in the evening. For students with a history of eating disorders, winter appetite changes can be a trigger, and it is worth raising the issue with a clinician early rather than trying to manage it alone. What guidelines say, and a sensible approach Professional guidance has moved toward caution. In 2024, the Endocrine Society, a leading medical organization for hormone-related conditions, released a clinical practice guideline on vitamin D for disease prevention. For generally healthy adults between the ages of 19 and 74, it recommended against taking vitamin D above the recommended daily intake, and against routine testing of blood vitamin D levels, in the absence of a specific medical reason. The guideline made separate recommendations for children, older adults, pregnant people and people with prediabetes, and it did not address the treatment of people with established deficiency, which remains a medical matter. In the United States, the recommended dietary allowance for adults aged 19 to 70, set by the National Academies and reported by the National Institutes of Health Office of Dietary Supplements, is 600 international units, or 15 micrograms, a day. The tolerable upper intake level for adults, the highest daily amount considered unlikely to cause harm, is 4,000 international units, or 100 micrograms. Vitamin D is fat-soluble and accumulates in the body, so very high doses over long periods can cause toxicity, including dangerously high calcium levels. Supplements sold online in doses of 10,000 international units or more should not be taken without medical supervision. For a student concerned about winter depression, several practical conclusions follow. First, vitamin D is not a substitute for the treatments with strong evidence behind them. A student with significant winter depression who takes a vitamin D supplement and waits to feel better is losing time that could be spent on light therapy, psychological treatment or medication. This is the most important point in the chapter, and the most common mistake. Consider a composite first-year student from Texas at a university in Wisconsin. By early December she feels exhausted and flat, and a friend suggests vitamin D. She orders a large bottle of 5,000-unit capsules, takes one every day, and decides to give it a month before doing anything else. The month runs straight into finals and winter break. In January she is worse, not better, and has spent six weeks on the one remedy in her plan with the weakest evidence behind it. Nothing about taking vitamin D was harmful at that dose for a few weeks. The harm was in the waiting. Had she treated vitamin D as a sensible background measure and simultaneously started a morning light routine or booked a counseling appointment, the month would have looked very different. Second, a student who has reason to believe they may be deficient should talk to a clinician rather than self-treat with large doses. Risk factors include darker skin, spending very little time outdoors, a diet low in fish and fortified foods, including many vegan diets, obesity, and conditions or surgeries affecting absorption. Symptoms of significant deficiency can include bone pain and muscle weakness, and a clinician may reasonably check levels in someone with those risk factors or symptoms even though routine screening of healthy adults is no longer recommended. Correcting a real deficiency is worthwhile for bone and general health, whatever it does for mood. Third, for a healthy student who wants to take a standard daily supplement over the winter, doing so at or near the recommended daily intake is generally safe. It is a reasonable choice for someone whose diet and lifestyle provide little vitamin D in winter. It should simply be understood as a measure for general health rather than as a treatment for depression, and it should not delay other steps. Fourth, a student deciding between spending money on vitamin D and spending it on a good light box should choose the light box. The evidence for correctly used bright light in SAD is far stronger. There is a broader lesson in the vitamin D story, and it applies well beyond this one nutrient. Many things are associated with winter depression: less sunlight, less exercise, more sleep, changes in diet, less time with friends. Associations suggest ideas worth testing. They do not prove that fixing any single factor will fix the depression. The treatments that have held up in rigorous trials, bright light, cognitive-behavioral therapy tailored for SAD and certain antidepressants, earned their place by being tested head to head, sometimes against each other, with outcomes measured over whole winters. Students surrounded by confident wellness advice about supplements, special diets and products do well to ask a simple question of each: has it been tested against a placebo in people like me, and what happened? That question also explains why the remaining chapters of this book lean so heavily on the psychological side of treatment. The single most important finding in recent SAD research concerns not a chemical but a set of skills. Chapter 6: Learning to Outlast Winter with CBT Light therapy works by acting on the body. It is effective, fast and relatively easy to use, but it has an important limitation: it helps only while it is being used, and it does nothing directly about the ways of thinking and behaving that can make winters worse. A student who relies on light alone may need to repeat the same routine every winter for the rest of their life, and may find that when life gets disrupted, by travel, a crisis, or simply forgetting, the protection disappears. For years, psychologists wondered whether a treatment that changed thinking and behavior could provide something more lasting. The most systematic answer has come from the work of Kelly Rohan, a clinical psychologist at the University of Vermont, who developed a form of cognitive-behavioral therapy tailored specifically for seasonal affective disorder, usually abbreviated CBT-SAD, and tested it against light therapy in a series of trials. The results are among the most practically important findings in the field. The head-to-head trials Cognitive-behavioral therapy, or CBT, is a structured, time-limited form of psychotherapy with strong evidence for treating depression. It rests on the observation that thoughts, feelings and behaviors influence one another, and that changing how a person thinks and what they do can change how they feel. Rohan's version adapts CBT to the specific features of winter depression, focusing on the thoughts people have about winter, darkness and their own ability to cope, and on the reduction in activity and social contact that winter encourages. In a trial published in 2015 in the American Journal of Psychiatry, Rohan and colleagues randomly assigned 177 adults with winter SAD to six weeks of either CBT-SAD or light therapy. The CBT-SAD group attended twelve group sessions, two 90-minute sessions a week, led by doctoral-level psychologists. The light therapy group used a 10,000 lux light box for 30 minutes each morning, with the dose adjusted according to response and side effects. At the end of six weeks, both groups had improved substantially, and to a similar degree. Remission rates based on clinician ratings were almost identical, at 47.6 percent for CBT-SAD and 47.2 percent for light therapy. The authors concluded that the two were comparably effective for treating an acute episode. The more striking results came later. The researchers followed the same participants through the next two winters. In the first winter after treatment, the two groups did not differ. By the second winter, a clear gap had opened. Among those who had received CBT-SAD, 27.3 percent experienced a recurrence of winter depression, compared with 45.6 percent of those who had received light therapy. The CBT-SAD group also had less severe symptoms, and a larger share were in remission by one self-report measure, 68.3 percent compared with 44.5 percent. The authors concluded that CBT-SAD was superior to light therapy two winters after acute treatment, suggesting that its benefits were more durable. An important detail helps explain the difference. Participants who received light therapy in the trial were free to continue using their light boxes in later winters, and many did not use them consistently. Light therapy's benefits depend on continued daily use; if people stop, the benefit stops. CBT-SAD, by contrast, teaches skills that people carry with them and can use without equipment. The finding is not that light therapy fails, but that its protection depends on continued effort, while skills learned in therapy seem to keep working. A further analysis from an earlier trial by Rohan's group, published in 2013, adds another layer. Participants who began treatment with more rigid negative attitudes and more frequent negative automatic thoughts had less severe depression the following winter if they had been treated with CBT than if they had been treated with light alone. People with especially entrenched negative thinking, in other words, seem to benefit most from a treatment that targets that thinking directly. It is worth keeping these findings in proportion. They come principally from one research program, and the participants were community adults rather than students specifically. Evidence on preventing SAD in people who are currently well is also limited: a 2019 Cochrane review of psychological therapies to prevent SAD found only a single small eligible trial, of mindfulness-based cognitive therapy, which produced an inconclusive result. More research is needed. But the head-to-head comparison of CBT-SAD and light therapy is a large, carefully conducted randomized trial with long follow-up, and it points clearly toward the value of learning skills rather than relying only on a device. Table 5 summarizes the main treatment options for winter depression discussed in this book, the strength of their evidence, and their main practical advantages and limitations. Table 5. Main treatment options for winter depression compared. Treatment Evidence for SAD Speed Main advantage Main limitation Bright light therapy Strong Often 1 to 2 weeks Fast; few side effects Works only while used CBT-SAD Good; fewer trials Several weeks Benefits last into later winters Takes time and effort Antidepressant medication Good Several weeks Well studied; can prevent episodes Side effects; needs prescriber Vitamin D supplements Weak and mixed Unclear Cheap; safe at standard doses Not shown to treat SAD Sources: Lam et al., 2006; Rohan et al., 2015 and 2016; Modell et al., 2005; Okereke et al., 2020; see Notes. What happens in CBT-SAD The CBT-SAD protocol is described in a therapist guide and a companion workbook by Rohan, both titled Coping with the Seasons and published by Oxford University Press. The treatment has three main components, which are worth understanding even for students who do not have access to the full program, because each can be applied on its own. The first component is psychoeducation: learning about SAD, its causes and its course. This might seem trivial, but it matters. Understanding that winter depression is a recognizable condition with known mechanisms, rather than a personal failing, changes how people interpret their symptoms and how they respond to them. Much of the material in the early chapters of this book serves the same purpose. The second component is behavioral activation, which is explored in more depth in Chapter 7 because it overlaps heavily with maintaining social activity. The core idea is that depression reduces activity, reduced activity reduces rewarding experiences, and reduced reward deepens depression. Behavioral activation breaks this cycle by scheduling activities deliberately, especially pleasant and meaningful ones, rather than waiting until one feels like doing them. In CBT-SAD, particular attention goes to activities that people enjoy in winter, or can adapt for winter, since many summer pleasures are unavailable. Behavioral activation is well supported as a treatment for depression in general; a 2007 meta-analysis by Pim Cuijpers and colleagues found activity scheduling to be an effective treatment, comparable to full cognitive therapy. The third component is cognitive restructuring: learning to notice, examine and revise the negative thoughts that deepen low mood. In CBT-SAD, many of these thoughts are specifically about winter. Common examples include "I can't handle winter," "It's too dark to do anything," "I'm useless from November to March," "There's no point making plans because I'll just cancel," and "Winter is going to last forever." The aim is not to replace such thoughts with forced cheerfulness, but to test them against evidence and find more balanced and useful alternatives. Doing the cognitive work on campus A worked example shows how cognitive restructuring operates in practice. Consider a composite sophomore on a Monday evening in late November. She had planned to go to a study group at the library at seven. At six-thirty, looking out at the dark, she notices a surge of heaviness and the thought: "It's pitch black and freezing, I'm exhausted, and I won't get anything done there anyway. I'll just stay in." In CBT-SAD, the first step is to catch the thought and name it. The second is to notice its effect: she feels more tired and hopeless, and she is about to cancel. The third is to examine it. What is the evidence that she will get nothing done? Last week she went to the same group feeling similarly reluctant and finished a problem set she had been stuck on. What is the evidence that the dark makes going out impossible? The library is an eight-minute walk on lit paths, and she walks further in the dark every day to dinner. Is she exhausted, or is she experiencing the heaviness of low mood, which often improves after starting something? In her own records, her energy has typically risen after the first half hour of any activity. The fourth step is to generate a more balanced thought: "I feel heavy, which is what winter does to me. It is dark, but the walk is short. The last two times I went, I felt better afterwards and got work done. I can go for an hour and leave if it is truly not working." The balanced thought does not make her feel wonderful. It makes going possible. That is the point. Cognitive restructuring in SAD is not about thinking positively, but about preventing thoughts from shutting down the activities that would help. Another common pattern in SAD is rumination: repetitive, passive dwelling on how bad one feels and why. Winter rumination often takes the form of long periods lying in bed thinking about how tired one is, how dark it is, how far behind one has fallen, and how the rest of the winter will go. Rumination feels like problem-solving but rarely solves anything, and it tends to deepen low mood. CBT-SAD teaches people to notice rumination and redirect themselves toward action, even small action. A student who catches herself having spent forty minutes lying in bed thinking about her backlog can recognize the pattern, get up, and do one small concrete task, such as opening the assignment and writing the first paragraph, which usually does more for mood than the rumination did. A second example shows the same method applied to the academic worries that winter so often amplifies. A composite junior, three weeks before finals, opens his course portal to find a disappointing grade on a lab report. Within seconds the thoughts cascade: "I'm failing chemistry. This always happens to me in winter. I'll lose my scholarship. I'm not smart enough for this major." By the time he closes the laptop he feels hopeless and heavy, and he does not start the next assignment that evening, or the next. Working through this with the CBT method, he first separates the facts from the predictions. The fact is one lab report grade of C-plus. His running grade in the course, when he actually calculates it, is a B-minus. The scholarship requires a cumulative average he is comfortably above. The claim that this always happens in winter contains a grain of truth, because his grades have dipped each December, but that observation is also information he can use: it points to a seasonal pattern, not to a lack of ability, and it suggests specific actions, such as starting light therapy in October next year and asking the professor about the lab report now. His balanced thought might be: "This grade is disappointing, and winter makes me feel worse about it than it is. My overall grade is fine. The pattern of December dips is real and I can plan for it. Right now, the useful move is to email the professor and spend an hour on the next assignment." Two features of this example deserve emphasis. The first is how quickly a single event can generate a chain of catastrophic predictions when mood is low, and how much of the resulting distress comes from the predictions rather than the event. The second is that the balanced thought leads directly to action. CBT for depression is not an exercise in reassurance; it is a way of turning distress into the next useful step. For students who want to practice these skills between sessions or on their own, a simple written routine helps. A thought record, kept in a notebook or phone, has a handful of columns: the situation, the mood and its intensity, the automatic thought, the evidence for and against it, a more balanced alternative, and the mood afterwards. Filling one in takes five minutes. Doing it a few times a week, especially at the moments when a winter thought is about to cancel a plan, builds the habit of catching thoughts before they take over. Over time, many people find they can do the process in their heads, quickly and almost automatically. Rohan's workbook includes forms of this kind adapted for winter, and many counseling centers can provide similar worksheets. When approaching a counseling center or therapist, students can ask specific questions to find the right help: whether the center offers CBT for depression, individually or in groups; whether any clinicians have experience with seasonal depression; whether sessions can be scheduled at a consistent time each week through the winter months; and what the plan is for continuity over winter break, when many centers reduce services. It is also reasonable to say directly that one is interested in learning skills to use in future winters, not only in getting through this one. That framing helps a clinician tailor the work toward the long-term protection the research suggests is possible. Students can use these skills in several ways. The fullest option is CBT-SAD or CBT for depression delivered by a trained clinician, which some campus counseling centers offer individually or in groups, and which students can also seek off campus. Where full treatment is unavailable, the published workbook provides a structured, self-guided version, ideally used alongside some professional contact. Many counseling centers also offer shorter CBT-based workshops, groups or digital programs for depression that cover the same core skills, even if they are not specific to seasonal depression. The timing of learning these skills matters. Cognitive and behavioral skills are easier to learn and practice when one is not severely depressed. For students with an established seasonal pattern, the ideal time to begin is early autumn, before the worst of winter, or in the first weeks of symptoms. Students who learn the skills during one winter can then review and reapply them each subsequent autumn, which is exactly the pattern of long-term benefit the Rohan trial suggests. CBT-SAD and light therapy are not rivals. Many people use both, and there is good reason to think they complement each other: light acts quickly on the biological side, while CBT builds durable skills on the psychological side. For a student in the middle of a bad December, light therapy may provide the quick lift that makes it possible to engage with therapy at all. For a student planning ahead in September, starting CBT skills early and adding light at the first signs of decline may provide the strongest protection. The right mix depends on the individual, the severity of their symptoms, and what is available to them, and is best worked out with a clinician. Chapter 7: Staying Connected When Everyone Hibernates Of all the ways winter depression damages a student's life, social withdrawal may be the most consequential and the least noticed. It rarely happens through a single decision. A student turns down one invitation because she is tired, skips a club meeting because it is dark and cold, eats dinner in her room because the dining hall feels like too much effort. Each choice is reasonable on its own. Over several weeks, they add up to a life with far less human contact than the student had in September, often without her ever consciously deciding to pull away. Friends, who are busy with their own coursework and may assume she is simply busy too, stop asking. By January, she may feel profoundly alone on a campus full of people. This withdrawal matters for more than one reason. Social contact is one of the most reliable sources of reward and meaning in most people's lives, and losing it deepens depression. Friends also provide practical help, perspective and early warning: they notice when something is wrong, and they can encourage someone to seek help. And isolation feeds the negative thinking described in Chapter 6, because a student alone in a dark room has no one to challenge the belief that everyone else is fine and that something is uniquely wrong with her. Maintaining social activity through the darker months is therefore not an optional extra in managing SAD. It is one of its central tasks. Behavioral activation in winter The main psychological tool for this task is behavioral activation, introduced in Chapter 6 as a component of CBT-SAD. Its logic runs against ordinary intuition. Most people believe they need to feel better before they can do things; behavioral activation holds that doing things is how people come to feel better. Mood follows action more reliably than action follows mood. In winter depression, waiting until one feels like going out means waiting until spring. Behavioral activation begins with observation. For a week, a student records what she actually does each day, hour by hour, and rates her mood and her sense of pleasure and accomplishment for each activity. This often reveals patterns that are invisible from inside the day: that the hours spent scrolling in bed are consistently the lowest-rated of the week, that a half hour walking with a friend lifted her mood for the rest of the afternoon, that a morning in which she got up at a consistent time went noticeably better than one in which she slept until noon. The record provides evidence to replace the depressed mind's vague conviction that nothing helps. The next step is scheduling. Using the record as a guide, the student plans specific activities in advance, with specific times and places, rather than leaving them to how she feels on the day. The activities should include a mix of pleasure, such as seeing friends, music, films or games; mastery, such as completing a piece of work or learning a skill; and connection. They should start small. A depressed student who plans to go to the gym five days a week, join two clubs and host a dinner party is setting up a failure. One who plans to eat lunch with a friend on Tuesday, go to one club meeting on Thursday and take a twenty-minute walk outside on Saturday morning is setting up a series of small successes that can be built on. Winter adds a particular twist. Many of the activities that sustained a student in September are seasonal: outdoor sports, spontaneous time on the quad, long evening walks. Part of behavioral activation for SAD involves deliberately developing a repertoire of winter activities to replace them. Some embrace the season, such as skating, sledding, winter hiking or cross-country skiing where available, while others move indoors: climbing gyms, pools, board game nights, cooking with friends, music rehearsals, theater productions and volunteering. The research on CBT-SAD emphasizes finding activities that can be enjoyed in winter rather than simply enduring the season until the summer activities return. A student who has only summer pleasures has, in effect, nothing to do for five months of the academic year. It also helps to identify the specific obstacles that stand between plan and action, and to solve them in advance. If the problem is that it is dark and cold at seven in the evening, perhaps the activity can move to the afternoon, or a friend can be arranged to walk over together. If the problem is waking up too late to make a morning commitment, the solution may lie in the sleep and light routines of Chapter 4. If the problem is that the student cancels when the moment arrives, the solution may be to make commitments that involve other people, who make cancellation harder, or to use the cognitive techniques of Chapter 6 on the thoughts that arise at the moment of cancelling. Building social structure that survives the dark The single most useful principle for maintaining social life in winter is to replace spontaneity with structure. In early autumn, much social contact on campus happens by accident: running into people, joining impromptu plans, lingering after class. In winter, those accidental contacts diminish sharply because people are indoors and hurrying between buildings. Students who rely on chance for their social lives will find themselves with much less of one. Students who build regular, scheduled contact into their weeks will find that it carries them through. Standing commitments are the backbone of this structure. A weekly meal with the same friends at the same time, a regular study group, a club with scheduled meetings, a recreational sports team, a choir, a religious community, a volunteer shift, or a regular exercise class all provide contact that does not depend on anyone taking the initiative each week. They also provide a reason to get out of bed and out of the room that is independent of mood. For students with SAD, the ideal is to set up several such commitments in September and October, while energy is still high, so that they are established habits by the time winter arrives. Commitments with some accountability built in are especially valuable. A team that needs a certain number of players, a volunteer role where others depend on the student turning up, a lab or project group with shared deadlines, or a friend who has agreed to be a regular workout partner all make it harder to drift away. This is not a trick; it is the same mechanism that makes most human communities work. People show up for others more reliably than they show up for themselves. It is also worth being deliberate about telling at least one or two friends what is going on. A student need not share a diagnosis with everyone, and there are good reasons to be selective. But a trusted friend who knows that winter is hard, and who knows the early signs, can notice withdrawal early, extend invitations when the student is least likely to initiate, and not take cancellations personally. The conversation can be simple: "I tend to get pretty low in the winter, and I end up hiding in my room. If you notice me disappearing, please keep asking me to things, even if I say no a few times." Many students find that friends are relieved to be told, and some discover that the friend has a similar experience. Structure is easiest to build in advance, but many students reading this will already be in the middle of a winter in which they have withdrawn. Re-entry can feel harder than starting fresh, because withdrawal brings its own embarrassment. A student who has missed six weeks of club meetings, left group chats unanswered and turned down every invitation may assume that the door has closed, and that turning up again will require an explanation she cannot face. In practice, most people are far less focused on her absence than she imagines, and a short, honest message is usually enough: "Sorry I've been off the radar, winter has been rough for me. Are you going to the meeting on Thursday? I'd like to come back." No detailed explanation is owed. Friends generally respond with warmth, and the relief of having sent the message tends to be out of all proportion to the effort. It helps to re-enter through the easiest door first. That might be the friend who has kept texting, the class study group that meets anyway, or a structured activity where showing up requires little conversation, such as a fitness class, a volunteering shift or a rehearsal. The goal of the first week is not to rebuild an entire social life but to break the pattern of avoidance with one or two contacts. Each successful contact makes the next one easier, which is behavioral activation working as intended. Introverted students sometimes worry that advice about social activity means they should become someone they are not. It does not. The aim is not maximum socializing but enough regular, meaningful contact to prevent isolation from feeding depression. For an introvert, that might be two close friends seen reliably each week and one quiet group activity, which is quite different from a packed social calendar. The useful question is not "How much am I going out?" but "Compared with the version of me who was well in September, how much have I pulled back, and how much of that pulling back was a choice rather than the depression choosing for me?" Students living alone, whether in off-campus apartments, single dorm rooms or graduate housing, face particular risk because they lose the incidental contact of shared living. A roommate notices when someone has not left the room in three days; an empty apartment does not. These students benefit most from deliberately scheduled contact, from arranging to work in shared spaces such as libraries, cafes or departmental lounges rather than at home, and from having at least one person who checks in regularly. Physical activity is worth singling out because it combines several benefits. Exercise has good evidence as a treatment for depression generally, though specific trials in SAD are few. When done outdoors in daylight, it adds light exposure; when done with others, it adds social contact; when scheduled, it adds structure. A regular midday walk or run with a friend may be one of the most efficient single habits for a student with seasonal depression, since it addresses so many of the mechanisms at once. On campuses where winter weather makes outdoor exercise difficult, an indoor facility with large windows, visited in the morning or around midday, captures some of the same benefit. Consider a composite graduate student in a biology program in Minnesota. In her first winter she worked alone in a windowless lab from nine to seven, ate at her bench and went home in the dark. By February she was barely functioning. The following autumn, working with a counselor, she made three changes. She joined a departmental running group that met at noon three days a week, so she was outside in daylight and with other people during the brightest part of the day. She arranged a standing Thursday dinner with two classmates. And she agreed with her advisor to start her lab days at a consistent time, after a morning light box session at home. None of these changes eliminated her winter symptoms, but together they transformed her second winter, and the running group became the place where she made her closest friends in the program. Traps, helpers and the role of others Several common winter habits deserve particular caution because they can feel like coping while making things worse. Alcohol is the most important. On many campuses, drinking is a central part of social life, and in winter it can seem like a way to feel warmer, more sociable and less low. Alcohol is a depressant, however. It disrupts sleep, particularly the later part of the night, worsens mood in the following days, and interacts badly with many antidepressant medications. For students with depression, heavy drinking is also associated with a higher risk of suicidal behavior. Students with SAD need not avoid all social events involving alcohol, but they benefit from keeping their drinking moderate, from finding winter social activities that do not revolve around it, and from noticing whether they are using it to manage their mood. Screens and online contact are more complicated. Messaging, video calls and online communities can provide real connection, especially for students far from home or with limited mobility, and they are far better than no contact at all. But they are not a full substitute for in-person contact, and late-night screen use can disrupt sleep and delay the body clock, as Chapter 2 explained. Passive scrolling through others' curated lives, in particular, can worsen the sense that everyone else is thriving. A useful rule of thumb is to use online contact to arrange and supplement in-person contact rather than replace it, and to keep the last hour before bed as screen-light as possible. Oversleeping is a subtler trap. Because hypersomnia is a symptom of winter depression, students often feel that they need more sleep, and some extra sleep in winter is normal. But sleeping late into the day reduces morning light exposure, delays the body clock, cuts into daytime activity and social contact, and often leaves people feeling worse. A consistent wake time, reinforced by morning light, generally does more good than extra hours in bed. Long afternoon naps can have similar effects. If daytime sleepiness is overwhelming despite adequate sleep, it is worth raising with a clinician, since it can signal a sleep disorder or another medical problem. Friends, roommates and residence staff can do a great deal to help, and the most useful help is usually simple and persistent. Keep inviting, even after refusals; a student with SAD may decline five invitations and accept the sixth, and the sixth may matter a great deal. Make invitations specific and easy: "I'm going to the dining hall at six, come with me" works better than "We should hang out sometime." Offer to walk together to classes or activities, which reduces the barrier of going out in the dark alone. Do not lecture about light boxes or vitamins, but it is reasonable to mention, once and kindly, that the counseling center might help. And take any mention of hopelessness, not wanting to be alive, or thoughts of self-harm seriously; Chapter 8 describes what to do. Parents can help in parallel ways. A regular, predictable call or video chat, particularly in November and January, provides contact and an opportunity to notice changes. Parents can support structure during winter break by encouraging a consistent wake time, time outside and time with friends, rather than treating break as a period of pure rest. They can also make it clear that seeking help at the counseling center is normal and welcome, and that a lighter course load or a difficult semester is not a disgrace. Many students hide their struggles from their families for fear of disappointing them, and a parent who has said plainly and in advance that they would rather know is giving their child a real gift. Institutions matter too. Campuses that concentrate orientation and social programming in September and let it thin out as the semester progresses are, in effect, offering the most support when students need it least. Residence life programs, student organizations and departments can deliberately schedule events for the darker months, such as communal meals, bright and welcoming evening study spaces, winter outdoor activities, and events in the first weeks of the spring semester, when students return to campus at their lowest. Graduate programs can build regular social contact into the winter schedule for students who otherwise spend their days alone in labs and offices. None of this requires large budgets. It requires recognizing that winter is a predictable period of risk and planning for it, which is the argument of this book applied to a whole community rather than a single student. Chapter 8: Getting Help on Campus and in a Crisis Many students with winter depression try to manage it alone for a long time before seeking help. Some do not recognize their symptoms as depression. Some believe they should be able to cope by themselves, or that their problems are not serious enough to justify taking up a counselor's time. Some worry about cost, about what it will mean for their academic record, or about being pushed toward medication. Some simply cannot find the energy, which is itself a symptom. The result is that students often reach help in February, when they are already deep into an episode and behind in their work, rather than in October, when intervention would be easier and more effective. This chapter is about shortening that delay. It covers what campus services typically offer and how to use them, the role of medication, academic accommodations, and, most importantly, what to do when depression becomes dangerous. Using campus counseling and health services Most colleges and universities in the United States and Canada have a counseling center, and many have a student health service with physicians, nurse practitioners and sometimes psychiatrists. Services vary widely between institutions, but many provide free or low-cost short-term counseling, group therapy, workshops on skills such as managing mood and stress, psychiatric evaluation and medication management, referrals to community providers for longer-term care, and crisis services, including same-day appointments and after-hours phone lines. Some campuses offer programs specifically for seasonal depression, and some lend light boxes or have light therapy stations available to students. The first contact is often the hardest part. Many counseling centers now use a brief initial consultation, sometimes by phone or video, to understand what a student needs and direct them to the right service. It helps to prepare a few points in advance. What are the symptoms, and when did they start? Has this happened in previous winters? How are sleep, appetite, energy and concentration? How is academic work going? Has there been any use of alcohol or other substances to cope? Have there been any thoughts of death, suicide or self-harm? The mood log described in Chapter 1 is useful here, because it replaces vague impressions with concrete information. A student can say something like: "For the last three winters I've started sleeping ten hours a day, eating a lot more and pulling away from friends around early November, and it lifts in April. It's happening again and it's affecting my classes." That sentence gives a clinician almost everything needed to begin. Students often worry about confidentiality. Counseling records are generally kept separate from academic records and are protected by privacy laws and professional ethics; they are not shared with professors, parents or the university administration without the student's consent, except in limited circumstances such as a serious and imminent risk of harm. Students who have concerns should ask about the confidentiality policy at the first appointment, and clinicians are used to answering this question. Demand for campus counseling often peaks at predictable times, including the weeks before and during final exams. Some centers have waiting lists for ongoing therapy. This is another reason to make first contact early in the autumn, when waits tend to be shorter, and to ask about interim options, such as groups, workshops or drop-in hours, if there is a wait. It is also worth asking what happens over winter break, when many centers reduce services, and whether telehealth appointments are available to students who travel home. Because licensing rules often restrict clinicians from providing therapy to clients located in other states or provinces, students who will spend break elsewhere should check in advance whether sessions can continue. Consider a composite senior at a university in Michigan who has had three hard winters and has never told anyone. In the second week of October, after reading about seasonal patterns, she sends a short message through the counseling center's online portal asking for an initial consultation. The fifteen-minute phone call that follows is less daunting than she feared: the counselor asks about her sleep, mood, grades, drinking and safety, and about previous winters. She is offered a place in a six-week skills group that starts the following week, an appointment with a physician at student health to check her thyroid and talk about light therapy, and information about the accessibility office in case she needs flexibility during finals. None of it is dramatic. By the time the clocks change in November, she has a plan, a light box she is using each morning, a group she attends on Wednesdays and a clinician who knows her history. Her winter is not free of low days, but it is the first in four years in which she does not fall behind. Cost and access are real barriers for some students, and there are ways around most of them. Many counseling centers charge nothing for short-term care, and student health insurance plans usually cover mental health treatment, though students should check what their plan includes and whether off-campus providers are in network. Students covered by a parent's insurance may have access to providers in their home area and, increasingly, through telehealth. Community mental health centers, university training clinics run by psychology or social work graduate programs, and practitioners who offer sliding-scale fees can provide lower-cost options. Where the campus center refers a student off campus for longer-term care, it is worth asking the center for help identifying providers who accept the student's insurance, since navigating this alone while depressed is exactly the kind of task that stalls. Primary care is another entry point. A physician or nurse practitioner at student health can assess symptoms, rule out medical causes such as thyroid problems or anemia, discuss light therapy safety, prescribe medication where appropriate, and make referrals. For students with more complex needs, including those with possible bipolar disorder, severe depression or depression that has not responded to initial treatment, a psychiatric evaluation is appropriate. Medication and academic accommodations Antidepressant medication is an established treatment for seasonal depression, and for some students it is the right choice, either alone or combined with light therapy or CBT. The Can-SAD trial described in Chapter 4 found that fluoxetine, a selective serotonin reuptake inhibitor, or SSRI, was as effective as light therapy over eight weeks, though somewhat slower and with more side effects. Several other SSRIs are also commonly used. One medication has a particular role. Bupropion, in its extended-release form known as bupropion XL, is approved by the US Food and Drug Administration for the prevention of seasonal major depressive episodes in people with a history of SAD. The approval rested on three randomized, placebo-controlled trials published by Jack Modell and colleagues in 2005, involving 1,042 patients across the northern United States and Canada who began treatment in autumn, while still well, and continued until spring. Recurrence rates of major depression over the winter were lower with bupropion than with placebo in each trial, at 19 versus 30 percent, 13 versus 21 percent, and 16 versus 31 percent, amounting to a relative reduction in risk of 44 percent across the three studies. Notably, although participants reported an average of 13 previous seasonal episodes, almost 60 percent had never been treated for depression before. That statistic says a great deal about how many people with SAD endure it for years without help. Medication decisions are individual and should be made with a prescriber who knows the student's full history. A few general points are worth knowing in advance. Antidepressants usually take several weeks to reach their full effect, so starting in autumn gives them time to work before the darkest months. All have possible side effects, which vary between drugs and individuals and often ease over the first weeks. Some can trigger mania in people with bipolar disorder, which is another reason a careful assessment matters. Stopping suddenly can cause unpleasant withdrawal effects and risks relapse, so changes should be planned with a prescriber. And people under 25 who start antidepressants should be monitored for any increase in suicidal thoughts in the early weeks of treatment; the FDA requires a warning to this effect on antidepressant labels. That risk is small and is weighed against the substantial risk of untreated depression, but it means that students starting medication should know to report any worsening promptly, and it is a good reason for a friend or family member to know that treatment has begun. Winter depression can seriously affect academic performance, and students have options that many do not know about. In the United States, students with a documented condition that substantially limits a major life activity may be eligible for academic accommodations under the Americans with Disabilities Act and Section 504 of the Rehabilitation Act; Canada and other countries have comparable frameworks. Depression, including seasonal depression, can qualify. Accommodations are arranged through the campus disability services or accessibility office, usually with documentation from a clinician, and might include flexibility on deadlines or attendance during periods of acute symptoms, extended time on examinations, a reduced course load, or priority registration that allows a student to schedule classes at times that support a stable routine. A student need not disclose a diagnosis to individual professors; the accessibility office communicates only the accommodations. Other academic options exist for students who are already struggling. Many institutions allow late withdrawal from individual courses, incomplete grades that allow work to be finished after the term ends, or, in more serious situations, a medical leave of absence. These options are almost always easier to arrange before a semester has fully collapsed than after, which is one more argument for early contact. Academic advisers and deans of students can explain the options, and are generally far more understanding than anxious students expect. When depression becomes dangerous Seasonal depression is depression, and depression can lead to suicidal thinking. Suicide is among the leading causes of death for people of college age. Most students with winter depression will never reach a crisis, but every student, and everyone who cares about one, should know the warning signs and what to do. Warning signs that someone may be at risk include talking about wanting to die or to kill themselves; talking about feeling hopeless, trapped, or like a burden to others; searching for ways to end their life; increasing their use of alcohol or drugs; withdrawing from friends and activities; sleeping too much or too little; showing extreme mood swings; giving away possessions; or saying goodbye in unusual ways. For students with seasonal patterns, the transitions into winter and into spring both deserve attention, as discussed in Chapter 3. Table 6 sets out a simple way to match the level of concern to an appropriate response. It is a general guide and does not replace clinical judgment; when in doubt, choose the more urgent option. Table 6. Matching the level of concern to the response. Situation Examples What to do Winter low, no safety concerns Fatigue, oversleeping, withdrawal, slipping grades Book a counseling or health appointment; start the plan in this book Worsening depression Hopelessness, passive thoughts of not wanting to be alive Seek same-day help from counseling; call or text 988 Active suicidal thoughts Thinking about how or when; urges to self-harm Call or text 988 now; tell someone; stay with others Immediate danger A plan with means at hand; an attempt in progress Call 911 or campus police; go to an emergency department The 988 Suicide and Crisis Lifeline, which launched in the United States in July 2022, connects callers, texters and online chatters with trained crisis counselors at any hour. It can be reached by calling or texting 988, or through the chat service at 988lifeline.org, and it is not only for people who are actively suicidal; it can be used by anyone in emotional distress, and by anyone worried about someone else. Veterans and service members can press 1 after dialing to reach the Veterans Crisis Line. Canada launched its own 9-8-8 suicide crisis helpline in November 2023. In the United Kingdom and Ireland, Samaritans can be reached at 116 123. Students studying in other countries should look up the local crisis line and emergency number at the start of the year and save them in their phones, since this is not information anyone wants to search for in a crisis. Many campus counseling centers also run their own after-hours crisis lines, and campus police or security can respond to emergencies on campus. A student who is having thoughts of suicide can take several immediate steps. The first is to tell someone, whether a friend, a family member, a resident adviser, a counselor or a crisis line. The second is to put distance between themselves and any means of self-harm, such as medications, weapons or other items, by asking someone to hold them or by going somewhere else. The third is to avoid alcohol and drugs, which increase impulsivity. The fourth is to stay with other people until the most intense feelings pass or professional help is in place. Many counseling centers help students write a short safety plan: a written list of personal warning signs, coping strategies, people to contact and professional resources, kept somewhere easy to find. Writing such a plan in a calm period, especially for a student with a history of severe winter episodes, is a sensible part of autumn preparation. Friends and roommates who are worried about someone should know that asking directly about suicide does not plant the idea or increase risk; research reviewing the evidence on this question has found no sign that asking causes harm, and it may bring relief. A simple, direct question works best: "Sometimes when people feel this bad, they think about suicide. Are you thinking about that?" If the answer is yes, the next steps are to listen without judgment, not to promise secrecy, to help the person contact 988 or a counselor, and to stay with them if the danger seems immediate. In an emergency, calling 911 or campus police is the right choice, even if it feels like an overreaction. A friend does not need to be a therapist, only a bridge to one. Friends should also remember that supporting someone in crisis is demanding, and that they are entitled to seek help and advice for themselves from the same counseling services. Resident advisers, faculty and staff often have specific protocols for responding to students in distress, and most campuses have a behavioral intervention or care team that coordinates support for students of concern. Staff who are unsure what to do can usually consult the counseling center directly. A short phone call to ask for advice is always better than waiting to see whether a worrying situation resolves itself. Help is most effective when it is sought early, but it is never too late to seek it. The student who reaches out in February is not failing to plan; they are doing exactly the right thing at the moment they are able to do it. The next winter can then be approached differently, with the benefit of everything learned, and that is the subject of the conclusion. Conclusion Seasonal depression presents a strange asymmetry. The time when a student is best able to do something about it is the time when it seems least necessary. In September, energy is high, friends are plentiful, the days are long, and the memory of last February feels distant and faintly unreal. Setting up a light box, booking a counseling appointment, arranging standing commitments with friends or asking a doctor about preventive medication can seem like an overreaction, like buying a snow shovel in August. By December, when the need is obvious, the capacity to act has shrunk. The same fatigue, heaviness and withdrawal that define the illness make it hard to research a light box, make a phone call, or ask a friend for help. Depression is, among other things, a disorder of initiative, and winter depression arrives precisely when initiative is needed most. The chapters of this book, read together, suggest a way out of that trap, and it is more specific than the general advice to "take care of yourself in winter." It rests on three observations that emerge only when the evidence is laid side by side. What the evidence adds up to The first observation is that the treatments differ in speed, and the differences point to a sequence rather than a choice. Light therapy works within a week or two but stops working when it stops being used. Antidepressants take several weeks to act. CBT-SAD takes six weeks or more to learn but continues to protect people in later winters. Social and behavioral structure takes weeks to build and is far easier to build in October than in January. A student who waits until symptoms are severe has access only to the fastest tool, light, and even that requires a level of daily consistency that is hard to muster in the depths of an episode. A student who starts earlier can put the slower, more durable tools in place first and add light as the days shorten. Starting early is not merely better in some general sense; it is the only way to use the full range of treatments. The second observation is that the most durable protection comes from what a student learns rather than what a student buys. The contrast between the first and second winter after treatment in Kelly Rohan's trial is the clearest single finding in this book. Two groups of people improved equally in the short term. Two winters later, those who had learned cognitive and behavioral skills were markedly less likely to have become depressed again than those who had used light alone, largely because light only works when it is used and skills travel with the person. The vitamin D story points the same way from the opposite direction: the remedy that required the least effort had the weakest evidence. None of this makes light therapy less valuable; it remains the fastest and most practical treatment for an active episode, and many people rightly use it every winter. It means that a student who relies only on a device has an incomplete plan. The third observation is that prevention has actually been demonstrated. The bupropion trials enrolled people in autumn while they were still well and reduced the risk of a winter episode. The CBT-SAD follow-up showed that skills learned in one winter reduced recurrence two winters later. The research on preventive light therapy remains thin, but clinical practice supports starting at the first signs. Taken together, these findings establish that winter depression is not simply something that happens to a person and must be endured. Its course can be changed by action taken before it begins, and the earlier the action, the more options exist. What follows from these observations is that the most useful single thing a student with a seasonal pattern can do is to make a written plan for winter while they are well, and to make it concrete enough that a depressed version of themselves can follow it without having to think. A plan in one's head tends to dissolve at the first dark morning. A plan on paper, shared with at least one other person, has a much better chance of surviving. A plan made in October Such a plan need not be long. Drawing on the preceding chapters, it might cover the following: 1. A personal record of past winters: when symptoms usually begin, what the first signs are, and what has helped or hurt before. 1. A fixed daily wake time for the academic year, including weekends, and a plan to keep it within an hour or so over winter break. 2. A light plan: a suitable light box obtained and tested in advance, a start date or trigger, such as the first two days of early warning signs or the end of October, and a clinician's approval where Chapter 4's cautions apply. 3. A decision about vitamin D made on the basis of evidence, not hope: a standard dose for general health if appropriate, a clinician's advice if deficiency is likely, and no expectation that it will treat depression. 4. A skills plan: a CBT-SAD group, individual therapy, or a structured self-guided workbook, begun early in the autumn, and the thought-record habit described in Chapter 6. 5. Two or three standing social commitments established before November, and the names of two friends who have been told what the early signs look like and asked to keep inviting. 6. A regular form of physical activity, ideally outdoors in daylight and with other people. 7. The key dates of the academic calendar marked in advance: the clock change, the weeks before finals, the break, and the first week of the spring term, with a specific plan for each. 8. Professional contacts saved: the counseling center, student health, the accessibility office, and a prescriber if medication is part of the plan. 9. A safety section: personal warning signs, the 988 Lifeline and local emergency numbers, and what the student will do and whom they will contact if things get dangerous. Every item on this list is ordinary. None requires special resources beyond what most campuses already provide. The value lies not in any single item but in the decision to put them in place before they are needed, and in the fact that the list, once written, can be reused and improved every year. Beyond the individual student That last point matters because seasonal depression, for those who have it, is not a one-year problem. It is a long relationship with the calendar. The student who learns to manage their first bad winter at nineteen is also learning how to manage the winters of their twenties and thirties, in jobs and cities and relationships not yet imagined. College, for all its particular pressures, is an unusually good place to learn this. It offers free or inexpensive counseling, health services, structured communities, flexible schedules and a clearly marked calendar. Few settings later in life will offer as much support for building the habits that protect against winter depression. A student who uses these years to understand their own pattern and practice managing it leaves with something that will outlast any degree. The same argument applies to the people and institutions around students. Parents, friends, residence staff, faculty and counseling centers all know, or can know, when winter falls. The risk is visible on the calendar. Programming in November and January, check-ins after the clocks change and at the start of the spring term, loanable light boxes, winter skills groups offered in October rather than February: these are modest measures, and they follow directly from taking the predictability of seasonal depression seriously. Seasonal affective disorder will continue to arrive with the shortening days for many students. What this book has argued is that it need not arrive unannounced, and it need not be faced alone or unprepared. The darkness is predictable. So, with a plan made in the light, is much of the response. Notes Introduction 1. Rosenthal NE, Sack DA, Gillin JC, et al. Seasonal affective disorder: a description of the syndrome and preliminary findings with light therapy. Archives of General Psychiatry. 1984;41(1):72-80. https://doi.org/10.1001/archpsyc.1984.01790120076010 2. 988 Suicide and Crisis Lifeline. https://988lifeline.org Chapter 1: Winter Blues or Winter Depression? 1. Kasper S, Wehr TA, Bartko JJ, Gaist PA, Rosenthal NE. Epidemiological findings of seasonal changes in mood and behavior: a telephone survey of Montgomery County, Maryland. Archives of General Psychiatry. 1989;46(9):823-833. https://doi.org/10.1001/archpsyc.1989.01810090065010 2. Kim YH, et al. Global prevalence of seasonal affective disorder by latitude: a systematic review and meta-analysis. Journal of Affective Disorders. 2025;390. https://www.sciencedirect.com/science/article/abs/pii/S0165032725012492 3. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition, Text Revision (DSM-5-TR). Washington, DC: American Psychiatric Association Publishing; 2022. Specifier "with seasonal pattern." 4. Rosenthal NE, et al. 1984 (see Introduction, note 1), for hypersomnia, overeating, carbohydrate craving and the proportion of bipolar patients in the original series. 5. Seasonal Affective Disorder. StatPearls. Treasure Island, FL: StatPearls Publishing. https://www.ncbi.nlm.nih.gov/books/NBK568745/ Chapter 2: Why Darkness Gets Into Your Head 1. Rosen LN, Targum SD, Terman M, et al. Prevalence of seasonal affective disorder at four latitudes. Psychiatry Research. 1990;31(2):131-144. https://doi.org/10.1016/0165-1781(90)90116-m 2. Mersch PP, Middendorp HM, Bouhuys AL, Beersma DG, van den Hoofdakker RH. Seasonal affective disorder and latitude: a review of the literature. Journal of Affective Disorders. 1999;53(1):35-48. 3. Kim YH, et al. 2025 (see Chapter 1, note 2). 4. Lewy AJ, Lefler BJ, Emens JS, Bauer VK. The circadian basis of winter depression. Proceedings of the National Academy of Sciences of the USA. 2006;103(19):7414-7419. https://doi.org/10.1073/pnas.0602425103 5. Roenneberg T, Kuehnle T, Pramstaller PP, et al. A marker for the end of adolescence. Current Biology. 2004;14(24):R1038-R1039. 6. Mc Mahon B, Andersen SB, Madsen MK, et al. Seasonal difference in brain serotonin transporter binding predicts symptom severity in patients with seasonal affective disorder. Brain. 2016;139(Pt 5):1605-1614. https://doi.org/10.1093/brain/aww043 7. Sitnikov L, Rohan KJ, Evans M, Mahon JN, Nillni YI. Cognitive predictors and moderators of winter depression treatment outcomes in cognitive-behavioral therapy vs. light therapy. Behaviour Research and Therapy. 2013;51(12):872-881. https://doi.org/10.1016/j.brat.2013.09.010 Chapter 3: The Academic Year Meets the Solar Year 1. Rohan KJ, Meyerhoff J, Ho SY, et al. Outcomes one and two winters following cognitive-behavioral therapy or light therapy for seasonal affective disorder. American Journal of Psychiatry. 2016;173(3):244-251. https://doi.org/10.1176/appi.ajp.2015.15060773 (follow-up visits in January or February of each year). 2. American Psychiatric Association, DSM-5-TR (see Chapter 1, note 3), on seasonal psychosocial stressors as an alternative explanation. Chapter 4: Bright Light Therapy 1. Golden RN, Gaynes BN, Ekstrom RD, et al. The efficacy of light therapy in the treatment of mood disorders: a review and meta-analysis of the evidence. American Journal of Psychiatry. 2005;162(4):656-662. https://doi.org/10.1176/appi.ajp.162.4.656 2. Pjrek E, Friedrich ME, Cambioli L, et al. The efficacy of light therapy in the treatment of seasonal affective disorder: a meta-analysis of randomized controlled trials. Psychotherapy and Psychosomatics. 2020;89(1):17-24. https://doi.org/10.1159/000502891 3. Lam RW, Levitt AJ, Levitan RD, et al. The Can-SAD study: a randomized controlled trial of the effectiveness of light therapy and fluoxetine in patients with winter seasonal affective disorder. American Journal of Psychiatry. 2006;163(5):805-812. https://doi.org/10.1176/ajp.2006.163.5.805 4. Terman JS, Terman M, Lo ES, Cooper TB. Circadian time of morning light administration and therapeutic response in winter depression. Archives of General Psychiatry. 2001;58(1):69-75. https://doi.org/10.1001/archpsyc.58.1.69 5. Seasonal Affective Disorder. StatPearls (see Chapter 1, note 5), for equivalent intensity-duration regimens, dawn simulation intensities, and ophthalmological precautions. 6. Nussbaumer-Streit B, Forneris CA, Morgan LC, et al. Light therapy for preventing seasonal affective disorder. Cochrane Database of Systematic Reviews. 2019;3:CD011269. https://doi.org/10.1002/14651858.CD011269.pub3 Chapter 5: Vitamin D and the Limits of a Simple Story 1. Webb AR, Kline L, Holick MF. Influence of season and latitude on the cutaneous synthesis of vitamin D3: exposure to winter sunlight in Boston and Edmonton will not promote vitamin D3 synthesis in human skin. Journal of Clinical Endocrinology and Metabolism. 1988;67(2):373-378. 2. Okereke OI, Reynolds CF, Mischoulon D, et al. Effect of long-term vitamin D3 supplementation vs placebo on risk of depression or clinically relevant depressive symptoms and on change in mood scores: a randomized clinical trial. JAMA. 2020;324(5):471-480. https://doi.org/10.1001/jama.2020.10224 3. Frandsen TB, Pareek M, Hansen JP, Nielsen CT. Vitamin D supplementation for treatment of seasonal affective symptoms in healthcare professionals: a double-blind randomised placebo-controlled trial. BMC Research Notes. 2014;7:528. https://doi.org/10.1186/1756-0500-7-528 4. Mikola T, Marx W, Lane MM, et al. The effect of vitamin D supplementation on depressive symptoms in adults: a systematic review and meta-analysis of randomized controlled trials. Critical Reviews in Food Science and Nutrition. 2023;63(33):11784-11801 (published online 2022). https://doi.org/10.1080/10408398.2022.2096560 5. Nussbaumer-Streit B, Greenblatt A, Kaminski-Hartenthaler A, et al. Melatonin and agomelatine for preventing seasonal affective disorder. Cochrane Database of Systematic Reviews. 2019;6:CD011271. https://doi.org/10.1002/14651858.CD011271.pub3 6. Erland LA, Saxena PK. Melatonin natural health products and supplements: presence of serotonin and significant variability of melatonin content. Journal of Clinical Sleep Medicine. 2017;13(2):275-281. 7. Okereke OI, Vyas CM, Mischoulon D, et al. Effect of long-term supplementation with marine omega-3 fatty acids vs placebo on risk of depression or clinically relevant depressive symptoms and on change in mood scores: a randomized clinical trial. JAMA. 2021;326(23):2385-2394. 8. Demay MB, Pittas AG, Bikle DD, et al. Vitamin D for the prevention of disease: an Endocrine Society clinical practice guideline. Journal of Clinical Endocrinology and Metabolism. 2024;109(8):1907-1947. Summary: https://www.endocrine.org/clinical-practice-guidelines/vitamin-d-for-prevention-of-disease 9. National Institutes of Health, Office of Dietary Supplements. Vitamin D: Fact Sheet for Health Professionals. https://ods.od.nih.gov/factsheets/VitaminD-HealthProfessional/ Chapter 6: Learning to Outlast Winter 1. Rohan KJ, Mahon JN, Evans M, et al. Randomized trial of cognitive-behavioral therapy versus light therapy for seasonal affective disorder: acute outcomes. American Journal of Psychiatry. 2015;172(9):862-869. https://doi.org/10.1176/appi.ajp.2015.14101293 2. Rohan KJ, Meyerhoff J, Ho SY, et al. 2016 (see Chapter 3, note 1). 3. Sitnikov L, et al. 2013 (see Chapter 2, note 7). 4. Forneris CA, Nussbaumer-Streit B, Morgan LC, et al. Psychological therapies for preventing seasonal affective disorder. Cochrane Database of Systematic Reviews. 2019;5:CD011270. https://doi.org/10.1002/14651858.CD011270.pub3 5. Rohan KJ. Coping with the Seasons: A Cognitive-Behavioral Approach to Seasonal Affective Disorder. Therapist Guide and Workbook. Treatments That Work series. New York: Oxford University Press; 2008. 6. Cuijpers P, van Straten A, Warmerdam L. Behavioral activation treatments of depression: a meta-analysis. Clinical Psychology Review. 2007;27(3):318-326. 7. Table 5 draws on Lam et al. 2006 (Chapter 4, note 3), Rohan et al. 2015 and 2016, Modell et al. 2005 (Chapter 8, note 1) and Okereke et al. 2020 (Chapter 5, note 2). Chapter 7: Staying Connected When Everyone Hibernates 1. Cuijpers P, et al. 2007 (see Chapter 6, note 6). 2. Rohan KJ. Coping with the Seasons (see Chapter 6, note 5), on developing winter-compatible activities. Chapter 8: Getting Help on Campus and in a Crisis 1. Modell JG, Rosenthal NE, Harriett AE, et al. Seasonal affective disorder and its prevention by anticipatory treatment with bupropion XL. Biological Psychiatry. 2005;58(8):658-667. https://doi.org/10.1016/j.biopsych.2005.07.021 2. Lam RW, et al. 2006 (see Chapter 4, note 3). 3. US Food and Drug Administration. Antidepressant medicines: boxed warning on suicidality in children, adolescents and young adults. https://www.fda.gov 4. Dazzi T, Gribble R, Wessely S, Fear NT. Does asking about suicide and related behaviours induce suicidal ideation? What is the evidence? Psychological Medicine. 2014;44(16):3361-3363. 5. 988 Suicide and Crisis Lifeline. https://988lifeline.org 6. 9-8-8 Suicide Crisis Helpline (Canada). https://988.ca 7. Samaritans (UK and Ireland). https://www.samaritans.org Further Reading Norman E. Rosenthal, Winter Blues: Everything You Need to Know to Beat Seasonal Affective Disorder, fourth edition (Guilford Press, 2012). A readable, practical book by the psychiatrist who led the team that first described SAD. Strong on light therapy and on the lived experience of winter depression. Kelly J. Rohan, Coping with the Seasons: A Cognitive-Behavioral Approach to Seasonal Affective Disorder, Workbook (Oxford University Press, Treatments That Work series).** The client workbook for the CBT-SAD program discussed in Chapter 6. Best used alongside a clinician, but clear enough to guide self-directed practice. Michael Terman and Ian McMahan, Chronotherapy: Resetting Your Inner Clock to Boost Mood, Alertness, and Quality Sleep (Avery, 2012).** A detailed guide to light therapy, dawn simulation and body-clock timing from one of the field's leading researchers. Useful for anyone who wants to understand the timing questions in Chapters 2 and 4 in more depth. Michael E. Addis and Christopher R. Martell, Overcoming Depression One Step at a Time: The New Behavioral Activation Approach to Getting Your Life Back (New Harbinger, 2004).** A practical workbook on behavioral activation, the approach at the heart of Chapter 7. Not specific to SAD, but directly applicable to winter withdrawal. Center for Environmental Therapeutics (cet.org). A nonprofit organization founded by light therapy researchers that provides information on light therapy, device selection and self-assessment questionnaires for timing. 988 Suicide and Crisis Lifeline (988lifeline.org). Call or text 988 in the United States, or chat online, at any hour. The website also has guidance for people worried about someone else. Your campus counseling center, student health service and accessibility office. The most important resources of all for students. Their websites usually list crisis numbers, workshop schedules and how to request accommodations; it is worth reading them in September. Hashtags: #SeasonalAffectiveDisorder #SADAwareness #WinterDepression #SeasonalDepression #WinterMentalHealth #StudentMentalHealth #CollegeMentalHealth #UniversityWellbeing #AcademicYearStress #WinterBlues #MentalHealthAwareness #DepressionAwareness #SeasonalMoodChanges #BrightLightTherapy #LightTherapy #DawnSimulation #VitaminD #CircadianRhythm #SleepHealth #SleepHygiene #CognitiveBehavioralTherapy #CBTForSAD #BehavioralActivation #MoodManagement #WinterWellness #MentalHealthSupport #CampusCounseling #AcademicAccommodations #SocialConnection #SeasonalDepressionPrevention

Download the Book (PDF): Introduction A cup of coffee is a hard thing to buy with Bitcoin. Not because the coffee shop objects, and not because the customer lacks the funds, but because the base Bitcoin network was never designed to settle a four-dollar purchase in the time it takes to hand over a paper cup. A Bitcoin transaction is confirmed when a miner includes it in a block, blocks arrive roughly every ten minutes on average, and a cautious merchant waits for several of them. The space inside each block is fixed, so when many people want to transact at once they bid against each other for it, and the fee for a single payment can climb from a few cents to several dollars, or on the worst days to tens of dollars. A system that is excellent at moving value without trusting anyone is, at the counter, slow and occasionally expensive. The Lightning Network is the most developed answer to that problem. It is a second layer built on top of Bitcoin: a network of two-party contracts, called payment channels, that let people exchange bitcoin many times while touching the blockchain only twice, once to open the channel and once to close it. Between those two moments, payments are signed messages passed directly between the parties. They take effect in fractions of a second, cost a fraction of a cent, and can be as small as a single satoshi, one hundred-millionth of a bitcoin, or even smaller in internal accounting. By linking channels together with a clever piece of conditional contract called a Hashed Timelock Contract, a payment can hop across several intermediaries who neither know nor trust each other, and arrive at a recipient the sender has never dealt with. That summary is accurate and almost useless, because it leaves out everything that makes Lightning interesting and everything that makes it hard. The interesting part is how Lightning keeps Bitcoin's central promise intact. A payment channel is not an IOU and the intermediaries are not banks. At every moment, each party holds a fully signed Bitcoin transaction that would pay them their correct share if they broadcast it, and the protocol is arranged so that cheating is not merely forbidden but financially suicidal. The hard part is that this guarantee comes with conditions that the base layer never imposed. You must be online, or have someone watching on your behalf. Your ability to receive money depends on someone else having committed capital toward you. Payments can fail because no path with enough capacity exists. And every one of those enforcement guarantees ultimately rests on being able to get a transaction into a Bitcoin block in time, which is precisely the resource Lightning was invented to economize. The argument of this book This booklet argues a single idea: the Lightning Network scales Bitcoin by converting the problem of global agreement into a problem of local enforcement, and nearly everything distinctive about Lightning, its speed, its cheapness, its privacy properties, its liquidity headaches and its security caveats, follows from that conversion. On the base layer, every node in the world checks every transaction, and that universal verification is what makes Bitcoin trustworthy and also what makes it scarce. Lightning takes most payments out of that global process. Instead of the whole world agreeing that Alice paid Bob, Alice and Bob agree between themselves, and they each hold the means to drag the matter before the global court, the blockchain, if the other misbehaves. Most of the time nobody goes to court, so most of the time the blockchain is not involved at all. That is where the scaling comes from. But a system of private agreements backed by a public court has very different properties from a system where the court hears every case. It is fast because nobody waits for the court. It is cheap because the court's fees are paid rarely. It is constrained because each private agreement can only move the money that was locked into it. And it is only as secure as your ability to reach the court before a deadline expires. Seen this way, the rough edges of Lightning stop looking like immaturity that will simply be engineered away and start looking like the honest price of a particular design choice. Some of them can be reduced, and a decade of work has reduced several of them a great deal. Others are structural. A reader who understands which is which can judge Lightning claims, from the enthusiastic to the dismissive, on their merits. What the book covers The chapters follow the mechanism from the bottom up and then turn outward to the point of sale. The first chapter sets out the problem Lightning was built to solve: why blockspace is scarce by design, why the Bitcoin community chose not to simply enlarge blocks, and why payments and settlement are different jobs that need not happen on the same layer. The second chapter builds a payment channel from scratch: the funding transaction, the commitment transactions that encode each balance, and the revocation scheme that makes publishing an old balance a way to lose everything. The third chapter introduces the Hashed Timelock Contract, the conditional payment that lets channels be chained into routes, and walks through how timelocks are staggered so that every intermediary is protected, and how onion encryption hides the route from the people forwarding it. The fourth chapter is about liquidity, the concept that does more than any other to separate Lightning from ordinary intuitions about money. A channel's capacity is split between its two ends, and a payment can only travel in a direction where enough balance sits. Routing, pathfinding, multipart payments and the economics of routing fees all follow from that. The fifth chapter examines the security model honestly: the requirement to watch the chain, the danger of fee spikes, and the catalogue of attacks researchers have found, from channel jamming to replacement cycling. The sixth chapter describes how the network is actually run today, by a handful of node implementations and by Lightning Service Providers who supply inbound liquidity to mobile wallets, and what that has done to the network's shape. The seventh chapter goes to the shop counter: invoice formats, Lightning addresses, the reusable offers standard known as BOLT12, and real deployments from El Salvador's beaches to Square's point-of-sale terminals. The eighth chapter looks at the unresolved frontier: the pull toward custodial services, the proposals for sharing a single on-chain output among many users, and the protocol changes that would make all of this cheaper and safer. What the book assumes No prior knowledge of Bitcoin's internals is needed, though readers who already know what a transaction input and output are will move faster. Every term is defined where it first appears. The book does not discuss the price of bitcoin, investment questions, or the politics of Bitcoin beyond what is needed to explain design choices. It is not a guide to setting up a particular wallet or node, because those instructions go stale within months. It is an explanation of the machine, written so that a developer, a payments professional, a regulator or a curious merchant can understand what is actually happening when a Lightning payment goes through, and what it would take for it not to. Lightning is now more than eight years past its first mainnet software releases. It has outlasted predictions that it would never work and predictions that it would conquer payments by the end of the decade. What it has become is something more specific and more instructive than either: a working demonstration of how far bilateral cryptographic contracts can stretch a scarce global ledger, and of exactly where the stretching stops. Chapter 1: Scarce by Design Every Bitcoin full node keeps a complete copy of the ledger and checks every new transaction against it. It verifies that the coins being spent exist, that they have not been spent before, that the signatures authorizing them are valid, and that the transaction obeys every consensus rule. Nobody is asked to take anyone else's word for anything. This is the property that makes Bitcoin unusual among payment systems: there is no institution at the center whose records are authoritative, only a set of rules that tens of thousands of independent computers enforce for themselves. That property has a cost, and the cost scales with use. If each node must process every transaction, then the total throughput of the system is bounded by what an ordinary node can handle, not by what the most powerful participant can handle. Raise the throughput and you raise the hardware, bandwidth and storage needed to run a node, and fewer people will run one. Fewer nodes means more people trusting someone else's node, and the system drifts back toward the trusted intermediaries it was designed to avoid. The scarcity of blockspace is therefore not an accident or an oversight. It is the mechanism that keeps verification cheap enough for anyone to do. How much room there is Bitcoin produces a block roughly every ten minutes. The interval is enforced statistically by the proof-of-work difficulty adjustment, which every 2,016 blocks retunes the mining puzzle so that the average pace stays close to the target. Individual gaps vary widely; a block can follow its predecessor in seconds or take an hour. The size of each block is capped. For Bitcoin's first years the cap was one megabyte. Since the Segregated Witness upgrade activated in August 2017, the limit is expressed as four million weight units, a measure that counts signature data at a discount. In practice this allows somewhat under two megabytes of typical transactions per block, more if transactions are signature-heavy, and up to nearly four megabytes in unusual cases. A simple payment with one input and two outputs occupies a few hundred bytes. Putting those together gives the much-repeated estimate that Bitcoin can process something on the order of seven transactions per second, a figure that varies with transaction shapes but is the right order of magnitude. By comparison, large card networks handle thousands of transactions per second on average and claim capacity for tens of thousands at peak. The gap between those numbers is not a matter of a few years of optimization. It is three orders of magnitude. No adjustment to the base layer that preserves cheap verification can close it. If Bitcoin is to serve as a means of everyday payment for a large population, most of those payments will have to happen somewhere other than in a block. The fee market When demand for blockspace is below supply, transactions confirm cheaply and quickly. When demand exceeds supply, users compete. Each transaction carries a fee, the difference between the value of its inputs and its outputs, which goes to the miner who includes it. Miners maximize revenue by selecting the transactions that pay the most per unit of weight. The result is an auction held every ten minutes, in which the price of inclusion is set by the marginal bidder. That auction has a property that is excellent for a settlement system and terrible for retail payments: fees depend on the size of the transaction in bytes, not on the value moved. Moving a hundred million dollars costs the same as moving a hundred dollars if the two transactions are the same shape. For a large transfer, a fee of even fifty dollars is a rounding error. For a coffee, a fee of fifty cents is a significant surcharge, and a fee of five dollars makes the purchase absurd. Bitcoin users have lived through several episodes in which this became painfully concrete. In December 2017, during a speculative surge, average fees rose above fifty dollars for a short period and many ordinary payments simply stopped making economic sense. In 2021 and again in 2023 and 2024, waves of demand, including the burst of inscription and token activity that followed the introduction of the Ordinals scheme in early 2023, drove fees up for weeks at a time. Between such episodes fees often fall to near the minimum relay level, a small fraction of a dollar. The volatility is itself the problem for a merchant. A shop cannot run a till on a payment method whose transaction cost may rise a hundredfold during a week of excitement on the other side of the world. Confirmation time adds a second difficulty. A payment broadcast to the network is visible within seconds, but until it is mined it can in principle be replaced or double-spent. For a small purchase many merchants historically accepted unconfirmed payments and accepted the risk, but that practice has become less safe over time as the network has moved toward policies that make replacing unconfirmed transactions easier. Waiting for even one confirmation means waiting an average of ten minutes, and sometimes much longer. No queue at a coffee counter tolerates that. The block size war and the choice it settled The obvious response to scarce blockspace is to create more of it. Between roughly 2015 and 2017 the Bitcoin community fought a long and bitter dispute over exactly that question, now usually called the block size war. One camp argued that blocks should be enlarged, by a fixed step or according to a schedule, so that Bitcoin could process more payments directly. Another camp argued that larger blocks would raise the cost of running a node, concentrate validation among large operators, and undermine the very property that gave Bitcoin value, and that throughput should instead come from layers built on top. The dispute was settled less by argument than by what users and node operators chose to run. Segregated Witness, which modestly raised effective capacity and, more importantly, fixed a long-standing flaw known as transaction malleability, activated in August 2017. Groups who wanted much larger blocks split off to form separate currencies, of which Bitcoin Cash is the best known. A later proposal to double the block size, the so-called SegWit2x plan, was abandoned in November 2017 before its scheduled activation. Since then the base layer has kept its capacity roughly constant and has changed in ways that make it a better foundation for other systems, most notably with the Taproot upgrade of November 2021. The relevance of that history here is that the Lightning Network is not an optional enhancement to a system that could have scaled another way. It is the path the Bitcoin community deliberately chose. The decision to keep blockspace scarce was, in effect, a decision that most small payments would move to a second layer, and Lightning was the design ready to receive them. Segregated Witness mattered to Lightning in a second and more technical way. Before it, a third party could alter certain non-essential bytes of a transaction's signature without invalidating it, which changed the transaction's identifier. Any protocol that relied on signing a transaction that spends another transaction before the first was confirmed was exposed to this: if the parent's identifier changed, the pre-signed child became invalid. Lightning channels rely on exactly that pattern. Segregated Witness moved signature data out of the part of the transaction that determines its identifier, which made the pattern safe. Lightning as it exists was not really possible until that change activated. Settlement and payment are different jobs The most useful way to think about Bitcoin's scaling is to separate two jobs that ordinary experience runs together. Settlement is the final, irreversible transfer of ownership, recorded in a way nobody can later dispute. Payment is the act of one person giving value to another at the moment of a transaction. In the conventional financial system these are already separate. When you pay by card, the payment happens in seconds at the terminal, but settlement between banks happens later, often in batches, through central bank systems that handle far fewer, far larger transfers. The fast, cheap layer that consumers see rests on a slow, expensive, highly secure layer they never touch. Bitcoin's base layer is an extraordinary settlement system: open to anyone, final after a few confirmations, and enforced by nobody's permission. It is a poor retail payment system for the reasons described above. Lightning keeps Bitcoin as the settlement layer and builds the payment layer on top of it. The crucial difference from the card system is in how the two layers relate. In the conventional system, the fast layer is a set of promises between trusted institutions, and the slow layer settles those promises. In Lightning, the fast layer is a set of transactions that are already valid on the slow layer and could be broadcast to it at any moment. Each Lightning balance is not a promise to settle later. It is a settlement that has been prepared and signed but not yet published. The contrast between the two layers can be summarized along a handful of dimensions, as Table 1 sets out. Every entry in the right-hand column is a consequence that later chapters will explain. Table 1. Bitcoin's base layer and the Lightning Network compared. Property Base layer (on-chain) Lightning Network Who verifies a payment Every full node Only the parties on the route Time to finality Minutes to an hour Under a second to a few seconds Fee basis Bytes of blockspace Amount forwarded, plus small fixed fee Smallest practical amount Thousands of satoshis One satoshi or less Requirement to be online None to receive Needed, or delegated to a service Capacity constraint Block size Channel balances along a route Public record Every transaction Only channel opens and closes Earlier attempts at payment channels The idea of moving repeated payments off the blockchain is nearly as old as Bitcoin itself. Satoshi Nakamoto discussed a form of high-frequency transaction replacement in early correspondence, and the original software contained fields that hinted at it, though the scheme was not safe as designed. In 2013 Jeremy Spilman described a simple unidirectional payment channel: a customer locks funds in a two-party output and then signs successively larger payments to a service provider, who only needs to broadcast the last one. Because money only ever flows one way, the provider has no reason to publish an old state; every older state pays them less. A refund transaction with a timelock lets the customer recover the funds if the provider disappears. Unidirectional channels work well for a single customer paying a single provider over time, such as a stream of payments for bandwidth or computation. They fail for anything more general. Money cannot flow back, so the channel is used up as it is spent, and there is no way to pay anyone except the provider at the other end. In 2015 two lines of work converged on a general solution. Christian Decker and Roger Wattenhofer published a design for duplex micropayment channels, which allowed bidirectional payment using decrementing timelocks and a tree of invalidation transactions. Joseph Poon and Thaddeus Dryja circulated a draft paper titled "The Bitcoin Lightning Network: Scalable Off-Chain Instant Payments," the full version of which appeared in January 2016. Poon and Dryja's design made two contributions that defined everything that came after. The first was a bidirectional channel in which old states are invalidated not by timelocks running out but by a penalty: each party hands the other a secret that allows them to seize all channel funds if the old state is ever published. The second was the Hashed Timelock Contract, which allows payments to be routed across a chain of channels without any party in the chain being able to steal the payment in transit. The combination turned a pairwise payment tool into a network. With bidirectional channels and conditional routing, a user needs only a handful of channels to reach anyone else connected to the network, just as a computer needs only one link to its internet provider to reach any site in the world. From paper to network The years after the paper were spent turning it into interoperable software. Several independent teams formed: Lightning Labs, which builds the implementation called LND; Blockstream, which builds Core Lightning, originally called c-lightning; and ACINQ, which builds Eclair. Rather than let a single codebase define the protocol, the teams agreed on a set of written specifications called the Basis of Lightning Technology, universally abbreviated as BOLTs. Each BOLT covers one layer of the protocol: the message framing and encryption between peers, the channel establishment and closing procedures, the format of transactions, the onion routing scheme, the gossip protocol by which nodes learn about channels, and the invoice format for requesting payment. Any implementation that follows the BOLTs can open a channel with any other and route payments through it. Interoperability testing among the implementations took place through 2017, and the first versions intended for real money on Bitcoin's main network appeared in early 2018, accompanied by stern warnings about the risk of loss. The network grew from a few hundred nodes to thousands within its first year. Early users were explicitly experimenters. A cap on channel size, about 0.168 bitcoin, deliberately limited the damage bugs could do. That cap was relaxed in 2020, when the implementations added support for larger channels, colloquially called wumbo channels. Today Lightning is a mature protocol in the sense that its core mechanisms are well understood and have moved real money for years, and an immature one in the sense that substantial pieces of its user experience, liquidity management and security posture are still being reworked. The remaining chapters take the mechanism apart in order, beginning with the smallest unit of the network: a single channel between two people who do not trust each other. The alternatives Lightning competes with Lightning is not the only proposal for moving Bitcoin activity off the base layer, and its design is easier to understand against the alternatives. The simplest is custody. An exchange or payment company can hold bitcoin for millions of customers and record transfers between them in its own database, with no blockchain involvement at all. This scales without limit and is how a large share of bitcoin ownership has always worked. It also abandons Bitcoin's central property: the customer owns a claim on the company, not bitcoin, and the company can freeze, lose or misreport it. A second approach is the federated sidechain. Blockstream's Liquid Network, launched in 2018, is a separate blockchain whose blocks are produced by a federation of known member organizations, and whose bitcoin is backed by coins held in a multisignature arrangement controlled by that federation. Liquid offers faster blocks and confidential amounts, and it is used for trading and for issuing other assets. Its users must trust that a majority of the federation will not collude. A third family, often called rollups or validity-proof systems, has flourished on other blockchains. Activity is batched off-chain and a compact proof of its correctness is posted on-chain. Bitcoin's current scripting language cannot verify such proofs directly, so Bitcoin rollup designs either await changes to the consensus rules or rely on more complex constructions with additional trust or interaction assumptions. Lightning occupies a distinctive place among these. It does not ask users to trust a custodian or a federation, and it needs no change to Bitcoin's rules. Each user's balance is enforceable on the base layer by that user alone. The price of that independence is the set of constraints this book examines: the need for liquidity, the need to be watching, and the dependence on timely access to blockspace. Whether those constraints are worth bearing depends on how much a given user values not trusting anyone, which is exactly the question Bitcoin itself poses. Why the base layer still matters It would be a mistake to read the separation of settlement and payment as making the base layer less important to Lightning users. The opposite is true. Every Lightning balance is ultimately a claim that can only be enforced by getting a transaction confirmed on the base layer. When a channel is opened, it requires a confirmed transaction. When a channel is closed, the same. When something goes wrong, when a counterparty disappears or tries to cheat, the honest party's remedy is a transaction that must reach a block before a deadline. So Lightning does not escape blockspace scarcity. It amortizes it. A user who opens a channel, makes three hundred payments over a year, and closes it has paid for two on-chain transactions instead of three hundred. If fees spike while that channel is open, the payments are unaffected. But if the user needs to close the channel during a fee spike, or needs to enforce a claim against a cheating counterparty, the full weight of the fee market comes back. Scarce blockspace is what makes Lightning necessary, and also what bounds how safely and how widely it can be used. That tension runs through every chapter that follows. Chapter 2: Building a Channel A payment channel is a pair of people, a pot of bitcoin they have locked together, and a running agreement about how that pot would be divided if they stopped now. Everything about Lightning's security comes from how that agreement is written. It must be possible to change the agreement thousands of times without telling the blockchain. Each version must be enforceable by either party acting alone. And once a version has been superseded, publishing it must be worse than useless for whoever tries. This chapter builds such a channel step by step, using two parties called Alice and Bob, who have no reason to trust each other. The funding transaction A channel begins with an on-chain transaction that moves bitcoin into an output that can only be spent with two signatures, one from Alice and one from Bob. This is a two-of-two multisignature output, and the transaction that creates it is called the funding transaction. Once it confirms, the coins in that output belong to neither party alone. Neither can move them without the other's cooperation. That creates an obvious danger. Suppose Alice funds the channel with one million satoshis and Bob then vanishes. The money is locked in an output that needs Bob's signature, and Bob is gone. Alice would never put money into such an output unless she already held a way to get it back. So before Alice signs and broadcasts the funding transaction, she and Bob construct and sign the first commitment transaction: a transaction that spends the funding output and pays Alice her one million satoshis. Bob signs it and gives the signature to Alice. Only once Alice holds that signed refund does she broadcast the funding transaction. This ordering is the reason malleability mattered so much. The commitment transaction spends the funding output by referring to the funding transaction's identifier. If a third party could alter the funding transaction's identifier between signing and confirmation, the pre-signed commitment would point at nothing and Alice's refund would evaporate. With Segregated Witness, the identifier depends only on the parts of the transaction that cannot be tampered with, and the refund is safe. In the original protocol, only one party funds the channel. The funder chooses how much to commit, and the other party starts with a zero balance. Later extensions, called dual funding, let both parties contribute in the same funding transaction, and the interactive transaction construction protocol that supports this is now used by several implementations. The single-funder case remains the easiest to understand and is still very common. Commitment transactions A commitment transaction is a fully signed Bitcoin transaction that spends the funding output and divides it between the two parties according to the current balance. At the start, Alice's commitment pays all the funds to her. Suppose Alice now pays Bob 200,000 satoshis. The two of them construct a new commitment transaction that pays 800,000 to Alice and 200,000 to Bob, and exchange signatures on it. Nothing goes on the blockchain. The payment is complete the moment both parties hold the signed new state. Either party could broadcast their copy of the current commitment at any time, and the blockchain would accept it, because it carries both required signatures. That is the meaning of saying that a Lightning balance is a settlement prepared but not yet published. Alice does not need Bob's permission to leave. She needs only to broadcast. Here the problem becomes clear. After the payment, Alice still holds the old commitment that paid her the full million. It is just as validly signed as the new one. If she broadcast it, the blockchain would have no way of knowing it was out of date, because the blockchain never saw the update. There is no sequence number the consensus rules check, no central record of which version is current. Bitcoin will confirm whichever spending transaction it receives first. A channel that allowed this would be worthless: every payment could be undone by publishing an earlier state. The entire ingenuity of the Poon-Dryja design lies in how it makes old commitments dangerous to publish. Asymmetric commitments and the revocation key The first trick is that Alice and Bob do not hold the same commitment transaction. Each holds their own version, and the two versions are mirror images. Alice's version, the one she would broadcast, pays Bob his share immediately. It pays Alice her share only through a more complicated output: Alice can claim the money, but only after a delay, measured in blocks, called the to_self_delay. During that delay there is a second way to spend the output, which requires a key called the revocation key. Bob's version is the reverse: it pays Alice immediately and pays Bob through a delayed output that Alice could seize with the corresponding revocation key. The second trick is how the revocation key is built. It is derived from two pieces, one contributed by each party, such that the full private key can only be computed by someone who knows a secret held by the other side. When Alice and Bob move from one state to the next, each reveals to the other the secret for the state they are abandoning. After Alice and Bob agree on the 800,000/200,000 state, Alice hands Bob the revocation secret for her old commitment that paid her a full million. From that moment Bob can compute the private revocation key for Alice's old commitment. Now consider what happens if Alice tries to cheat by broadcasting that old commitment. It confirms on the blockchain. Bob's share, which in that old state was zero, is paid to him directly. Alice's share, the full million, sits in the delayed output, which she cannot touch until the to_self_delay expires. During that window, Bob notices the old state on chain, uses the revocation key he now holds, and sweeps the entire million to himself. Alice does not merely fail to steal the 200,000 she owed Bob; she loses her own 800,000 as well. This transaction is called the penalty or justice transaction, and the scheme as a whole is sometimes called LN-penalty. It is worth walking through the timing of such an attempt. Suppose Bob has set Alice's to_self_delay at 144 blocks, about a day. At block 900,000 Alice broadcasts her revoked commitment, and it confirms in block 900,001. From that block, Alice's delayed output cannot be spent by her until block 900,145. Bob's node, which checks every new block for transactions spending any of its funding outputs, sees the funding output spent in block 900,001. It compares the spending transaction against its records, recognizes a commitment number that has been revoked, derives the revocation private key from the stored secret, and broadcasts a justice transaction that spends Alice's delayed output to Bob, with a fee high enough to confirm quickly. If that transaction confirms anywhere in the next 143 blocks, Alice's attempt has cost her everything. Bob has a whole day of block time in which to get a single transaction mined, which under normal conditions is a wide margin. The margin narrows only if Bob is offline for most of that day or if blockspace is so congested that his fee cannot win a place, and those two conditions are where the security analysis of later chapters concentrates. The asymmetry matters because it ensures each party can only penalize the other's cheating. When Bob broadcasts his own current commitment, his money is delayed and Alice's is immediate, so the delay applies only to the party who chose to publish. An honest party who publishes the current state merely waits for the delay before spending their funds. A dishonest party who publishes an old state gives the other side a window to take everything. The delay is a design parameter with a real trade-off. A longer delay gives the victim of a cheating attempt more time to notice and respond, which is safer. But it also means that anyone who closes a channel unilaterally, even honestly, waits longer to use their own money. Implementations commonly choose delays from about a day to two weeks of blocks, often scaled with channel size, and each party sets the delay that applies to the other. Storing the secrets A busy channel may go through millions of states. If each state required storing a separate revocation secret, the storage requirements for each party would grow without bound, and a phone wallet would eventually fill up. The BOLT specifications solve this with a structure sometimes called shachain. Revocation secrets are generated from a single seed in a tree-shaped pattern, chosen so that each new secret revealed to the counterparty lets them derive many earlier ones. The receiving party needs to store at most about forty-nine secrets, regardless of how many states the channel has passed through, and can reconstruct any revoked secret on demand. What cannot be compressed is the need to remember that cheating happened. To punish a stale commitment, Bob must recognize it when it appears on chain and construct the correct penalty transaction. That requires keeping enough information about every past state to identify it and sweep it. The data per state is small, but it accumulates. It is also the reason a Lightning node's backup is a delicate matter, a subject the fifth chapter returns to, because restoring a node from an old backup can make an honest user look exactly like a cheater. How a state update actually happens In practice, updates are a choreographed exchange of messages. When Alice wants to pay Bob, she sends an update_add_htlc message proposing the change, followed by a commitment_signed message containing her signature on Bob's new commitment. Bob responds with revoke_and_ack, which reveals his revocation secret for his previous commitment and acknowledges the new one, then sends his own commitment_signed with his signature on Alice's new commitment. Alice replies with her own revoke_and_ack. At the end of this dance, each party holds a signed copy of their own new commitment, and each holds the revocation secret for the other's previous one. The order is chosen so that at no point does either party give up their ability to enforce a valid state. Nobody revokes an old commitment until they hold a signed new one. If the connection drops halfway, the protocol includes a reestablishment procedure in which the two nodes compare what each has signed and revoked and resume from a consistent point. All of this happens over an encrypted, authenticated connection between the two nodes, using a handshake based on the Noise protocol framework, specified in BOLT 8. The payment completes as fast as the messages can cross the network: typically well under a second between two well-connected nodes. Fees inside the channel A commitment transaction is a real Bitcoin transaction that may one day need to be confirmed, so it must carry a fee. This created one of the most persistent practical difficulties in early Lightning. The fee had to be agreed when the commitment was signed, but the commitment might not be broadcast until weeks later, by which time the fee market could have changed completely. If fees rose sharply, a pre-signed commitment with a low fee might sit unconfirmed while a deadline ran out. The parties could periodically update the fee with update_fee messages, but disagreements over fee estimates were a common cause of channels being closed unexpectedly. The modern answer is anchor outputs. A commitment transaction built with anchors carries a minimal fee and includes two tiny extra outputs, one for each party, each worth 330 satoshis. Either party can spend their anchor output in a new transaction that pays a high fee, and miners, who evaluate a parent and child transaction together, will confirm both. This technique, called child-pays-for-parent, lets the party who needs the commitment confirmed set the fee at the moment of broadcast rather than weeks in advance. Anchor channels became the default in the major implementations around 2021 and 2022. They do impose a new requirement: the party closing the channel must hold some separate on-chain bitcoin to pay the fee for the child transaction. A wallet with all its money inside channels has no way to bump a commitment's fee. Work on Bitcoin's transaction relay policy has continued to refine this. Newer rules adopted in Bitcoin Core during 2024 and 2025, including package relay for one-parent-one-child pairs and a restricted transaction type known as version 3 or TRUC transactions, are designed to make commitment transactions with zero fee reliably relayable and fee-bumpable. Lightning implementations have been adapting their commitment formats to take advantage of them. Closing the channel There are three ways a channel ends, and they differ sharply in cost, speed and risk, as Table 2 summarizes. Table 2. The three ways a Lightning channel closes. Close type How it happens Wait for funds Typical cost Main risk Cooperative (mutual) Both parties sign a new closing transaction One confirmation One simple transaction None beyond fee level Unilateral (force close) One party broadcasts their latest commitment Delay of up to weeks for the closer Commitment plus any pending payments, fee bumps Fee spikes, pending payments to resolve Breach (penalty) One party broadcasts a revoked commitment; the other sweeps it Victim sweeps within the delay Penalty transaction Victim must notice in time The cooperative close is the normal ending. Alice and Bob negotiate a fee and sign a simple transaction that spends the funding output and pays each their current balance, with no timelocks and no revocation paths. Once it confirms, both have their money. It looks on chain like an ordinary two-of-two spend, and with Taproot-based channels it can look like an ordinary single-signature payment. The unilateral close, universally called a force close, is what happens when cooperation is impossible: the counterparty is offline, unresponsive, or in dispute. One party broadcasts their latest commitment transaction. The other party gets their balance immediately; the closer waits out the delay. If payments were in flight at the moment of closing, each becomes its own output on chain that must be resolved separately, which adds transactions, fees and complexity. Force closes are the most common source of user frustration with Lightning, because they are slow, cost more than a cooperative close, and can happen without the user doing anything wrong, for instance when a peer's software crashes or two nodes disagree about fees. The breach is the scenario the whole revocation machinery exists to deter. Its practical frequency is very low, because the penalty makes it irrational, but its possibility shapes everything. The protocol can only deter cheating if the victim is watching. What a channel guarantees and what it does not At this point we can state precisely what a payment channel provides. Each party can, at any time and without the other's cooperation, recover their current balance from the channel, subject to waiting out a delay if they are the one closing. Neither party can steal from the other by publishing an outdated state, provided the victim or someone acting for them sees the attempt and responds before the delay expires, and provided the response can be confirmed on chain in time. Payments between the two parties are final as soon as the revocation exchange completes, without any blockchain involvement. What a channel does not provide is equally important. It cannot move more money than was locked into it. If Alice funded a channel with one million satoshis and has paid Bob all of it, she cannot pay him more over that channel until he pays some back. It cannot pay anyone other than the counterparty. And its guarantees are conditional on the party being online or represented, and on blockspace being available when it is needed. Those conditions are the price of trading global verification for local enforcement. The next chapter introduces the mechanism that removes the second limitation, the inability to pay anyone else, and in doing so turns isolated channels into a network. Chapter 3: Hashed Timelock Contracts A single channel lets Alice pay Bob. A network lets Alice pay Carol, whom she has never met, through Bob, whom she may not trust either. The obvious way to do this is dangerous. Alice could pay Bob and ask him to pay Carol. But then Bob might simply keep the money. Or Alice might ask Bob to pay Carol first and promise to repay him, in which case Alice might keep the money. Any routing scheme built on sequential promises reintroduces exactly the trust that Bitcoin was built to remove. The Hashed Timelock Contract, or HTLC, solves this with a conditional payment that links the hops of a route together, so that either every hop is paid or none is, and no intermediary can take the money and run. It is the single most important construction in Lightning, and most of the network's behavior, good and bad, flows from its details. Two primitives An HTLC combines two capabilities that Bitcoin's scripting language has supported for years. The first is a hashlock. A cryptographic hash function, SHA-256 in Lightning's case, takes any input and produces a fixed-size output from which the input cannot feasibly be recovered. If Carol picks a random secret, called the preimage, and publishes its hash, then anyone can check whether a proposed value is the preimage by hashing it and comparing, but nobody can work out the preimage from the hash. A Bitcoin output can be locked so that it can only be spent by someone who reveals the preimage of a given hash. Revealing the secret is the key. The second is a timelock. Bitcoin can lock an output so that a certain path for spending it only becomes valid after a given block height or after a given number of blocks have passed. Lightning HTLCs use absolute timelocks, enforced by the OP_CHECKLOCKTIMEVERIFY operation, which specify a block height before which a refund path cannot be used. An HTLC is an output with two spending paths. The recipient can claim it at any time by revealing the preimage. The sender can reclaim it after a specified block height if the recipient has not done so. In words, it says: this money goes to you if you can show me the secret before block N; after block N, it comes back to me. Chaining the condition across a route Suppose Alice wants to pay Carol 50,000 satoshis, and the route runs from Alice to Bob to Carol. Carol begins by generating a random preimage and sending its hash to Alice as part of a payment request, an invoice. Alice now knows the hash but not the preimage. Alice adds an HTLC to her channel with Bob: Bob can claim 50,010 satoshis if he reveals the preimage of that hash before block height H plus 80; otherwise the money returns to Alice. The extra ten satoshis are Bob's fee for forwarding. Bob does not know the preimage either, so he cannot claim it yet. But he now has a reason to find it. Bob adds an HTLC to his channel with Carol: Carol can claim 50,000 satoshis if she reveals the same preimage before block height H plus 40; otherwise the money returns to Bob. Carol knows the preimage, because she created it. She reveals it to Bob, and Bob's HTLC to her settles: the 50,000 satoshis become part of her balance in their channel. Bob now knows the preimage. He reveals it to Alice, and Alice's HTLC to him settles: 50,010 satoshis move to his side of their channel. Bob is up ten satoshis, Carol is up 50,000, and Alice is down 50,010. Consider what each party could do to cheat, and why it fails. Bob cannot claim Alice's HTLC without the preimage, and he can only get the preimage from Carol, who reveals it only when claiming Bob's HTLC to her. So Bob can only be paid by Alice after he has paid Carol. If Bob refuses to forward, Carol never reveals the preimage, and both HTLCs eventually expire and refund. Alice loses nothing but time. If Carol reveals the preimage to claim Bob's payment but then Bob and Alice's channel connection drops, Bob still holds the preimage and can enforce his claim on Alice's HTLC on chain if necessary. Nobody can take money without passing it along. The preimage also serves as a receipt. Alice learns it only when the payment completes, and only Carol could have generated it. Possession of the preimage is cryptographic proof that Carol was paid, which matters for merchants and for dispute resolution. Why the timelocks must be staggered The heights in the example were not chosen arbitrarily. Alice's HTLC to Bob expires later than Bob's HTLC to Carol. This staggering protects the intermediary. Imagine the expiries were the other way round, with Alice's HTLC to Bob expiring first. Carol could wait until just after Alice's HTLC expired and Alice had reclaimed her money, then reveal the preimage and claim Bob's HTLC. Bob would have paid Carol but would have no way to collect from Alice, because Alice's HTLC had already timed out. Bob would lose the full amount. To prevent this, each hop must give the next hop a deadline that leaves the forwarding node enough time to learn the preimage and still enforce its own incoming claim, on chain if necessary. The gap between a node's incoming expiry and its outgoing expiry is called the CLTV delta, after the timelock operation. Each node advertises its own delta, and senders build routes in which the expiry steps down by at least that amount at each hop. Deltas of a few dozen to over a hundred blocks are typical, chosen to give a node time to notice a problem, broadcast transactions and get them confirmed even during congestion. The staggering for a longer route can be laid out hop by hop. Table 3 shows a four-hop payment from Alice to Erin, with illustrative deltas of 40 blocks per intermediary, current block height 800,000, and Erin requiring a final expiry at least 18 blocks in the future. Table 3. Staggered HTLC expiries on a four-hop route (illustrative values). Channel Amount offered HTLC expires at block Delta protecting the forwarder Alice to Bob 50,030 sat 800,138 40 blocks for Bob Bob to Carol 50,020 sat 800,098 40 blocks for Carol Carol to Dave 50,010 sat 800,058 40 blocks for Dave Dave to Erin 50,000 sat 800,018 Final hop Two consequences follow. First, the sender's funds may be locked for the longest time: if something goes wrong anywhere on the route, Alice's HTLC might not refund until block 800,138, nearly a day away at ten minutes per block. Longer routes and larger deltas mean longer worst-case lockups. Implementations therefore cap the total timelock a route may require, commonly at around two weeks of blocks. Second, the total delta is a cost the network pays for every payment in reserve, even though almost all payments settle in seconds and never approach their deadlines. What an HTLC looks like inside a channel So far HTLCs have been described as if they were on-chain outputs, and in a sense they are. When Alice adds an HTLC to her channel with Bob, she and Bob sign new commitment transactions that include an additional output: the HTLC itself, holding the 50,010 satoshis, with its hashlock and timelock paths. Alice's balance output shrinks by that amount. If either party force-closed the channel while the HTLC was pending, the HTLC output would appear on chain and be resolved there: Bob claims it with the preimage, or Alice reclaims it after the timeout. When the HTLC settles off-chain, by Bob sending an update_fulfill_htlc message with the preimage, the two parties sign yet another pair of commitments in which the HTLC output is gone and Bob's balance has grown. When it fails, by Bob sending update_fail_htlc, the new commitments remove the HTLC and restore Alice's balance. In both cases the revocation procedure from the previous chapter applies: the old commitments containing the HTLC are revoked. There is an additional layer of complexity in how HTLC outputs are spent on chain, because the revocation guarantees must extend to them. If Alice cheated by broadcasting an old commitment that included an HTLC, Bob must be able to penalize that too. And if an honest party claims an HTLC on chain from their own commitment, they must be subject to the same to_self_delay as their main balance, to give the counterparty a chance to punish them if the commitment was revoked. The specification achieves this with second-stage transactions, called HTLC-success and HTLC-timeout transactions, that are pre-signed by both parties and pay into delayed, revocable outputs. These are among the most intricate parts of the protocol and a major reason Lightning implementations are hard to get right. Each HTLC also carries an on-chain cost if the channel is force-closed, because it becomes an output that must be claimed by its own transaction. For very small payments that cost exceeds the payment's value. Lightning handles such payments, called dust HTLCs, by not creating outputs for them at all: they are tracked in the channel's accounting but, if the channel closes while they are pending, their value goes to miners as fees. This is the honest limit of Lightning's microtransaction promise. A one-satoshi payment works perfectly off-chain, but it is not enforceable on chain, and its security in flight depends on the fact that nobody would force-close a channel to steal a few satoshis. Channels also limit the total dust exposure they will accept, because an attacker could otherwise stuff a channel with dust HTLCs worth a meaningful sum in aggregate. A channel can hold at most 483 pending HTLCs in each direction, a limit derived from the maximum size of a valid commitment transaction. This number turns out to matter a great deal for security, as the fifth chapter shows. Onion routing In the simple example, Bob knew he was forwarding a payment to Carol. A naive design would have Alice tell every node on the route the full route, which would leak who is paying whom to everyone involved. Lightning avoids this with source routing wrapped in onion encryption, using a scheme called Sphinx, originally designed by George Danezis and Ian Goldberg in 2009 for anonymous messaging and adapted for Lightning in BOLT 4. The sender chooses the entire route. She then builds a packet in layers, like an onion. The outermost layer is encrypted so that only the first hop can read it, and it contains just what that hop needs: the next channel to forward to, the amount to forward and the expiry to use. Inside that is a layer readable only by the second hop, and so on, down to the final layer for the recipient. Each hop peels off its layer, learns its instructions, and passes the rest along. Because the packet is padded to a fixed size of 1,300 bytes regardless of route length, and because each hop re-randomizes the packet as it peels it, a forwarding node cannot tell how many hops remain or how many came before. Bob knows that the payment came from Alice and went to Carol, but not whether Alice was the original sender or Carol the final recipient. Errors travel backward along the same path, encrypted so that only the sender can read which hop failed and why. This is what allows the sender to learn from failures and try a different route. Onion routing gives Lightning real privacy advantages over on-chain payments, which are permanently public. It is not perfect. An intermediary that is adjacent to the sender or recipient can often guess their role from timing, amounts and the structure of the network. The same payment hash appears at every hop, so an entity controlling two nodes on the same route can recognize the payment at both points and link them. Researchers have documented a related attack, the wormhole attack, in which two colluding nodes on a route skip the honest nodes between them and collect their fees. These weaknesses are a principal motivation for the replacement of hashes with a different construction, discussed in the final chapter. Other uses of the same contract The HTLC was designed for routing, but its structure, money released by a secret before a deadline and refunded after it, turned out to be a general-purpose tool, and several important Lightning features are simply HTLCs used in new ways. A hold invoice is an invoice for which the recipient deliberately delays releasing the preimage. The payment arrives and the HTLCs are locked along the route, but the recipient neither settles nor fails them until some condition is met. A merchant might hold a payment until she confirms that an item is in stock, then settle; if it is not, she fails the HTLCs and the money returns to the customer without ever having moved. The same structure supports simple escrow, deposits and bets resolved within the HTLC's timelock. Its cost is that every channel along the route has funds frozen for the duration, which is acceptable for a few minutes and corrosive for days, and which links hold invoices to the jamming problem discussed later. A spontaneous payment, sometimes called keysend, inverts the usual flow. Instead of the recipient generating the preimage and sending its hash in an invoice, the sender generates the preimage and places it, encrypted, in the final layer of the onion. The recipient decrypts it and uses it to claim the payment. This allows payment without any prior request, which is useful for tips, streaming payments and machine-to-machine transfers, at the cost of losing the proof of payment, since the sender knew the preimage all along. A submarine swap, mentioned again in the next chapter, uses one hash to lock both an on-chain output and a Lightning HTLC. Whoever reveals the preimage to claim one side necessarily reveals it to the other party, who can then claim the other side. This lets bitcoin move between Lightning and the base layer without either party trusting the other, by the same logic that lets a route of strangers carry a payment. Atomic swaps between different blockchains, which predate Lightning as an idea, rest on the identical principle. What these uses share is the property that makes HTLCs suitable for routing in the first place: the release of a single secret binds several otherwise independent transfers together, so that either all of them happen or none does. The atomicity guarantee, stated carefully It is common to say that Lightning payments are atomic: either they complete fully or not at all. That is true in the sense that no honest party ends up paying without being paid. It is not true in the stronger sense that the payment either completes instantly or fails instantly. An HTLC that has been forwarded but not resolved is in limbo, and the funds in it are locked until it either settles or reaches its expiry. A recipient or intermediary that holds an HTLC without settling or failing it, whether through malice, a crash or a deliberate design called a hold invoice, can keep those funds locked for up to the full timelock. Most of the time this never matters, because payments resolve in seconds. But the guarantee is precise: Lightning ensures that money cannot be stolen in transit, at the cost of allowing it to be frozen in transit. That trade is at the root of the liquidity and jamming problems examined in the chapters to come. For now, the key insight is that HTLCs allow a payment to cross a network of strangers with each step enforced by the same blockchain that backs each channel, so that a route of untrusted hops carries the same safety as a direct channel, extended by a staircase of deadlines. Chapter 4: Liquidity and Routing Ordinary intuitions about money break down inside Lightning in one specific place. In a bank account, having money means being able to spend it and being able to receive more without limit. In a Lightning channel, having money on your side means you can spend it, but your ability to receive depends on how much money sits on the other side. A channel is not a balance; it is a fixed quantity of bitcoin split between two ends, and every payment slides some of it from one end to the other. Understanding that single fact explains most of what is difficult about running Lightning in practice, and most of the industry that has grown up around it. The bead on a wire A helpful way to picture a channel is as a wire with beads on it. The channel's capacity is the total number of beads, fixed by the funding transaction. Each bead sits at one end or the other. When Alice pays Bob, she slides beads from her end to his. She can slide as many as are on her end and no more. Bob can then slide beads back to her, up to the number on his end. The beads on Alice's side are her outbound liquidity: what she can send through this channel. The beads on Bob's side, seen from Alice's perspective, are her inbound liquidity: what she can receive through this channel. When Alice opens a channel to Bob with one million satoshis, she has one million of outbound liquidity and zero inbound. She can pay through Bob, but nobody can pay her through this channel until she has spent some of it. This is the inbound liquidity problem, and it is the first practical obstacle any Lightning user meets. A merchant who opens a few channels with her own money can spend but cannot be paid. To receive, she needs channels where the other side holds funds, which means someone else has to commit capital toward her. That capital is locked up and could be earning a return elsewhere, so it is not free. Much of the commercial structure of Lightning, discussed in the sixth chapter, exists to sell inbound liquidity to people who need it. The constraint applies to every hop of a route, not just the first and last. For a payment of 50,000 satoshis to travel from Alice through Bob and Carol to Dave, Alice must have at least 50,000 plus fees on her side of the Alice-Bob channel, Bob must have at least that much on his side of the Bob-Carol channel, and Carol on her side of the Carol-Dave channel. The total capacity of a channel tells you its maximum possible use; only the distribution of balance tells you whether a payment can pass in a particular direction right now. Channel reserves tighten this slightly. Each party is normally required to keep a small fraction of the channel, by convention about one percent, on their own side at all times. The reason is that the penalty mechanism only deters cheating if the cheater has something to lose; a party with a zero balance could broadcast an old state with impunity. Reserves mean that a channel can never be fully drained in either direction. What the network knows To route a payment, the sender must choose a path. To choose a path, she needs a map. Lightning nodes build that map from a gossip protocol, defined in BOLT 7, through which nodes broadcast three kinds of announcement. A channel_announcement proves that a public channel exists, by referring to its funding output on chain and carrying signatures from both nodes and their funding keys. A node_announcement gives a node's alias, network addresses and supported features. A channel_update, sent separately by each end, gives that node's routing policy for its direction of the channel: its fees, its CLTV delta, and the minimum and maximum HTLC sizes it will forward. What gossip deliberately does not reveal is the current balance in each channel. Every node knows the total capacity of every public channel, because the funding output is on chain, but not how that capacity is split. Publishing balances would be a severe privacy leak, because every payment would be visible as a shift in the public map, and it would be impractical, because balances change constantly. So the sender routes with a map that shows every road and its width but not how much traffic is on it. The public graph has in recent years held on the order of ten to fifteen thousand reachable nodes and several tens of thousands of public channels, with total public capacity of several thousand bitcoin; the exact figures fluctuate and depend on who is counting and how. A large and growing share of real channels, however, are unannounced: private channels between a mobile wallet and its service provider that never appear in gossip. A recipient behind such a channel tells the sender how to reach her by including routing hints in her invoice, naming the last hop. Pathfinding as guessing Because balances are hidden, pathfinding in Lightning is an exercise in educated guessing. The sender looks at the graph, finds candidate routes from herself to the recipient, and ranks them by fee, total timelock, and estimated probability of success. She then attempts the best one. If an intermediate channel lacks the balance to forward, the node at that hop returns an error, which travels back through the onion to the sender. The sender records what she learned, that a particular channel could not carry this amount in this direction at this moment, and tries another route. A single payment may go through several such attempts. Each failed attempt locks funds in HTLCs only briefly, because the failure is reported quickly, but each attempt adds latency. A payment that succeeds on the first try may take well under a second; one that requires a dozen attempts may take several seconds or fail entirely. Implementations have become much better at this over time, mostly by learning to estimate probabilities more intelligently. A crucial insight, developed in work by René Pickhardt and Stefan Richter published in 2021, is that the probability a channel can forward an amount falls as the amount approaches the channel's capacity. If balances are unknown and assumed roughly uniformly distributed, a payment of ten percent of a channel's capacity is very likely to fit, while a payment of ninety percent is not. Treating routing as a problem of maximizing the joint probability of success, rather than minimizing fees, and splitting payments to keep each part small relative to channel capacities, produces substantially higher success rates. Variants of this probabilistic approach are now used in the major implementations, and nodes keep a memory of recent successes and failures, sometimes called mission control, to refine their estimates of where liquidity lies. A payment, attempt by attempt It helps to follow a single payment through the process. Suppose Maria, using a self-run node with three channels, wants to pay a shop 400,000 satoshis. The shop's node is three hops away in the public graph. Maria's node finds a candidate route through a large exchange node and a regional payment processor. The cheapest route uses a channel between the exchange and the processor with a capacity of one million satoshis, so the requested amount is forty percent of capacity: plausible but far from certain to fit. Maria's node sends the first attempt. The exchange node forwards it, but the processor's side of the channel holds only 250,000 satoshis at that moment, so the exchange node cannot pass the HTLC on. It fails the HTLC back to Maria with an error saying, in effect, that the channel lacked capacity. The whole attempt has taken perhaps half a second. Maria's node now knows something it did not know before: that the exchange-to-processor channel cannot carry 400,000 satoshis in that direction right now, which also means it probably cannot carry anything larger. It records an upper bound on that channel's available balance. The second attempt uses a different middle hop, a route through a routing node that charges slightly higher fees but has two channels of five million satoshis each toward the shop's side. Four hundred thousand is eight percent of those capacities, so the estimated chance of success is much higher. The HTLCs lock in sequence, the shop's node receives the payment and reveals the preimage, and the settlement messages cascade back. Total elapsed time from first attempt to success is a little over a second. Maria paid a few satoshis more in fees than the cheapest route would have cost, and she will never notice. Had no single route been able to carry the full amount, the next step would have been to split it. Multipart payments The probability argument leads directly to one of the most important upgrades in Lightning's history: the ability to split a single payment across several routes. With multipart payments, a sender who needs to deliver 500,000 satoshis can send it as five parts of 100,000 along different paths. Each part is a separate HTLC chain, and the recipient waits until all the parts have arrived before revealing the preimage and settling them together. Basic multipart payments, standardized and widely deployed around 2020, have a structural weakness: all parts use the same payment hash, so the recipient could in principle claim one part and not others, and any observer who sees two parts can link them. A more robust design, often called atomic multipath, derives a different hash for each part from shares of a secret, so the recipient can only claim any part once she has received all of them. Its full benefits depend on cryptographic changes discussed in the final chapter. Multipart payments change the arithmetic of liquidity profoundly. Without them, a payment can succeed only if a single path exists where every hop has enough balance for the whole amount. With them, a payment can succeed if the total outbound liquidity across several paths is enough, even if no single path suffices. They also let a user spend from several of her own channels at once, so her spendable balance is closer to the sum of her outbound liquidity rather than the largest single channel. The price of routing Each routing node sets a fee for forwarding through each of its channels. The fee has two components: a base fee, a fixed amount per forwarded HTLC expressed in millisatoshis, and a proportional fee rate expressed in parts per million of the amount forwarded. A channel with a base fee of one satoshi and a rate of 100 parts per million charges 1 plus 100 satoshis on a one-million-satoshi forward. Many node operators set the base fee to zero, partly because a fee that is purely proportional makes pathfinding mathematically simpler for senders. Typical rates on the network range from essentially free to several hundred or even a few thousand parts per million on channels where demand is high. For a retail payment of a few dollars across two or three hops, total routing fees are commonly a fraction of a cent to a few cents, which is the basis of the claim that Lightning payments are near fee-less. That claim is true for small payments across well-connected routes. It is less true for large payments, where proportional fees add up, and it ignores the on-chain costs of opening and closing channels, which someone, somewhere in the system, must pay. Routing fees have an economic role beyond compensating forwarders. They are the price signal by which the network allocates scarce liquidity. A channel that is constantly being drained in one direction is in demand in that direction; raising its fee in that direction slows the drain and earns more from the traffic that remains. A channel with balance sitting idle can lower fees to attract traffic. Some implementations allow negative or discounted inbound fees, introduced in recent years, so that a node can encourage payments to flow into a particular channel. Sophisticated node operators adjust fees continuously, and a body of tooling exists to automate it. Whether routing is a profitable business is a matter of real debate among operators. The revenue is small per payment, the capital requirement is large, and capital locked in channels carries an opportunity cost. Operators who earn meaningfully tend to be those who place liquidity where flows genuinely need it, for example between large exchanges, payment processors and the providers that serve mobile wallets. Running a routing node is closer to running a small, specialized market-making operation than to operating passive infrastructure. Channels that drift Payment flows are rarely balanced. A merchant receives far more than she sends. A user who receives salary through an exchange spends it outward at shops. Over time, channels drift toward one end, and a drained channel is useless in the direction it has been drained. The process of restoring balance is called rebalancing, and there are several ways to do it. The simplest is a circular rebalance: a node pays itself, sending a payment out through a channel where it has too much outbound liquidity and receiving it back through a channel where it has too little. Nothing changes in its total holdings except the routing fees paid, but liquidity has shifted from one channel to another. Circular rebalancing works only if a suitable circle exists and the fees are worth paying. A second approach uses the blockchain. A submarine swap exchanges Lightning funds for on-chain funds atomically, using the same hashlock principle as an HTLC. A merchant with too much inbound-side balance can pay a swap provider over Lightning and receive on-chain bitcoin in return, which empties her channels' local side and so restores her ability to receive. Lightning Labs' Loop service and the Boltz exchange are prominent providers. The reverse direction, paying on-chain to receive Lightning funds, refills outbound liquidity. A third approach changes the channel itself. Splicing, now deployed in production by several implementations, lets the two parties add funds to or remove funds from an existing channel through a single on-chain transaction, without closing it. A user can top up a depleted channel, or move part of a channel's funds to an on-chain address, while the channel keeps operating. Splicing has become the backbone of some mobile wallets' liquidity management, because it lets a single channel grow and shrink with the user's balance. Finally, liquidity can be bought. A node that needs inbound capacity can pay another to open a channel toward it. This has evolved from informal arrangements into marketplaces and standardized protocols, including Lightning Labs' Pool, liquidity advertisements in Core Lightning, and the specifications for Lightning Service Providers discussed later. Liquidity is the real capacity When people describe Lightning's capacity, they often quote the total bitcoin locked in public channels. That number is not useless, but it misleads in two directions. It overstates usable capacity, because much of it sits on the wrong side of channels for any given payment, and because a single bitcoin locked in a channel can only be on one side at a time. And it understates the network's throughput, because the same liquidity can carry an unlimited number of payments over time as long as flows roughly balance. A channel of one million satoshis can carry a billion satoshis of payments if money goes back and forth. The true constraint on Lightning is not total capacity but the match between where liquidity sits and where payments need to flow. That is a coordination problem, not a cryptographic one, and the network's solution has been a combination of probabilistic routing, multipart payments, rebalancing tools, price signals and, increasingly, specialized intermediaries who manage liquidity professionally. Each of those solutions works, and each pushes the network's shape in a particular direction, toward well-capitalized hubs that hold balance on many sides at once. That shape has consequences for privacy, censorship resistance and the economics of using Lightning, and the network's custodians spend much of their energy deciding how far to accept it. Before turning to those questions, though, we have to examine what can go wrong with the machinery itself. Chapter 5: The Security Model and Its Fault Lines On-chain Bitcoin offers a simple security promise. If you control your private keys and your transaction confirms, your money is yours, and you can go to sleep for ten years and wake to find it untouched. Lightning keeps that self-custody, but it replaces the simple promise with a conditional one. Your channel balance is yours provided that you, or someone acting for you, are watching the blockchain, that you can get a transaction confirmed before certain deadlines expire, and that your node's records of past states are intact. Each of those provisos is a fault line along which real failures have occurred or been demonstrated. This chapter goes through them in turn. None of this means Lightning is unsafe in the ordinary sense. Losses from protocol-level attacks in the wild have been rare, and the network has moved real value for years. But a reader who wants to judge Lightning fairly has to understand the precise shape of its risks, because they are unlike those of any other payment system. The obligation to watch The penalty mechanism described in the second chapter deters cheating only if the victim notices. If Bob broadcasts an old commitment while Alice's node is offline, and Alice does not come back online before Bob's to_self_delay expires, Bob can sweep the funds from that old state and Alice cannot punish him. The delay is typically between one day and two weeks. A merchant whose node is always on faces little risk. A phone wallet that is opened once a week is in a different position. The response is the watchtower: a third-party service that monitors the blockchain on a user's behalf and broadcasts the penalty transaction if a revoked state appears. The protocol is designed so that the watchtower does not learn the user's balances or payments. After each state update, the user gives the watchtower an encrypted penalty transaction, keyed to an identifier derived from the commitment it punishes. The watchtower cannot decrypt it until the commitment it relates to actually appears on chain, at which point it can decrypt and broadcast. Until then it holds opaque blobs. Watchtowers are built into LND and available for Core Lightning and other implementations, and many mobile wallets rely on watchtowers run by their providers. A watchtower introduces a mild form of trust: the user relies on it to be online and to act. It cannot steal the user's funds, because the penalty transaction pays the user, but it could fail to protect them. Using several independent watchtowers reduces that risk. In practice, the strongest protection is often economic rather than technical. A counterparty who might cheat does not know whether the victim is watching, and the cost of being caught is everything in the channel. Deadlines and the fee market Every enforcement action in Lightning races a clock. The penalty transaction must confirm before the cheater's delay expires. An intermediary who learns a preimage must claim its incoming HTLC on chain before that HTLC's timeout, if the upstream party will not settle cooperatively. A party whose counterparty has gone silent with HTLCs pending must force-close and resolve them before their deadlines. All of these races are run on Bitcoin's base layer, which means they are subject to the fee market. If blockspace is congested, confirming a transaction quickly requires paying a high fee. If the fee needed exceeds the value at stake, enforcement becomes uneconomic. This is why small HTLCs are treated as dust and not enforced on chain at all, and why a node with many small pending HTLCs during a fee spike may face a situation where defending them costs more than they are worth. The more alarming version of this is a mass-exit scenario. If a large number of channels needed to close at once, perhaps because a widely used implementation had a critical bug, or a large hub failed, the resulting flood of force-close and HTLC transactions would compete for blockspace. Some would not confirm before their deadlines. Poon and Dryja recognized this risk in the original paper and described it as a forced expiration spam attack. In 2020, Jona Harris and Aviv Zohar formalized a version called Flood and Loot, in which an attacker opens many channels to victims, routes many payments to itself through them, and then refuses to settle, forcing the victims to close their channels all at once and race to claim their HTLC refunds on chain. Their simulations showed that an attacker who could trigger enough simultaneous closures would expect some victims to miss their deadlines, at which point the attacker could claim the funds. Anchor outputs, which let the closer set the fee at broadcast time, made such attacks considerably harder by removing the problem of pre-signed fees being too low. Limits on the number and total value of pending HTLCs in a channel bound the attacker's leverage. But the fundamental point stands. Lightning's security depends on blockspace being available at a price the honest party can pay, at the moment they need it. That is a weaker assumption than the base layer requires, and it is the place where Lightning's design most directly depends on the scarce resource it economizes. Attacks on the mempool A subtler class of attack exploits not the block but the mempool, the pool of unconfirmed transactions that each node holds and relays before they are mined. For an honest party's transaction to confirm, it must first reach miners. If an attacker can prevent it from propagating, or can arrange for it to be evicted from mempools, it does not matter how high the fee is. The most serious such attack disclosed so far is replacement cycling, made public in October 2023 by Antoine Riard, a longtime Lightning and Bitcoin contributor. It targets a forwarding node, call it Bob, in the middle of a route between an attacker's two nodes. The attacker routes a payment through Bob, then lets the downstream HTLC time out on chain. Bob broadcasts his timeout transaction to reclaim his outgoing HTLC before he must settle the incoming one. The attacker then broadcasts a competing transaction that spends the same HTLC output using the preimage, paying a higher fee, which replaces Bob's timeout transaction in mempools. Then the attacker replaces their own transaction with yet another that spends a different input, which removes the preimage-revealing transaction too. Bob's timeout transaction is gone, but so is the preimage. By repeating this cycle until Bob's incoming HTLC has also timed out, the attacker can claim the incoming payment on one side and the refund on the other, leaving Bob out of pocket by the HTLC amount. Mitigations deployed in the major implementations include rebroadcasting timeout transactions more aggressively, scanning mempools for preimages, and adjusting deltas, which make the attack costlier and less reliable without fully eliminating it. A complete fix is generally thought to require changes to Bitcoin's transaction relay policy or consensus rules. Riard's disclosure was accompanied by his announcement that he was stepping back from Lightning development, which drew attention to how thin the pool of people who deeply understand these interactions is. Earlier work had documented related problems. Pinning attacks, in which an attacker attaches a large, low-fee child transaction to a commitment or HTLC transaction to make it expensive to replace, motivated the design of anchor outputs and later the new TRUC transaction rules, which restrict the size of children of certain transactions precisely to make pinning impractical. Time-dilation attacks, described by Antoine Riard and Gleb Naumenko in 2020, isolate a victim node from the honest Bitcoin network and feed it blocks slowly, so that its sense of the current block height lags reality and its deadlines pass before it realizes. Defending against these requires a Lightning node to have reliable, diverse connections to the Bitcoin network, which is a point against light clients that depend on a single server. Channel jamming A different family of attacks does not aim to steal but to deny service, and it exploits the property noted at the end of the third chapter: HTLCs can freeze funds in transit. In a channel jamming attack, an attacker sends payments to itself through a target channel and then simply holds the HTLCs without settling or failing them. While the HTLCs are pending, the liquidity they reserve cannot be used for anything else. Because a channel can hold at most 483 pending HTLCs in each direction, an attacker can also fill those slots with tiny payments, blocking the channel regardless of its capacity. The attack costs the attacker little. Failed payments in Lightning pay no fees, because fees are only collected when a payment succeeds. The attacker's capital is locked for the duration of the attack, but it is returned afterward. Ayelet Mizrahi and Aviv Zohar described the economics of such congestion attacks in 2020, showing that a modest amount of capital could disable a meaningful share of the network's routing capacity for days. Channel jamming is widely regarded as the hardest unsolved problem in Lightning's protocol design, because the obvious cures harm ordinary use. Charging fees for failed or slow payments, so-called upfront fees, would make jamming expensive but would also charge honest users for failures they could not control, and is hard to design without creating new attacks. Requiring users to stake reputation, through tokens or identity, threatens privacy and permissionless access. Current proposals combine several ideas: local reputation, in which each node tracks how its peers' forwarded payments have behaved and reserves part of its liquidity and HTLC slots for peers with good histories, and limited upfront fees for the remainder. Experimental implementations of HTLC endorsement, a signal by which a node vouches for a payment it forwards, have been developed in recent years. The problem is under active work, not solved. Backups and the danger of honesty The revocation scheme creates a hazard that has no parallel in on-chain Bitcoin: restoring from an old backup can cost you everything. If a node's database is lost and the owner restores a backup from last week, the node believes the channel state is what it was last week. If it broadcasts that state, it is broadcasting a revoked commitment, and the counterparty, behaving entirely correctly, will sweep all its funds as a penalty. For this reason, Lightning nodes cannot be backed up in the conventional sense. The solution adopted by the major implementations is a static channel backup, which records not channel states but the minimal information needed to identify each channel and the counterparty. After a data loss, the user restores this backup, and the node contacts each counterparty, announces that it has lost its state, and asks them to force-close. The data-loss protection protocol, part of the channel reestablishment procedure, has the counterparty broadcast its latest commitment, which pays the restored user's balance directly to them. This depends on the counterparty's cooperation: a malicious peer could instead broadcast an old commitment that favored itself, and the user, having lost state, could not punish it. In practice, the counterparty has no way to know whether the user really lost data or is testing it, so cheating carries risk. Static backups recover funds, but at the price of closing every channel, paying on-chain fees and losing the liquidity arrangement. Some implementations offer continuous replication of the channel database to remote storage, and some mobile wallets store encrypted channel state with their provider. The general lesson is that a Lightning node is a stateful financial application, and treating it like a stateless wallet with a seed phrase is a recipe for loss. Bugs in the machinery Not every risk comes from the protocol design. Lightning implementations are large, intricate programs handling money in real time, and they have had serious bugs. In 2019, developers disclosed a vulnerability that affected all three main implementations of the time. Nodes were not fully checking that the funding transaction a counterparty claimed to have broadcast actually paid the correct amount into the correct output. An attacker could open a channel that appeared funded but was not, and then route payments out of the victim's other channels against nonexistent balance. The flaw was fixed in coordinated releases before details were published, and users were urged to upgrade, but it showed how a single missing check in channel setup could undermine every guarantee built on top. In October 2022, a Bitcoin transaction with an unusually large Taproot witness, valid under the consensus rules, triggered a parsing limit in btcd, the Go library that LND used to read blocks. Many LND nodes stopped processing new blocks, which meant they could not see channel closes or enforce deadlines until they upgraded. A similar incident followed weeks later. No widespread losses were reported, because the problem was spotted quickly and patched, but the episode was a vivid demonstration of the second section's point: a Lightning node that cannot see the chain is a node whose security clock is running down. Smaller bugs, causing unexpected force closes, stuck payments or incompatibilities between implementations, have been routine throughout Lightning's history. Each is individually minor, but they explain why experienced operators keep software current, follow security announcements closely and maintain on-chain reserves for the days when something forces a channel shut. A summary of the fault lines The attacks and failure modes discussed in this chapter differ in what the attacker gains, what it costs them and how mature the defenses are, as Table 4 summarizes. Table 4. Principal attacks and failure modes in Lightning. Threat What the attacker does Effect on victim Main mitigations Revoked-state broadcast Publishes an old commitment while victim is offline Loss of channel funds if unanswered Penalty mechanism, watchtowers Flood and loot Forces many channels to close at once HTLC losses if deadlines pass Anchor outputs, HTLC limits Replacement cycling Repeatedly evicts victim's timeout transaction from mempools Loss of a forwarded HTLC Aggressive rebroadcast, policy changes Time dilation Isolates victim from Bitcoin network Missed deadlines Diverse block sources Channel jamming Holds HTLCs or fills HTLC slots Liquidity unusable, lost fees Local reputation, upfront fees (in development) Stale backup restore No attacker needed Penalty against oneself Static channel backups, state replication Where the guarantee actually lies Taken together, these fault lines show where Lightning's security guarantee sits. It is not cryptographic in the pure sense that on-chain ownership is. It is a game-theoretic guarantee resting on a cryptographic base. Cheating is possible; it is made unprofitable by penalties that the victim can impose, on a schedule enforced by the blockchain, at a price set by the fee market. That is a remarkable achievement. It allows a payment network of strangers to operate without custodians and with failure rates low enough for daily use. But it is a guarantee with operating conditions, and those conditions favor certain participants. A well-run, always-online node with good Bitcoin connectivity, reserves for fees and reliable backups is very safe. A casually managed phone wallet is safe largely because of the services that support it: the watchtowers, the backup storage, the liquidity provider who would gain little from cheating and lose much reputation. That dependence is not a failure of the protocol, but it shapes the network's social and commercial structure, which is where the next chapter turns. Chapter 6: How the Network Is Actually Run The early picture of Lightning, sketched in the original paper and in countless enthusiastic diagrams, was of a flat mesh of peers. Everyone would run a node, open a few channels to people they knew or to random peers, and route each other's payments. The network that actually exists looks rather different. Most people who use Lightning do not run a full node in any meaningful sense. They use a mobile application that holds keys on the phone and depends on a service provider for liquidity, routing information and protection, or they use an account at an exchange or payment company that holds their bitcoin for them and interacts with Lightning on their behalf. Behind those front ends sits a smaller set of well-capitalized nodes that move most of the volume. This shape was not imposed by anyone. It emerged from the constraints the previous chapters describe: the need for inbound liquidity, the need to be online, the difficulty of routing with hidden balances and the complexity of safe operation. Understanding it is necessary for understanding what Lightning is today and what it can become. The implementations Four independent codebases implement the BOLT specifications and carry most of the network's traffic. They have different design philosophies, different strengths and different user bases, as Table 5 summarizes. Table 5. The principal Lightning implementations. Implementation Developer Language Typical use LND Lightning Labs Go Routing nodes, merchants, node-in-a-box products Core Lightning Blockstream C, with plugins in many languages Routing nodes, developers, plugin-driven services Eclair ACINQ Scala Large hub node, server side of Phoenix wallet LDK Spiral (Block) Rust library Embedded in wallets and applications LND has historically been the most widely deployed, in part because it was packaged early into consumer node products and because its tooling for operators is extensive. Core Lightning emphasizes a small core extended by plugins, which has made it popular with developers building specialized services and with operators who want fine control; it has often been first to ship features such as BOLT12 offers and liquidity advertisements. Eclair powers ACINQ's own large routing node and the server side of its Phoenix wallet, and it has pioneered features designed for mobile users, including trampoline routing and production splicing. LDK, the Lightning Development Kit, is not a standalone node but a library from which developers can build Lightning into their own applications, choosing their own storage, networking and key management. Its approach suits wallets that want Lightning functionality without running a separate daemon, and it has been used in a number of mobile and server-side products. The existence of several independent implementations is a real strength. A bug in one does not take down the network, because the others continue to operate. It also makes protocol change slow, because a feature only works across the network when enough implementations support it, and the BOLT process requires interoperable implementations before a change is merged into the specification. Every major upgrade, from anchor outputs to Taproot channels to BOLT12, has unfolded over years as implementations shipped support at different times. Why most users do not run a node Running a Lightning node well requires a Bitcoin full node or a reliable connection to one, constant uptime, careful backup management, an understanding of liquidity and fee settings, and on-chain funds to open channels and pay for force closes. That is a reasonable burden for a business or a dedicated hobbyist. It is an unreasonable one for a person who wants to pay for lunch. A phone is a particularly hostile environment. It goes offline frequently, mobile operating systems restrict background processes, and a user may not open their wallet for days. A phone cannot easily route payments for others, because it is not reliably reachable. And a newly installed wallet has no channels, so it has no liquidity in either direction until someone opens a channel to it. The industry's answer to all of this is the Lightning Service Provider, or LSP. Lightning Service Providers An LSP is a well-connected, always-online node that serves end users. In the most common arrangement, a user installs a non-custodial wallet, which generates and holds keys on the phone. When the user first receives a payment, the LSP opens a channel to the phone and pushes the incoming payment through it, deducting a fee for the on-chain cost and the capital it has committed. From then on, the user's wallet has a single channel to the LSP, which handles routing to the rest of the network. When the user needs more inbound capacity, the LSP opens a larger channel or splices more funds into the existing one. This pattern goes by several names. A channel opened at the moment of receiving is often called a just-in-time channel. A channel the LSP opens before it is needed, on payment of a fee, is a purchased channel. Some LSPs also accept zero-confirmation channels, which the user can spend from before the funding transaction confirms; this is safe for the user to receive into only because the LSP is the party funding the channel, and the user trusts the LSP not to double-spend its own funding transaction for the brief period before confirmation. LSPs typically also provide watchtower service, encrypted backup storage, and routing assistance. Because a phone with a single channel has limited view of the network, some wallets delegate pathfinding to the LSP using trampoline routing. In a trampoline payment, the phone builds an onion that specifies only the LSP and the final recipient, with the LSP computing the actual route between them. This saves the phone from downloading and maintaining the full gossip graph, at some cost in privacy, since the LSP learns the destination. The relationship between the major wallets and their LSPs varies. Phoenix, from ACINQ, connects to ACINQ's own node and uses splicing to keep a single channel sized to the user's balance, which makes the wallet look and behave much like an ordinary bitcoin wallet. Breez has developed a software development kit that lets other applications embed non-custodial Lightning, backed by LSP services. Various wallets built on LDK use LSPs operated by their developers or by third parties. Interoperability standards for LSPs have been developed so that a wallet can in principle work with more than one provider, using shared specifications for ordering channels and requesting just-in-time liquidity. What an LSP cannot do, in the non-custodial model, is take the user's money. The user holds keys, signs commitments and can force-close the channel unilaterally. The LSP could refuse to route, could be shut down, or could charge high fees, and in those cases the user's recourse is to close the channel on chain and move elsewhere. That recourse is real but costly, which is why choosing a reliable LSP matters. A wallet's first week Following a new user of a self-custodial mobile wallet through her first days shows how much the LSP does. Sara installs the wallet and writes down a recovery phrase. At this point she has keys but no channels, no balance and no liquidity in either direction. A friend sends her 100,000 satoshis by scanning an invoice her wallet generated. That invoice did not name Sara's node in the public graph, because Sara has no public channels; it carried a routing hint pointing to her LSP and a special identifier telling the LSP to open a channel on arrival. The friend's payment travels to the LSP, which holds the incoming HTLC, opens a new channel to Sara's phone with somewhat more capacity than the payment, and forwards the HTLC through it minus a fee covering the on-chain cost and the capital committed. Sara's wallet reveals the preimage, the payment settles, and she has a balance. Because the LSP funded the channel, Sara can spend from it immediately, without waiting for the funding transaction to confirm. Over the next few days Sara pays for lunch twice and buys a gift card. Each payment leaves her phone as an onion, often delegating pathfinding to the LSP, and arrives within a second or two. Every update is backed up in encrypted form to the provider, and penalty data goes to a watchtower. Sara is unaware of any of it. At the end of the week a larger payment arrives, more than the inbound capacity remaining in her channel. Her LSP adds capacity, either by splicing funds into the existing channel or by opening a second one, and charges another fee. If Sara later wants to move her savings to a hardware wallet, her wallet can splice funds out to an on-chain address, or close the channel cooperatively with the LSP. If the LSP ever disappeared, Sara could force-close and recover her balance on chain, after a delay and at the prevailing fee. Every element of the protocol from the earlier chapters is present in that week, and Sara has dealt with none of them directly. She has paid for that convenience in fees, which are typically larger at the first channel opening than at any later point, and in a degree of dependence on one company's continued operation. Custodial Lightning A large share of Lightning payments, quite possibly the majority by count, are made by people who hold no Lightning keys at all. They use an account at an exchange, a payment application or a custodial wallet, and that company runs Lightning nodes on their behalf. When a user of such a service pays a Lightning invoice, the company sends the payment from its own node and debits the user's account. When the user receives, the company's node receives and credits the account. From the network's perspective, this is ordinary Lightning traffic between large nodes. From the user's perspective, it is a bank account denominated in bitcoin with a fast external payment rail. The user enjoys convenience and avoids liquidity management, backups and on-chain fees entirely. The cost is that the user's funds are an IOU from the custodian, with all the counterparty, regulatory and censorship exposure that implies. Custodial services have been critical to Lightning's adoption. Strike, founded by Jack Mallers, uses Lightning to move value across borders and into local currency. Cash App, Block's consumer payments application, added the ability to send Lightning payments in 2022 and to receive them later. Major exchanges including Kraken and, from 2024, Coinbase support Lightning deposits and withdrawals, the latter through a partnership with the infrastructure company Lightspark. Payments between users of such services and users of self-custodial wallets flow through the same network. The prevalence of custodial use is a point of genuine contention. Critics argue that it makes Lightning a messaging layer between banks, which recreates the system Bitcoin was meant to escape. Defenders argue that custody is a choice, that the presence of a non-custodial option disciplines custodians, and that interoperability between custodial and non-custodial users is exactly what an open network should provide. Both points have force. What is certain is that the technical constraints of self-custodial Lightning push users toward custody, and that every improvement in non-custodial usability, from LSPs to splicing, has been in part an effort to push back. The shape of the graph The combined effect of LSPs, custodians and professional routing is a network that is highly centralized in its topology. A modest number of large nodes, operated by exchanges, payment processors, wallet providers and specialist routers, hold a substantial share of public capacity and sit on most payment paths. Peripheral users connect to one or two of these hubs. Researchers studying the public graph have repeatedly found that it resembles a hub-and-spoke network more than a mesh, with a small number of nodes whose removal would significantly fragment it. This shape is efficient. Payments between peripheral users take short routes through well-capitalized hubs, so they are likely to succeed and cost little. Liquidity is concentrated where it can be managed professionally. But it has costs. Hubs see a large share of traffic, which weakens the privacy that onion routing is meant to provide, since a hub adjacent to both sender and recipient can often link them. Hubs are identifiable businesses, often regulated, which makes them potential points of pressure for censorship or surveillance. And a failure at a major hub, technical or legal, would disrupt many users at once. None of this makes Lightning a centralized system in the sense that a bank is. No hub can take a user's funds in a non-custodial channel, and a user who is refused service can route around a hub or close a channel and open another. The network's resistance to capture comes from the ease of exit, not from the absence of large participants. But the ease of exit is bounded by on-chain costs, and that bound tightens whenever fees rise. Privacy in practice The privacy properties of Lightning as actually used are therefore mixed. On-chain observers see channel opens and closes, but not the payments in between, which is a large improvement on the base layer. Onion routing hides routes from most intermediaries. Unannounced channels keep mobile users out of the public graph. Against that, invoices in the widely used BOLT11 format reveal the recipient's node identity, and for users behind an LSP, the hints in an invoice reveal the LSP. A user's LSP or custodian sees much of what the user does. Repeated payment patterns, timing and the shared payment hash across hops create linkage opportunities for anyone who controls multiple points on a route. Researchers have shown that balances in public channels can be probed by sending payments that are designed to fail, revealing how much liquidity sits on each side. Several protocol changes target these weaknesses. Route blinding allows a recipient to hide its position in the network behind an encrypted path segment that the sender uses without decrypting. It is a core part of the BOLT12 offers standard and has been deployed in several implementations. The move from hashes to point-based locks, discussed in the final chapter, would remove the shared identifier across hops. Each of these narrows the gap between Lightning's theoretical privacy and its practical privacy, without closing it. The operator's reality For the businesses that run the network's core, Lightning is an operational discipline. A routing or LSP operator must manage dozens or hundreds of channels, set fees in response to flows, rebalance, maintain on-chain reserves for force closes, keep software current with an evolving protocol, and respond to incidents, including bugs in their own implementation or a peer's that cause unexpected closures. The margins on routing are thin, and the capital costs are real. LSPs earn fees for channel opens and liquidity, and custodians earn from the broader financial services they offer. The profitability of these businesses, and therefore their persistence, is one of the less discussed determinants of Lightning's future. That operational reality is also what makes Lightning workable at the counter. A merchant does not need to run any of this machinery herself. She can rely on a payment processor, a point-of-sale provider or a wallet with an LSP, each of whom has absorbed the complexity. The next chapter looks at that point of contact: how a Lightning payment is requested, how it is made, and where it has actually been used to buy things. Chapter 7: At the Counter Everything described so far, the channels, the HTLCs, the staggered deadlines, the liquidity management, exists so that a customer can stand at a till, point a phone at a code and walk away with a coffee two seconds later, having paid a fee too small to notice. This chapter examines that moment: how a merchant requests payment, how the customer's wallet responds, how the standards for doing so have evolved, and what real deployments have revealed about where Lightning works at the point of sale and where it struggles. The invoice A Lightning payment almost always begins with the recipient. Because the HTLC chain depends on a hash whose preimage only the recipient knows, the recipient must generate that preimage and communicate its hash to the payer before any money can move. The standard vehicle for this is the invoice defined in BOLT 11. A BOLT11 invoice is a string, typically several hundred characters long, beginning with a prefix that identifies the network and the amount, such as lnbc for Bitcoin's main network. It is encoded in bech32, the same character set used for modern Bitcoin addresses, which makes it compact enough to fit in a QR code. Inside, it carries the payment hash, the amount requested, a short description, an expiry time, the recipient's node public key, the final CLTV delta the recipient requires, any routing hints needed to reach a recipient behind an unannounced channel, and a signature by the recipient's node over all of it. The default expiry is one hour, though merchants commonly set shorter or longer values. At a shop, the till or payment terminal requests an invoice from the merchant's node or payment processor, displays it as a QR code, and waits. The customer scans the code with a wallet, which decodes the invoice, shows the amount and description, and asks for confirmation. On confirmation, the wallet finds a route, sends the HTLCs, and within a second or two receives the preimage back. The merchant's system, which has been watching for the payment, sees it settle and marks the sale complete. From the customer's point of view, it resembles a contactless card payment with an extra step of scanning. This design has strengths that matter at the counter. Settlement is final: once the preimage is released, there is no chargeback mechanism, which removes a major cost and fraud risk for merchants who sell to strangers. Payment confirmation is cryptographic: the customer's wallet holds the preimage as proof of payment. And the fees paid by the customer are typically small fractions of a cent for retail amounts, while the merchant may pay nothing at all to receive, depending on the provider. It also has limitations. Each BOLT11 invoice is meant to be paid exactly once, because paying the same hash twice would let any intermediary who saw the first payment's preimage steal the second. A merchant cannot print a single static QR code on a sign and have customers pay it repeatedly. Invoices expire. The invoice reveals the recipient's node identity. And the payer must be online at the moment of payment, as must the recipient's node or its provider. Static codes: LNURL and Lightning addresses The single-use nature of invoices led quickly to a set of conventions, collectively known as LNURL, that let a static code or identifier lead to a fresh invoice on demand. The most widely used is LNURL-pay. A merchant publishes a static QR code that encodes a web address. When a customer's wallet scans it, the wallet contacts that address over HTTPS, receives information about what amounts the merchant accepts, lets the customer choose or confirm an amount, and then requests an invoice from the server for that amount. The server generates a new invoice, and the wallet pays it. To the customer, it looks like paying a static code. The Lightning address, introduced in 2021, builds on LNURL-pay by giving it a familiar form: an identifier that looks like an email address, such as name@example.com. A wallet that is given a Lightning address converts it into a well-known web address on the domain, fetches the LNURL-pay information, and proceeds as before. Lightning addresses have become a common way to share a payment destination on social media, in tip jars and on invoices, and many wallets and custodial services issue them to their users. Other LNURL variants handle related tasks. LNURL-withdraw lets a service offer a code that, when scanned, pays the scanner, which is useful for vouchers, refunds and cash-out. It is also the basis of the Bolt Card, an NFC card that holds a key and, when tapped on a compatible point-of-sale device, authorizes a withdrawal from the cardholder's wallet to pay the merchant. Tap-to-pay with a Bolt Card brings the experience close to contactless card payments and has been used in a number of community deployments. The weakness of LNURL is that it depends on a web server. The merchant must run an HTTPS endpoint, or rely on a provider who does, and the payer's wallet must make a web request that reveals its network address to that server. The domain owner can be compelled, hacked or shut down. LNURL works well, and it is widely deployed, but it moves part of the payment process off Lightning and onto the conventional web. BOLT12 offers The native protocol answer to these problems is BOLT12, the offers specification, developed principally by Rusty Russell of Blockstream and merged into the Lightning specification in 2024 after years of development. An offer is a static, reusable payment code that a merchant can print, post or embed, and that never expires unless the merchant sets an expiry. A customer's wallet that scans an offer does not contact a web server. Instead, it sends a message across the Lightning network itself, using onion messages, to request an invoice from the merchant's node. The merchant's node replies, again via onion messages, with a fresh invoice. The wallet pays it. Because the request and reply travel through onion-routed messages rather than the web, neither side needs a domain or a server, and neither side reveals its network address to the other. Offers use route blinding, so the merchant can hide its node behind an encrypted path, and the payer learns a way to reach the merchant without learning where the merchant is. Offers also support features that BOLT11 lacked: refunds, in which the merchant pays a customer back by requesting an invoice from them; recurring payments, which a subscription service can request on a schedule; and payer proofs, by which a customer can prove they paid a specific invoice. Support for BOLT12 has spread unevenly. Core Lightning, Eclair and LDK implemented it, and wallets including Phoenix and several LDK-based applications adopted it. Some custodial services, among them Strike, added support for paying offers. LND, the most widely deployed implementation, was slower, and for much of the period since the specification was merged its users relied on external projects to handle offers. As with other Lightning upgrades, the network's practical capability is set by the slowest widely used implementation, and the full benefits of offers depend on broad deployment. The three ways of requesting a payment differ in reusability, infrastructure and privacy, as Table 6 sets out. Table 6. Ways of requesting a Lightning payment. Method Reusable Infrastructure needed Recipient privacy Status BOLT11 invoice No, single use with expiry Lightning node only Node identity revealed Universal LNURL-pay and Lightning address Yes Web server and domain Domain and server visible Widely supported BOLT12 offer Yes Lightning node with onion messages Route blinding hides node Supported in several implementations, spreading The merchant's real problems The technical payment flow is the easy part of point-of-sale Lightning. The harder parts are the problems every merchant has, which Lightning solves in some cases and not in others. The first is volatility. A merchant whose costs are in dollars or euros does not necessarily want to hold bitcoin, whose price can move substantially in a day. Most merchants who accept Lightning do so through a processor that converts the payment into local currency at the moment of sale, or that lets the merchant choose what share to convert. Services such as OpenNode, Strike and various regional processors offer this. Converted at once, Lightning becomes for the merchant a payment rail like a card network, with faster settlement and lower fees, rather than a store of value. The second is accounting and tax. In many jurisdictions, spending bitcoin is a taxable disposal, and in jurisdictions such as the United States each purchase can create a capital gain or loss that the customer must in principle track. This is a significant friction for customers who hold bitcoin as savings and is not something Lightning can solve. It has motivated, among other things, interest in denominating Lightning payments in stable assets, discussed in the final chapter. The third is liquidity, in its specific form at the counter: a merchant receives far more than she spends, so her channels fill up on the receiving side and she runs out of inbound capacity. Processors and LSPs handle this for the merchant by periodically moving the received funds out, either to on-chain bitcoin, to fiat accounts or onward through the network. A merchant running her own node must do it herself, through swaps, splicing or purchased channels. The fourth is customer reach. A payment method is only useful if customers have it. Lightning's user base is large in absolute terms and small compared with cards. For most merchants, accepting Lightning has been a marketing choice or a service to a niche clientele rather than a significant revenue channel, though there are places where that is not true. Refunds, disputes and the absence of chargebacks The finality of Lightning payments is a double-edged property at the counter. For the merchant, it is a clear gain: card networks allow customers to reverse payments weeks after a sale, and the cost of fraudulent chargebacks, together with the fees and administration of handling legitimate ones, is substantial in many retail and online sectors. A Lightning payment, once the preimage is released, cannot be pulled back by anyone. For the customer, it means the protection that a card issuer provides is absent. If goods are not delivered or a service is not performed, the customer's recourse is the merchant's goodwill, the law of contract or consumer protection, not a button in a banking app. This is how cash works, and for small in-person purchases most customers accept it without thinking. For larger or remote purchases, it is a real reason some customers prefer cards. Refunds themselves have historically been awkward. With BOLT11, a merchant who wanted to refund a customer needed the customer to produce a fresh invoice, which meant asking for it by some other channel. LNURL-withdraw codes and BOLT12's refund mechanism make the process smoother, letting the merchant offer a refund the customer's wallet can claim directly. Hold invoices offer a partial alternative to refunds for some cases, letting a merchant accept a payment provisionally and only settle it once the order is confirmed. None of these recreates the consumer protection of card networks, and that absence is a deliberate feature of a bearer payment system rather than a gap the protocol will eventually fill. Deployments The most famous Lightning point-of-sale experiment began in El Zonte, a small beach town in El Salvador, where a project known as Bitcoin Beach, started in 2019 with an anonymous donation, set out to build a local economy in which residents could pay each other and local businesses in bitcoin. Its experience fed directly into El Salvador's Bitcoin Law, which took effect on 7 September 2021 and made bitcoin legal tender, requiring businesses to accept it where they had the technical means. The government launched its own Lightning-capable wallet, Chivo, with a thirty-dollar bitcoin incentive for each citizen who signed up. The results were mixed in ways that illustrate Lightning's realities. The technology worked: millions of people downloaded wallets and Lightning payments were completed at scale. But Chivo's early rollout suffered from serious operational problems, sustained use remained limited for most of the population, and surveys found that many users stopped after spending the sign-up bonus. In early 2025, as part of an agreement with the International Monetary Fund, El Salvador amended the law so that businesses were no longer required to accept bitcoin. Bitcoin Beach and similar community projects have continued, and the country remains a place where paying in bitcoin at a café is routine in certain areas. The lesson most observers draw is that the payment rail was not the obstacle; the obstacles were habit, volatility, merchant incentives and institutional execution. Elsewhere, Lightning has found footholds where existing payment systems are expensive or unavailable. Community circular economies in places from South Africa to Guatemala to Kenya use Lightning for local commerce, often through custodial wallets designed for people without bank accounts. Services have been built to let people with basic feature phones send and receive Lightning payments over USSD menus, without smartphones or internet access, at the cost of using a custodian. Cross-border remittance services use Lightning as a fast intermediate rail, converting from one currency to bitcoin and from bitcoin to another within seconds, and often without either end ever holding bitcoin. In online commerce, BTCPay Server, an open-source, self-hosted payment processor, has made Lightning acceptance available to any merchant willing to run it, with integrations into common e-commerce platforms. Lightning has also become a favored rail for small payments that card networks handle poorly: tips, pay-per-use content, streaming payments for podcasts, and, more recently, machine-to-machine payments in which software agents pay for computation or data in small increments. The most significant point-of-sale development of recent years came from Block. In 2025 the company announced that Square, its merchant payments platform, would allow sellers to accept bitcoin payments over Lightning at the point of sale, with funds settling to the merchant either in bitcoin or converted to dollars, and with processing fees waived for an introductory period. The rollout to eligible sellers began later that year. Because Square's terminals are already present in millions of shops, this removed, for those merchants, the question of whether accepting Lightning was worth the setup effort. It does not on its own create customers who want to pay that way, and how much volume such integrations attract will be one of the more informative signals about Lightning's retail future. What the counter teaches Point-of-sale deployments show that Lightning delivers what it promises at the counter: payments in seconds, at fees far below cards, with final settlement and no chargebacks. They also show that a superior payment rail does not by itself change how people pay. Cards offer credit, fraud protection, rewards and universality, and customers are slow to abandon them. Lightning wins most clearly where those advantages do not apply: where cards are expensive or unavailable, where the amounts are too small for card fees, where payments cross borders, or where the parties prefer a bearer instrument that does not pass through a bank. The infrastructure for the counter is now largely built. Invoices, static codes, offers, NFC cards and point-of-sale integration exist and work. The question that remains is less about technology than about whether enough people will hold and spend bitcoin, or bitcoin-settled assets, for merchants to care. The final chapter considers the technical developments that might change the answer, and the unresolved tensions that might not. Chapter 8: The Unfinished Layer Lightning has a structural ceiling that no amount of better routing or friendlier wallets can lift. Every self-custodial Lightning user needs at least one channel, and every channel needs an on-chain transaction to open and another to close. Bitcoin can confirm a few hundred thousand transactions a day. If every person in the world wanted their own channel, and each channel lasted several years, the base layer would need years of blockspace just to open them, and a single wave of closures would overwhelm it. The arithmetic does not work for billions of self-custodial users with one channel each. That ceiling is why Lightning's future is debated less in terms of its own protocol than in terms of what sits around it: proposals to share a single on-chain output among many users, changes to Bitcoin's consensus rules that would make channels cheaper and safer, and alternative constructions that trade some of Lightning's properties for scale. This chapter surveys those developments and the tensions they expose. Taproot and the new cryptography The Taproot upgrade, activated on Bitcoin's main network in November 2021, introduced Schnorr signatures and a new way of structuring spending conditions. Both matter to Lightning. Schnorr signatures can be aggregated. Using a protocol called MuSig2, two parties can jointly produce a single signature under a single combined public key, so that a two-of-two output looks on chain exactly like an ordinary single-key output. Taproot also allows complex spending conditions, such as revocation paths and HTLC timeouts, to be hidden in a tree of scripts that is revealed only if used. A channel built on these primitives, called a simple Taproot channel, opens with a funding transaction indistinguishable from a normal payment and, if closed cooperatively, closes with one too. Only a force close reveals that a channel existed. Simple Taproot channels have been implemented in LND and are being adopted across the network, though, as with every upgrade, both ends of a channel must support them. The deeper promise of Schnorr is the replacement of hashes in HTLCs with points on an elliptic curve. A Point Time Locked Contract, or PTLC, works like an HTLC except that the condition for claiming the payment is knowledge of a discrete logarithm, a secret scalar, rather than a hash preimage, and it uses a technique called adaptor signatures to tie the revelation of that secret to the completion of a signature. The crucial difference is that each hop can use a different point, derived by adding a random offset, while the whole route remains linked mathematically. No common identifier passes along the route, so two colluding nodes cannot recognize the same payment at both points, and the wormhole attack becomes impossible. PTLCs would also make atomic multipath payments and several other features cleaner. They require changes to the commitment format and to every implementation, and they are the most widely anticipated privacy upgrade in Lightning's roadmap, though their deployment has been slower than many hoped. LN-Symmetry and the end of penalties The penalty mechanism at Lightning's core is effective but unforgiving. Every old state is a trap that the owner must never spring and the counterparty must remember forever. Backups are dangerous. Channels between more than two parties are almost impossible, because a penalty in a multiparty channel would have to punish one party without harming others, and the bookkeeping becomes unmanageable. In 2018, Christian Decker, Rusty Russell and Olaoluwa Osuntokun proposed an alternative called eltoo, now often referred to as LN-Symmetry. Instead of punishing the publication of old states, it allows any later state to replace any earlier one on chain. If Alice publishes an old state, Bob simply publishes the latest one, which spends the old state's output and supersedes it. Nobody is punished; the latest state wins. Each party needs to store only the latest state, and backups lose their danger: restoring an old backup at worst leads to a slower close, not to the loss of all funds. Multiparty channels become feasible. LN-Symmetry requires a capability Bitcoin does not currently have: a signature that is not bound to a specific previous transaction output, so that a single signed update can spend whichever earlier state was published. The proposed soft fork that would enable this, SIGHASH_ANYPREVOUT, specified in BIP 118, has been discussed for years without reaching the broad consensus a Bitcoin soft fork requires. Prototype implementations of LN-Symmetry on test networks have shown the design works. Covenants and the soft-fork debate SIGHASH_ANYPREVOUT is one of a family of proposed changes often grouped under the heading of covenants: mechanisms that let a Bitcoin output restrict how its funds may be spent in the future, not just who may spend them. Other proposals in this family include OP_CHECKTEMPLATEVERIFY, which commits an output to a specific set of future spending transactions, and the restoration of OP_CAT, an opcode removed early in Bitcoin's history that would, in combination with Schnorr signatures, enable a range of covenant constructions. The relevance to Lightning is direct. Covenants would make it possible to construct channel factories, in which a group of users shares a single on-chain output that can be split into many channels off-chain, and to commit to an exit path for each user without requiring all users to cooperate. They would simplify the construction of vaults, payment pools and the shared-output schemes discussed below. And they would strengthen the defenses against some mempool attacks. The debate over covenants is as much about Bitcoin's philosophy as about technique. Supporters argue that without them, self-custodial scaling hits the ceiling described at the start of this chapter, and users will be pushed into custody. Skeptics worry about unintended uses, about the complexity added to Bitcoin's consensus rules, and about the precedent of frequent soft forks on a system whose value rests on stability. Bitcoin's change process is deliberately conservative, and as of the time of writing no covenant proposal has been activated. Lightning's long-term design depends materially on how that debate resolves. Channel factories and multiparty channels The simplest way to see why multiparty constructions matter is to count transactions. If ten people each open a channel to a service provider, that is ten on-chain funding transactions and, eventually, ten closes. If the same ten people and the provider instead lock their funds into a single shared output, and then create channels among themselves off-chain from that output, the on-chain footprint is one funding transaction for the whole group. Channels can be opened, closed and resized inside the group by off-chain agreement, and only the group's final dissolution, or the departure of a member who cannot reach agreement, needs to touch the blockchain again. This construction, the channel factory, was described by Conrad Burchert, Christian Decker and Roger Wattenhofer in 2017, and it is the most natural way to multiply the number of users a given amount of blockspace can support. Its difficulty lies in the update mechanism. With the current penalty scheme, a shared output among eleven parties would need a way to punish whichever party published an outdated state of the group, without harming the others, and every member would have to store revocation data for every other. LN-Symmetry would remove that obstacle, because a later state simply replaces an earlier one. Covenants would remove another, the requirement that every member be online and cooperative to make changes, by committing in advance to each member's exit path. Even without consensus changes, designs that let a small number of parties share channels more efficiently have been explored, and some features already deployed, such as splicing and dual funding, move the network in that direction by letting several changes be batched into one on-chain transaction. A merchant processor that splices many customers' channel changes into a single transaction each block is, in a modest way, already amortizing blockspace across users. Sharing an output: Ark and its relatives The most active area of work on Lightning's scaling ceiling does not wait for a soft fork. Ark, proposed by Burak Keceli in 2023, is a protocol in which a coordinating server, the Ark service provider, periodically creates on-chain transactions that commit to the balances of many users at once. Each user holds an off-chain claim, called a virtual output, that is backed by a pre-signed tree of transactions leading from the shared on-chain output to an output the user alone controls. A user can exit unilaterally by broadcasting their branch of the tree. In normal operation, users transfer their claims among themselves by participating in the server's next round, and the server can connect Ark payments to Lightning, so that an Ark user can pay a Lightning invoice and vice versa. Ark's appeal is that a new user does not need a channel or any inbound liquidity, and many users share the cost of each on-chain transaction. Its trade-offs are real. Virtual outputs expire after a period and must be refreshed by participating in a later round, or the server can reclaim them. Users must come online periodically. The server needs significant capital, because it fronts liquidity for transfers until the old outputs expire. And without covenants, the current designs require users to co-sign rounds or to accept certain trust assumptions in the interval between rounds. Several companies have been building Ark implementations, and early versions have run on Bitcoin's main network, with designs that would become more efficient if covenant opcodes were activated. Related constructions such as statechains, in which ownership of an on-chain output is transferred off-chain with the help of a server that cannot steal but must be trusted to delete old keys, offer other points in the trade-off space. Several of these systems describe themselves as complements to Lightning rather than replacements, using Lightning as the interoperability layer through which payments between different schemes travel. Ecash and community custody A different response to Lightning's usability burden is to embrace custody but make it private and accountable. Chaumian ecash, based on blind signatures invented by David Chaum in the early 1980s, lets a mint issue tokens that it cannot link to the people who hold or spend them. Two prominent projects apply this to bitcoin. Cashu is a simple, single-mint protocol: a user deposits bitcoin over Lightning and receives ecash tokens that can be spent to anyone else using the same mint, or redeemed over Lightning. Fedimint distributes the mint among several guardians using threshold cryptography, so that a majority of guardians must collude to steal, and is aimed at communities that can choose trustworthy guardians among themselves. Both systems are custodial. A mint can steal or disappear, and users must trust it. But they offer privacy that ordinary custodial accounts cannot, they are cheap to use, and they connect to the Lightning network for payments outside the mint. For small balances, a well-run ecash mint may offer a better combination of usability, privacy and risk than either a self-custodial Lightning wallet or a conventional custodial account. They are, in a sense, an admission that the self-custodial ideal does not suit every user and every amount, and an attempt to make the alternative as good as possible. Stablecoins on Lightning A further development changes what is being paid rather than how. Taproot Assets, developed by Lightning Labs and launched on Bitcoin's main network in 2023, is a protocol for issuing assets other than bitcoin on top of Bitcoin outputs, with the assets transferable over Lightning channels. A payment in such an asset is routed through Lightning as an equivalent amount of bitcoin across intermediate hops, with conversion at the edges by nodes that hold channels in the asset. In January 2025, Tether announced that it would issue its dollar stablecoin, USDT, using Taproot Assets, which would make dollar-denominated payments over Lightning possible. Other teams have pursued similar goals through different designs. For the point of sale, this addresses the volatility problem from the previous chapter directly. A customer could hold dollars and pay in dollars, and a merchant could receive dollars, with Lightning as the routing layer. It also raises new questions. A stablecoin is a claim on an issuer, so the custodial risk returns in a different form. The edge nodes that convert between the asset and bitcoin must hold liquidity in both. And a Lightning network that carries mostly dollar-denominated payments would be a different kind of system from the one its designers imagined, closer to a neutral settlement layer for many assets than a bitcoin payment network. Where the limits really are Taken together, these developments show that Lightning is less a finished product than a layer in a stack that is still being designed. The core mechanism, the bilateral channel with conditional routed payments, has proven itself. Its constraints, though, are now clearly understood. Each user needs a share of an on-chain output, and blockspace limits how many outputs can exist. Each user needs liquidity committed toward them by someone else. Each user's security depends on being able to reach the blockchain before a deadline. Every proposal in this chapter relaxes one of those constraints by tightening another or by introducing some measure of trust. That pattern is not a sign of failure. It is the shape of an honest engineering trade. Bitcoin's base layer is scarce so that anyone can verify it. Lightning stretches that scarce resource by moving most activity into private agreements, and the further one wants to stretch it, the more the agreements must involve shared outputs, coordinating servers, custodians or new consensus rules. Which of these the Bitcoin community accepts, and which users choose, will determine how far Lightning's guarantees reach into everyday commerce. Conclusion The Lightning Network began as an answer to a narrow question: how can a blockchain that processes a handful of transactions per second support everyday payments? Its answer was to stop asking the blockchain to process them. Two people lock funds in a shared output, exchange signed transactions that redistribute those funds, and arrange matters so that any attempt to rewind the agreement forfeits everything. Chains of conditional payments, locked to a shared secret and staggered in time, let those bilateral agreements reach strangers. The blockchain becomes a court of final appeal that is rarely called, and the payments it would have had to hear become messages between machines. Having followed that mechanism from the funding transaction to the shop counter, we can say something more precise than either the enthusiasts or the skeptics usually do. What has been demonstrated Lightning has shown that self-custodial bitcoin payments can be fast, cheap and final at the point of sale. This is no longer a hypothesis. Payments of a few cents settle in seconds across several independent intermediaries, without any of them being able to steal the funds, and without a public record of the transaction. Every mechanism in the second and third chapters has been tested by years of real use and by sustained adversarial attention from researchers. When people say Lightning does not work, they are usually describing an inconvenience rather than a failure of the core design. Lightning has also shown that the conversion of global agreement into local enforcement carries a specific and unavoidable price. Liquidity must be committed in advance, in the right places, by someone. Participants must watch the chain or pay others to do so. Enforcement depends on blockspace being available when it is needed. These are not teething problems awaiting better software. They follow directly from the design, and the decade of work on Lightning has been, in large part, a decade of building services, protocols and markets to carry those costs on behalf of users who do not want to bear them. What follows for the reader For a merchant, the practical conclusion is that accepting Lightning is now technically easy and operationally cheap, provided she uses a processor or point-of-sale provider that manages liquidity and conversion. The decision is a commercial one about customers, not a technical one about risk. The places where Lightning gives a merchant a clear advantage are those where card fees bite hardest relative to the sale, where payments come from abroad, where chargebacks are a real cost, or where the customers already hold bitcoin and want to spend it. For a user, the practical conclusion is that the choice between custodial and self-custodial Lightning is a choice about which risks to hold, not whether to hold risk at all. A custodial account offers simplicity in exchange for counterparty exposure. A self-custodial wallet with a reputable service provider offers control over funds in exchange for some dependence on that provider's honesty and availability, and for on-chain fees if things go wrong. A self-run node offers the fullest guarantees in exchange for real operational effort. There is no option that offers base-layer simplicity with Lightning's speed. Users who understand which bargain they have made will make better decisions about how much to hold in each place. For anyone evaluating claims about Lightning, the most useful habit is to ask which of its three constraints a claim relaxes, and at what cost. A new wallet that removes the need for inbound liquidity is probably relying on a service provider who supplies it. A scheme that lets millions of users share one on-chain output is probably relying on a coordinator, a period of online participation, or a future change to Bitcoin's rules. A promise of perfect privacy should be tested against the shape of a network in which hubs see most traffic. None of these trade-offs disqualify a proposal. They locate it. What remains unsettled Three questions will decide what Lightning becomes over the next decade. The first is whether Bitcoin adopts consensus changes that make shared outputs and simpler channels possible. Without them, self-custodial Lightning faces a hard ceiling set by the number of on-chain outputs the base layer can create and enforce, and growth beyond that ceiling must come from custodians and semi-custodial designs. With them, channel factories, symmetric channels and more efficient shared-output schemes could carry self-custody much further. This is a question about Bitcoin's governance and appetite for change, and it cannot be answered by Lightning developers alone. The second is whether the remaining protocol-level vulnerabilities, most of all channel jamming and the mempool-level attacks on HTLC resolution, can be closed without sacrificing permissionless access. The current defenses raise the cost of attack considerably. Whether they hold under determined, well-funded adversaries is not yet known, because such adversaries have not yet had strong reasons to try. The third is whether demand materializes. The infrastructure for everyday Lightning payments now exists, including at the terminals of a major point-of-sale provider. Whether customers choose to use it, in bitcoin or in dollar-denominated assets routed over Lightning, depends on matters far outside the protocol: price stability, tax treatment, the quality of competing payment systems and the trust people place in them. The El Salvador experience suggests that a working rail is necessary but far from sufficient. The lesson of the design Lightning's most enduring contribution may be less the network itself than the lesson of its design. It shows that a scarce, slow, globally verified ledger can underpin a fast, cheap payment system without anyone having to trust a bank, provided the payment system is built from agreements the ledger can enforce. It also shows that every step away from global verification, however clever, brings the obligations of enforcement closer to the individual user: to be present, to be funded, to be ready to act in time. Those obligations can be shared, delegated and automated, and the history of Lightning is a history of doing exactly that. They cannot be wished away. A reader who keeps them in view will understand why Lightning works as well as it does, why it is harder than it looks, and where its next improvements must come from. Further Reading Andreas M. Antonopoulos, Olaoluwa Osuntokun and René Pickhardt, Mastering the Lightning Network: A Second Layer Blockchain Protocol for Instant Bitcoin Payments (O'Reilly Media, 2021). The most thorough book-length technical treatment of Lightning, covering channel construction, HTLCs, onion routing, gossip and pathfinding in detail. The full text is also available openly on GitHub. Andreas M. Antonopoulos and David A. Harding, Mastering Bitcoin: Programming the Open Blockchain, third edition (O'Reilly Media, 2023). The standard technical reference on Bitcoin's base layer, including transactions, scripts, timelocks, Segregated Witness and Taproot, all of which Lightning depends on. Joseph Poon and Thaddeus Dryja, "The Bitcoin Lightning Network: Scalable Off-Chain Instant Payments" (2016). The paper that introduced revocable bidirectional channels and Hashed Timelock Contracts. Parts of its design have been superseded, but its reasoning about the security model remains essential. Christian Decker and Roger Wattenhofer, "A Fast and Scalable Payment Network with Bitcoin Duplex Micropayment Channels," in Stabilization, Safety, and Security of Distributed Systems (SSS 2015). An independent, contemporaneous design for bidirectional channels that clarifies the range of options available. Christian Decker, Rusty Russell and Olaoluwa Osuntokun, "eltoo: A Simple Layer2 Protocol for Bitcoin" (2018). The proposal for symmetric channel updates, now usually called LN-Symmetry, and the clearest statement of what SIGHASH_ANYPREVOUT would enable. The Lightning Network specifications, known as the BOLTs, maintained in the lightning/bolts repository on GitHub. The authoritative definition of the protocol. BOLT 2 (channel management), BOLT 3 (transaction formats), BOLT 4 (onion routing), BOLT 7 (gossip), BOLT 11 (invoices) and BOLT 12 (offers) are the most relevant to this book. René Pickhardt and Stefan Richter, "Optimally Reliable and Cheap Payment Flows on the Lightning Network" (2021). The key paper on probabilistic pathfinding and the treatment of multipart payments as a flow problem, which reshaped how implementations route. Jona Harris and Aviv Zohar, "Flood and Loot: A Systemic Attack on the Lightning Network," in Proceedings of the 2nd ACM Conference on Advances in Financial Technologies (AFT 2020). A careful analysis of how mass channel closures can be weaponized against HTLC deadlines. Ayelet Mizrahi and Aviv Zohar, "Congestion Attacks in Payment Channel Networks," in Financial Cryptography and Data Security (FC 2021). The foundational analysis of channel jamming and its economics. Antoine Riard and Gleb Naumenko, "Time-Dilation Attacks on the Lightning Network" (2020). An analysis of how isolating a node from the Bitcoin network can cause it to miss enforcement deadlines. Bitcoin Optech, weekly newsletter and topic index (bitcoinops.org). The best ongoing record of technical developments in Bitcoin and Lightning, including replacement cycling, anchor outputs, splicing, BOLT12, package relay and covenant proposals, written for engineers but accessible to careful general readers. Hashtags: #LightningNetwork #BitcoinLightningNetwork #BitcoinScaling #BitcoinLayer2 #Layer2Scaling #BitcoinMicrotransactions #Micropayments #InstantBitcoinPayments #OffChainPayments #PaymentChannels #BidirectionalPaymentChannels #HashedTimelockContracts #HTLC #OnionRouting #SphinxProtocol #BitcoinBlockchain #BlockchainScalability #BitcoinTransactions #BitcoinPayments #LightningRouting #ChannelLiquidity #InboundLiquidity #OutboundLiquidity #MultipartPayments #LightningNetworkSecurity #BitcoinSelfCustody #LightningWatchtowers #LightningInvoices #BOLT12 #DecentralizedPayments

Download the Book (PDF): Introduction Most students with ADHD already know what they are supposed to do. They know that the essay due Friday should be started on Monday. They know that the phone belongs in another room during study time, that sleep matters, that the reading will not do itself. Many of them could give a better lecture on time management than the person who runs the orientation workshop, because they have been hearing that lecture since middle school. The problem was never a shortage of advice. The problem is that knowing and doing are separate operations, and ADHD lives in the gap between them. This book is built on that distinction. Its argument is simple to state and harder to live by: for a student with ADHD, the work of organizing, starting, remembering, and pacing should be moved out of the head and into the environment, the schedule, and the people around you, and it should be paced by energy rather than by willpower. Strategies that depend on trying harder, remembering better, or pushing through will work for a week and then fail, and the failure will be read as a character flaw. Strategies that change the setting in which effort happens tend to last. And any system, however clever, that runs on sprints followed by collapse will eventually reproduce the very burnout it was supposed to prevent. That second half of the argument deserves emphasis, because it is the part most productivity advice leaves out. Students with ADHD are often capable of extraordinary bursts of work. The all-nighter before a deadline, the six-hour session of hyperfocus on a project that finally caught their interest, the weekend spent reorganizing an entire semester in a fit of resolve: these are real and sometimes impressive. They are also expensive. The pattern of boom and bust, of heroic catch-up followed by days of depletion, is one of the most common routes into exhaustion, anxiety, and the quiet decision that college is not for people like you. A good system is not one that lets you do more in a burst. It is one that lets you do enough, steadily, without borrowing against next week. Who this book is for This book is written for university and college students who have been diagnosed with ADHD, who suspect they may have it, or who simply recognize themselves in the description of a mind that struggles with starting, sustaining, and finishing. It is also useful for the people who support them: parents, partners, roommates, advisers, and friends who want to understand why the obvious solutions keep not working. The scope is deliberately narrow. The book is about behavioral and environmental strategies: how to manage energy across a day and a term, how to build an external system for remembering, how to get started on tasks that feel impossible to begin, how to arrange your surroundings so that attention has a fighting chance, how to organize a semester that arrives as a pile of syllabi, how to handle the emotional side of ADHD that often does more damage than the attention problems, and how to find and use support. It does not attempt to cover diagnosis in depth, and it discusses medication only briefly and neutrally, as a decision that belongs to you and a qualified prescriber. Medication helps many people with ADHD and is well supported by research, but it is not the subject of this book, and nothing here is meant to replace it or to argue for or against it. This book is educational. It is not a substitute for individual assessment, diagnosis, or treatment by a qualified clinician, and it cannot account for your particular health history. If you are struggling, the campus counseling center, student health service, or disability services office is a good first call, and later chapters explain how to use them well. What ADHD looks like at university Estimates of how many college students have ADHD vary with how the condition is measured. A widely cited review by George DuPaul and colleagues found that roughly 2 to 8 percent of college students report clinically significant levels of ADHD symptoms, and that ADHD accounts for a substantial share of students registered with disability services. Those figures come from studies that differ in method, so treat them as a rough sense of scale rather than a precise count. What they suggest is that in any large lecture hall there are likely to be several students dealing with the same difficulties you are, most of them silently. University is a particularly demanding environment for an ADHD brain, and it is worth being precise about why. In secondary school, a great deal of executive function is supplied from outside. Someone sets the timetable. Teachers remind you about homework. Parents notice when you have not left for school. Assignments come in small pieces with frequent checkpoints. Then, in a single summer, most of that scaffolding disappears. A course might have three graded assignments and a final exam across fifteen weeks. Nobody takes attendance. The schedule has gaps of three hours between classes that must be filled by self-direction. Meals, laundry, sleep, money, and medical appointments become your responsibility at the same moment as the academic workload rises. For a student whose difficulties are precisely with self-direction, this is not a neutral transition. It removes the supports exactly where they were doing the most work. This is why many students with ADHD who did well enough at school hit trouble in their first or second year. It is also why some students are first diagnosed at university, when a lifetime of compensating strategies finally stops being enough. If that describes you, it is not evidence that you were fine before and have become worse. More often it means the environment changed, and the change exposed something that had been quietly managed by structure. How to use this book The chapters build on one another, but each can be read on its own. Chapter 1 sets out the core idea that ADHD is a problem of performance rather than knowledge, and explains what that means for choosing strategies. Chapter 2 treats energy as the resource that has to be managed before time can be managed at all, covering sleep, movement, food, and the hidden cost of hyperfocus. Chapter 3 shows how to build an external brain: a small, trusted system for capturing and scheduling what you would otherwise have to remember. Chapter 4 addresses the hardest single moment in ADHD, which is starting. Chapter 5 turns to the physical and digital environment. Chapter 6 applies everything to the practical problem of organizing a chaotic semester. Chapter 7 deals with emotion, shame, and the burnout cycle. Chapter 8 covers accommodations, therapy, coaching, and the brief, careful word on medication that a book like this owes its readers. A few principles run through all of it. First, fewer tools used consistently beat many tools used enthusiastically. The student with eleven productivity apps is usually worse off than the student with one paper notebook. Second, every strategy should be judged by what happens in week eight, not in week one. Novelty is a powerful motivator for ADHD brains, and almost anything works while it is new. Third, the goal is not to become a person without ADHD. It is to become a person with ADHD who has arranged their life so that the condition costs less. Do not try to implement everything at once. That impulse, the urge to overhaul your entire life on a Sunday afternoon, is itself one of the patterns this book will ask you to notice. Read a chapter, pick one change, run it for two weeks, and see what survives. The strategies that last are rarely the most elaborate. They are the ones that fit the way your mind actually works, rather than the way you wish it did. A final word about tone. Much of what is written about ADHD swings between two extremes: a deficit story, in which the condition is a list of failures to be corrected, and a superpower story, in which it is a hidden gift that only needs to be unleashed. Neither is useful. ADHD is a real, well-documented condition that causes genuine impairment for many people, and it also coexists with curiosity, energy, creativity, and a capacity for intense engagement. You do not need to love it or hate it. You need to understand how it works in your own life well enough to stop fighting it on its strongest ground and start meeting it where you can win. Chapter 1: The Performance Problem Consider a second-year history student who has a four-page paper due on Thursday. On Sunday evening she opens her laptop, reads the prompt twice, and decides she will start properly on Monday after class. On Monday she spends forty minutes looking for the right article, gets absorbed in a tangent about medieval trade routes, and ends the day with a folder of downloaded sources and no words written. Tuesday disappears into a lab report and a friend's crisis. On Wednesday night, with the deadline suddenly vivid, she writes the entire paper between ten and four in the morning, fueled by coffee and dread. It is not bad. It might even be good. She sleeps through her Thursday morning class, feels wretched for two days, and promises herself that next time will be different. Nothing in that story involves a lack of knowledge. She knew the deadline, knew the assignment, knew how to write a history paper, knew that starting early would be wiser. She did not lack intelligence, and she was not lazy in any meaningful sense: she worked hard, just at the wrong time and at a terrible cost. What failed was not information. What failed was the connection between knowing what to do and doing it at the moment it needed to be done. Knowing versus doing The psychologist Russell Barkley, whose work has shaped much of the modern understanding of adult ADHD, puts this idea at the center of his account. ADHD, in his framing, is not primarily a disorder of knowing what to do but of doing what you know. It impairs the capacity to use information about the future to guide behavior in the present. The student above had the information. What she lacked, on Monday afternoon, was the internal pressure that makes a Thursday deadline feel real on a Monday. This has a practical consequence that Barkley states directly: effective help for ADHD has to operate at the point of performance. That phrase, the point of performance, is worth holding onto. It means the place and moment where the behavior needs to happen. The point of performance for starting an essay is your desk on Monday at two o'clock, not the orientation seminar in September where someone explained the value of early starts. Advice delivered far from the point of performance tends to evaporate. Supports that are present at the point of performance, such as a reminder that appears on the screen, a friend sitting beside you, a timer running, or a plan written on a card taped to the laptop, have a much better chance of changing what happens. Much of this book is an elaboration of that one idea. If the problem is performance at the moment of action, then the solution is to put something useful at that moment. Motivational talks, resolutions, and better understanding all have their place, but they operate at a distance. The strategies that tend to work for ADHD shorten that distance until the help and the task are in the same room. Executive functions and the sense of time The phrase executive function appears often in discussions of ADHD, and it is used loosely enough to become fuzzy. For practical purposes, think of executive functions as the set of mental abilities that let you direct your own behavior toward goals over time. Researchers carve them up differently, but most lists include some version of the following: holding information in mind while you use it (working memory); stopping yourself from acting on the first impulse (inhibition); shifting between tasks or approaches (flexibility); planning and organizing steps toward a goal; regulating emotion enough to keep going; and sensing and managing time. In ADHD, these abilities are not absent, but they are less reliable, and they are especially unreliable when a task is boring, distant, or unstructured. That last point is important. Many students with ADHD are puzzled, and their families are frustrated, by how well they can focus on some things. They can play a demanding video game for hours, rebuild a bicycle, or read an entire novel in a night. This apparent contradiction leads people to conclude that the student could focus on coursework if they really wanted to. The more accurate description is that ADHD affects the regulation of attention, not the amount of it. Attention flows readily toward what is novel, urgent, interesting, or immediately rewarding, and it is very hard to steer toward what is none of those things. The skill that is impaired is precisely the skill of pointing attention where you choose rather than where it goes by itself. ADHD also rarely travels alone. Anxiety, depression, sleep problems, and specific learning difficulties such as dyslexia are all more common in people with ADHD than in the general population, and the international consensus statement published by the World Federation of ADHD in 2021 lists these associations among its well-supported conclusions. This matters for strategy. A student whose avoidance is driven mainly by anxiety needs something a little different from a student whose avoidance is driven mainly by boredom, even if the two look identical from outside: both leave the essay untouched, but one is flinching from a threat and the other is drifting toward stimulation. If your difficulties seem larger than the strategies in this book can reach, or if low mood, panic, or persistent sleeplessness are part of the picture, that is a reason to talk to a clinician, not a sign that you have failed at self-help. This is why deadlines work so well for students with ADHD and so badly at the same time. A deadline that is tomorrow is urgent, and urgency supplies the activation that the student cannot generate internally. The all-nighter is not a failure of attention. It is attention finally arriving, because urgency has made the task stimulating enough to engage with. The trouble is that urgency is a brutal fuel. It arrives late, it is accompanied by stress, and it leaves no margin for error, illness, or a second assignment due the same day. One of the most useful ideas for understanding ADHD at university is what clinicians sometimes call time blindness. It describes a weakened sense of time as something that passes and approaches. For many people with ADHD, time is divided into two zones: now, and not now. Things in the now zone feel real and pressing. Everything else, whether it is due next week or next month, feels roughly equally distant and equally abstract. A deadline in ten days and a deadline in thirty days can feel almost identical until one of them crosses into the now zone, at which point it feels like an emergency. This explains several patterns that otherwise look like carelessness. It explains why a student can be genuinely surprised by an exam they have known about for weeks. It explains chronic lateness, because the ten minutes needed to walk across campus are not felt in advance. It explains underestimating how long tasks will take, since the experience of doing a task is not easily projected forward. And it explains why planning ahead, the universal advice, feels strangely weightless: the future events you are planning for do not carry emotional force until they are close. The implication is not that you should try harder to feel time. It is that time needs to be made visible and external. Analog clocks that show the passage of time as a shrinking space, timers that count down, calendars that lay out the whole term on one wall, and backward plans that turn a distant deadline into a series of near ones all serve the same function. They drag the future into the now zone in small, manageable pieces, so that urgency arrives gradually rather than all at once. Later chapters will return to specific ways of doing this. For now, it is enough to see that the problem is structural, and so the solution must be structural too. Why the standard advice fails Most study-skills advice is designed for students whose executive functions work reasonably well and who mainly need information. It says: make a to-do list, use a planner, break big tasks into small ones, avoid distractions, start early. None of this is wrong. For a student with ADHD, though, each item quietly assumes the very capacities that are impaired. A to-do list assumes you will remember to look at it. A planner assumes you will consistently write in it and consult it. Breaking a task into small pieces assumes you can do the planning required to see what the pieces are, which is itself an executive task. Avoiding distractions assumes you can inhibit the pull toward them. Starting early assumes that a distant deadline can generate motivation. When a student with ADHD follows this advice and it fails, the usual conclusion is that they did not follow it properly or did not try hard enough. So they try harder, buy a more beautiful planner, download a new app, and make a fresh resolution. For a week or two the novelty supplies enough stimulation to keep the new system running. Then the novelty fades, the planner goes unopened, the app sends notifications that are swiped away, and the cycle concludes with a slightly deeper sense of personal failure. Many students with ADHD arrive at university with years of this pattern behind them and a firm belief that they are fundamentally disorganized people for whom systems do not work. The better conclusion is that the systems were designed for a different kind of mind. What works for ADHD has a recognizable shape. It reduces the number of things that must be remembered. It puts cues directly at the point of performance. It makes the next action obvious and small. It uses external structure, including other people, rather than relying on internal self-direction. It makes the invisible visible, whether that is time, progress, or the steps of a task. And it tolerates imperfection, because any system that collapses after one missed day will collapse, since missed days are guaranteed. There is one more piece to add before the practical chapters begin, and it concerns cost. Students with ADHD who are doing reasonably well at university are often doing so through enormous hidden effort. They reread instructions four times because they do not trust themselves to have caught everything. They stay up late to make up for time lost to distraction during the day. They spend energy masking restlessness in lectures, rehearsing what they will say in seminars, and checking and rechecking that they have their keys, their laptop, their ID. They manage deadlines through anxiety, letting fear of failure provide the urgency that interest does not. This compensation can work, sometimes for years. But it runs on a finite supply of energy, and it tends to fail suddenly rather than gradually. A student who has been holding things together through sheer effort can reach a point, often in the middle of a demanding term or after an illness or a personal setback, where the effort simply is not available any more. Grades fall, emails go unanswered, and the student may withdraw from classes or from friends. From the outside this can look like a sudden collapse. From the inside it is usually the end of a long period of overspending. This is why the argument of this book joins organization and energy. A strategy that helps you get more done but increases the total effort you are spending is not a good strategy for ADHD; it moves the collapse forward. The right question to ask about any technique is not only whether it works but what it costs to keep running. A reminder system that fires automatically costs almost nothing to maintain. A study session with a friend may actually give energy rather than consume it. A plan that asks you to override your natural rhythms every day, by contrast, may produce results for a while and then leave you more depleted than before. What this means in practice If ADHD is a problem of performance rather than knowledge, and if compensating through effort is costly and fragile, then a few working rules follow. They are worth stating plainly, because the rest of the book applies them to specific areas. First, externalize. Anything that relies on remembering, noticing, or feeling the future should be moved outside your head, into a calendar, a list, a visible object, an alarm, or an arrangement with another person. Your working memory is not a safe place to store commitments. Second, act at the point of performance. When you are designing a strategy, ask where and when the behavior needs to happen, and put the support there. The reminder to review your notes belongs on the lock screen of your phone at the moment you sit down on the bus, not in a resolution made the night before. Third, make the next step small and obvious. Most of the pain of ADHD procrastination is concentrated in the moment of starting, and starting is much easier when the first action is concrete and short. Fourth, borrow structure. Classes, study groups, library hours, meal times, and other people's schedules are all structures you can attach yourself to. Using them is not a crutch. It is how most people, with or without ADHD, get things done. Fifth, pace by energy. Plan your hardest work for times when your energy is naturally higher, protect sleep, and treat recovery as part of the work rather than as a reward you earn afterward. Sixth, expect the system to break and design it to restart. The goal is not perfect consistency but quick recovery. A student who misses three days of a routine and returns to it on the fourth is doing well. A student who misses three days and abandons the whole system because it is ruined is caught in an all-or-nothing trap that ADHD makes very easy to fall into. Before changing anything, it helps to find out where your own performance actually breaks down, because the answer differs from student to student. A simple way to do this is a two-week breakdown log. Keep a single note on your phone, and each time something goes wrong, whether a missed deadline, a task you could not start, a class you slept through, or an evening that vanished, write one line: what was supposed to happen, what happened instead, and at what moment the plan came apart. Do not analyze while you log. At the end of two weeks, read the list and look for clusters. Some students discover that nearly all their failures happen at transitions, in the gap between one class ending and the next task beginning. Others find that the trouble is almost entirely in the evening, when energy is low and the phone is close. Others find that tasks are started well enough but abandoned at the first point of confusion. Each of these patterns points to a different place to put support, and knowing which one is yours saves weeks of trying fixes aimed at someone else's problem. The log has a second benefit that is easy to underestimate. It turns a vague and shaming story, I am just bad at everything, into a specific and workable one, I lose the afternoon when I go back to my room between classes. The first story invites resolutions. The second invites a change of route. The student with the history paper did not need to be told that starting early was better. She needed a way to make Thursday's deadline feel real on Monday, a first step small enough to begin without dread, and a setting that did not offer a dozen easier things to do instead. She also needed to stop paying for her papers with her sleep and her health. The chapters that follow take each of these needs in turn, starting with the one on which all the others depend: energy. Chapter 2: Energy Before Time Time management is the phrase everyone uses, and it points in slightly the wrong direction. You cannot manage time; an hour is an hour whether you use it well or not. What you can manage is the state you are in when the hour arrives. A student who sits down to study at eight in the evening after five hours of sleep, no proper meal since breakfast, and a day of lectures has the same sixty minutes as a student who sits down rested and fed, but they do not have the same hour. For a student with ADHD, whose capacity to direct attention is already less reliable, the difference is magnified. Poor sleep, hunger, and depletion do not just make everything a bit harder. They remove the thin margin of self-regulation that was making anything possible at all. This chapter treats energy as the resource that comes first. It covers sleep, which is the foundation and the place where ADHD most often does quiet damage; movement, which is one of the best-supported non-drug ways to improve attention; food and caffeine, which are simpler than the internet suggests; and the pattern of boom and bust, including the hidden cost of hyperfocus. The aim is not a perfect wellness routine. It is a small number of habits that stop energy leaking away before the work even begins. Sleep: the foundation that ADHD undermines Sleep problems are strikingly common in ADHD. A 2019 review by Denise Bijlenga and colleagues reported that delayed sleep phase, a pattern in which the body clock runs late so that the person cannot fall asleep until well after midnight and struggles to wake in the morning, is highly associated with ADHD in both children and adults, with the studies they reviewed suggesting it affects a large majority of people with the condition. The exact proportion varies between studies and depends on how sleep is measured, but the direction of the finding is consistent: many people with ADHD have body clocks that run late. The authors went further and suggested that some of what is labeled ADHD may be driven or worsened by chronic sleep disruption, which is a reminder that the relationship runs both ways. Poor sleep worsens attention, and difficulties with self-regulation make good sleep harder to achieve. At university, this biological tendency meets an environment designed to exploit it. Nobody enforces a bedtime. Residence halls are loud until late. The most interesting social life happens at night. Screens are always available, and the internet is engineered to be endlessly engaging for exactly the kind of novelty-seeking brain that ADHD produces. Then the nine o'clock lecture arrives, and the student who fell asleep at two drags through it on six hours of broken sleep, absorbing little, and resolves to catch up in the evening, which pushes bedtime later again. There is no single fix, but a few changes carry most of the benefit, and they work best as a package. The first and most powerful is a consistent wake time. It is tempting to focus on bedtime, but for someone with a delayed body clock, going to bed early often means lying awake for hours, which is miserable and teaches the brain that bed is a place for restless thinking. Wake time is easier to control and does more to anchor the body clock. Choose a time you can hold on most days, including weekends within an hour or so, and hold it even after a bad night. The sleepiness that follows a short night will help you fall asleep earlier the next evening. Sleeping until noon on Saturday to recover feels restorative but pushes the whole cycle later, and Monday morning pays for it. The second is morning light. Bright light soon after waking is one of the strongest signals the body clock receives, and it tends to shift the clock earlier. A walk outside, breakfast by a window, or simply opening the curtains fully as soon as the alarm goes can help. The mirror image applies at night: dimmer, warmer light in the last hour or so before bed, and less bright screen time, makes it easier for sleepiness to arrive. Bijlenga and colleagues discuss light and timing as part of the treatment options for delayed sleep, and some clinicians use carefully timed light or other chronotherapy approaches. Those should be arranged with a clinician rather than improvised. The third is a wind-down routine that does not rely on willpower at the moment of greatest weakness. At eleven at night, the decision to put the phone down competes with a feed that has been designed by skilled engineers to keep you scrolling. You will lose that contest more often than not. The practical move is to decide earlier and arrange the environment so the decision is already made. Charge the phone across the room or outside the bedroom, which also forces you to get up to turn off the alarm. Use the phone's built-in downtime settings to restrict apps after a set hour. Keep a paper book or an audiobook by the bed as a low-stimulation alternative. The aim is not a monastic bedroom but a gentle slope toward sleep rather than a cliff. If you try these changes for several weeks and still cannot fall asleep until very late, wake unrefreshed however long you sleep, snore heavily, or feel sleepy enough in the day to nod off in lectures, raise it with a clinician. Sleep disorders are treatable, and they are common enough in ADHD that they are worth checking for rather than assuming poor sleep is just part of the package. If you take medication for ADHD, its timing can also affect sleep, and that is a conversation for your prescriber rather than something to adjust on your own. Movement, food, and caffeine Physical activity is one of the best-supported non-medication strategies for attention, although the strength of the evidence depends on the population studied. A 2015 meta-analysis by Cerrillo-Urbina and colleagues pooled randomized trials of exercise programs in children and adolescents with ADHD and found that short-term aerobic exercise was associated with moderate to large improvements in attention, hyperactivity, impulsivity, anxiety, and executive function. Those trials were small and involved children, so the results should not be stretched to promise a particular effect for university students. But the general finding, that moving your body tends to help attention and mood for a while afterward, is consistent with a broader body of research, and it matches the experience of many adults with ADHD who find that a run, a swim, or a brisk walk before studying makes the next hour noticeably easier. The practical question is not whether exercise is good for you, which you already know, but how to get it to happen at the point of performance. A few approaches work well for ADHD. Attach movement to the start of study sessions, so that a twenty-minute walk to the library, taking the long route, becomes the warm-up rather than a separate task competing for time. Use movement as a break inside long sessions, since a few minutes of stairs or stretching between blocks of work can reset attention more effectively than scrolling. Choose forms of exercise that carry their own structure and social obligation, such as a class at a fixed time, an intramural team, or a regular run with a friend, because a scheduled commitment with another person is far more reliable than an intention to go to the gym when you feel like it. And lower the bar: ten minutes of real movement that happens beats an hour-long workout that stays in the planner. Food is simpler. The main problem for many students with ADHD is not what they eat but that they forget to eat, or eat irregularly, or skip meals during hyperfocus and then crash in the late afternoon. Some ADHD medications also reduce appetite during the day, which is worth discussing with a prescriber if it is affecting you. The most useful habit is to make regular meals part of the external structure rather than something that depends on noticing hunger. Eat at fixed times tied to your class schedule. Keep simple, filling food in your bag so that a long day on campus does not become eight hours on coffee. Breakfast that includes some protein tends to hold steadier than a sugary pastry, though you do not need a strict diet to benefit. Claims that specific diets or supplements cure ADHD should be treated with caution. Some have been studied, and the evidence for most is limited or mixed. If you are curious about a particular approach, discuss it with a clinician rather than an influencer. Caffeine deserves a special word, because many students with ADHD use it heavily and some describe it as calming rather than stimulating. Moderate caffeine is safe for most healthy adults and can genuinely help alertness. The problems come from timing and quantity. Caffeine lingers in the body for hours, so an energy drink at four in the afternoon can still be affecting sleep at midnight, which then produces the tiredness that tomorrow's caffeine is meant to fix. A reasonable rule is to take caffeine early in the day and stop by early afternoon, and to notice whether you are using it to compensate for chronically short sleep. High-dose energy drinks, especially combined with alcohol or with prescribed stimulant medication, carry real risks, including a racing heart and anxiety, and anyone taking ADHD medication should ask their prescriber how caffeine fits with it. Boom, bust, and the price of hyperfocus The most important energy pattern for students with ADHD is not about any single habit. It is the rhythm of boom and bust. Because interest and urgency supply the activation that everyday tasks do not, ADHD tends to produce work in surges. When a deadline is close or a subject catches fire, a student can work for hours with remarkable intensity. This state is often called hyperfocus, and it can feel wonderful. The world narrows, time disappears, and work that was impossible yesterday flows easily. Hyperfocus has real value, and it would be wrong to treat it as a symptom to be eliminated. But it has costs that are easy to miss because they are paid later. During a long hyperfocus session, people often skip meals, ignore thirst, stay up far past their usual bedtime, and neglect other obligations entirely. The session ends with a finished project and a depleted body. The next day or two are foggy and flat, other work has piled up, and the pile creates the next emergency, which triggers the next surge. Over a term, this rhythm can become a cycle of crisis and exhaustion in which the student is always either sprinting or recovering and never simply working. Table 1 sets out several of the most common energy drains for students with ADHD, the early warning signs that each is starting to cost you, and a practical counter for each. None of the counters is dramatic. The point is to catch drains early, before they compound. Table 1. Common energy drains and practical counters. Energy drain Why it bites harder with ADHD Early warning sign Practical counter Late nights Body clock often runs late; screens are hard to put down Sleeping through morning alarms Fixed wake time; phone charged outside the bedroom Skipped meals Hunger goes unnoticed during focus Irritability or crash in late afternoon Meals at fixed times; food packed in bag Hyperfocus marathons Stopping is as hard as starting Losing track of hours; missed plans Hard stop alarm; water and food set out beforehand Deadline sprints Urgency supplies motivation that interest does not Several deadlines landing in one week Backward planning into smaller deadlines Constant switching Each restart costs effort Ending the day busy but with nothing finished Grouped tasks; fewer open tabs and apps Masking in social or class settings Effort spent appearing calm and attentive Exhaustion after ordinary days Recovery time scheduled after demanding blocks The central move against boom and bust is to put limits on the booms, which feels counterintuitive. When work is finally flowing, stopping seems wasteful. But a session that runs until three in the morning borrows from the next two days, and it is usually a bad trade. Some practical ways to cap hyperfocus include setting a hard-stop alarm before you begin, placed across the room so that you must stand up to silence it; setting out water and a snack beforehand so that basic needs do not depend on noticing them; and deciding in advance on a stopping point defined by output, such as finishing the introduction, rather than by exhaustion. Stopping while you still know what the next step is has an added benefit: it makes the next session much easier to start, because you are not facing a blank page. The other half of the move is to take the busts seriously as signals. If you routinely need a day or two to recover after heavy work, that is information about your real capacity, not evidence of weakness. Planning with that information, by spreading work across more days at a lower intensity, will almost always produce more total output over a term than a sequence of heroic sprints. This is the essence of pacing, and it is the same principle that athletes and people managing long-term health conditions use: work at a level you can sustain, so that you are not constantly paying back debt. A useful exercise is to think of your week as having a limited energy budget and to look honestly at what spends it. Lectures, seminars, social events, commuting, part-time work, and emotional demands such as a difficult relationship or a family problem all draw from the same account as studying. A student who has a heavy lab day on Tuesday, a shift at work on Tuesday evening, and a club meeting after that should not be surprised that Wednesday morning's study plan does not happen. The failure is not in Wednesday; it is in a Tuesday that spent more than the account held. Seeing energy this way makes it possible to plan more realistically, to protect recovery time after demanding days, and to stop blaming yourself for running out of something that was always going to run out. Consider how this looks in practice for a first-year engineering student who works two evening shifts a week at a campus café. For most of his first term he planned study as if every day were the same, and every Wednesday and Friday morning, after his shifts, the plan failed. He concluded that he had no discipline in the mornings. When he mapped his week by energy instead of by hours, the pattern was obvious. His shift days were his most expensive days, and the mornings after them were his lowest. He moved his hardest problem sets to Monday and Thursday mornings, when he was rested, used the post-shift mornings for lighter tasks such as reorganizing notes and watching recorded lectures, and blocked Saturday morning as unscheduled recovery. He did not study more hours than before. He studied the same hours in a better order, and his sense of failing every Wednesday disappeared. The lesson generalizes: before you add more study time, check whether the time you already have is placed where your energy can meet it. Energy, then, comes before time. The chapters that follow describe systems for capturing tasks, starting work, and organizing a semester. Every one of them works better on a rested brain with steady fuel, and every one of them breaks more easily on a depleted one. If you change only one thing after reading this book, a consistent wake time is a strong candidate. It is unglamorous, and it does more than most productivity tools to make a good day possible. Chapter 3: Building an External Brain Every student with ADHD has lived some version of this moment. You are walking out of a lecture when the professor mentions, almost in passing, that the reading for next week has changed. You think, I must remember that. Thirty seconds later a friend catches your arm to talk about the weekend, and the thought is gone. It does not resurface until next week's seminar, when everyone else has read the right chapter and you have read the wrong one. The failure here is not carelessness. It is working memory doing what working memory does in ADHD: holding information briefly and then dropping it when something new arrives. Working memory is the mental space where you keep information active while you use it, and it is small for everyone. In ADHD it is often less reliable, especially under distraction. Asking it to hold a list of commitments over days and weeks is like storing important papers on the dashboard of a moving car. Some of them will stay there for a while. Many will not, and you will not notice which ones have gone until you need them. The solution is to stop using your head as a storage system and to build an external one instead. This chapter describes how to do that in a way that suits ADHD: with as few parts as possible, with cues placed where they will be seen, and with a simple routine that keeps the whole thing from decaying. The rule of one: capture, calendar, and list The productivity writer David Allen, in Getting Things Done, made a point that is especially useful for ADHD: the mind is for having ideas, not holding them. Every open commitment that lives only in your head takes up some mental attention, and a head full of half-remembered obligations produces a constant low hum of anxiety. Moving those commitments into a trusted external system does two things. It makes them less likely to be forgotten, and it frees attention for the task in front of you. Allen's full system is elaborate, and many people with ADHD find it too heavy to maintain. The core idea, however, can be reduced to three components, each of which should exist in exactly one place. The first is a capture point: one place where every new commitment, idea, or piece of information goes the moment it appears. For most students this should be the phone, because the phone is almost always within reach, but it could be a small notebook that lives in your pocket. The important thing is that it is always available and that capturing takes seconds. A single note or a simple task app with a quick-add button works well. When the professor changes the reading, you take out the phone and type four words before your friend reaches you. You do not organize it at that moment. You just catch it. The second is a calendar: one place where everything that happens at a specific time lives. Classes, exams, appointments, shifts, deadlines, and the blocks of time you have set aside for particular work all belong here. A digital calendar is usually best for students because it can send reminders and sync across devices. The calendar is the most important part of the system for ADHD, because it makes time visible. A task list tells you what needs doing. A calendar tells you when, and it shows you honestly whether there is room. The third is a task list: one place where things that need doing, but not at a specific time, live. This list should be short enough to scan in under a minute. Long lists produce paralysis, because every item is visible at once and the brain cannot tell which matters most. A useful trick is to keep a master list somewhere out of sight and a short list, perhaps five items, for today or this week, which is what you actually look at. The rule of one matters more than the choice of tool. The most common way that ADHD organizational systems fail is not that the tool is wrong but that there are too many tools. A student has a paper planner, a calendar app, a task app, sticky notes on the desk, a whiteboard in the room, and an email inbox full of flagged messages. Each one holds some commitments, and none of them holds all of them. Checking the system means checking six places, which never happens, so the system is not trusted, so things are stored in the head again, and the whole structure quietly collapses. One capture point, one calendar, and one list, checked in one short routine, is enough. There is one more source of commitments that deserves attention, because it is where many students with ADHD lose track of things without realizing it: the inboxes they do not control. University email, the course management platform, group chats for each class, and messages from clubs and employers all generate tasks. An announcement on the course page that moves a deadline, or an email from an adviser asking for a form, is a commitment just as much as something a professor says aloud. The mistake is to treat these inboxes as part of the system, as if having read the message were the same as having recorded it. It is not. A message read at midnight on a phone and then scrolled past is effectively forgotten. The rule is that anything requiring action gets moved into the calendar or the list at the moment you read it, even if that means typing three words into the capture note. The inbox is a mailbox, not a filing cabinet. If you cannot deal with a message properly at the moment you see it, capture the action and leave the message behind. Consider how a first-year student might set this up in the first week of term. She creates one calendar and enters every class with its room, then every deadline from every syllabus, each with a reminder a week before and another two days before. She puts one note on her phone's home screen called Inbox and teaches herself to type into it whenever anything new comes up. She keeps a small paper card in her pocket with no more than five items for the day, copied from her master list each morning while her coffee brews. She deletes the three other task apps she downloaded over the summer. None of this takes more than an afternoon, and none of it is clever. Its strength is that there is exactly one answer to the question of where something is written down. Choosing tools is a matter of fit rather than virtue. Table 2 compares the most common options on what each does well and where each tends to fail for students with ADHD. Many people end up with a digital calendar, a phone-based capture note, and either a digital or paper short list, but the right combination is the one you will actually open. Table 2. Common organization tools compared for ADHD. Tool Best used for Main strength Main risk with ADHD Digital calendar Classes, deadlines, time blocks Reminders arrive at the point of performance Notifications ignored if there are too many Phone capture note or task app Catching new tasks in seconds Always within reach Phone opens onto distractions Paper notebook or planner Daily short list, planning by hand Tangible and distraction-free Left at home; forgotten when novelty fades Wall calendar or whiteboard Seeing the whole term or week at once Makes time visible without opening anything Becomes wallpaper that the eye stops seeing Phone alarms with labels Transitions and time-critical actions Hard to miss; labeled with the action Alarm fatigue if used for everything Put the cue where the action happens A system that holds everything but is never seen is only slightly better than no system. The second principle, taken directly from the idea of the point of performance, is that the information must reach you at the moment you need it, without you having to remember to go looking. Digital calendars help here because they can send reminders. The trick is to use reminders sparingly and precisely. A reminder that says Essay at nine in the morning is easily dismissed. A reminder that fires ten minutes before you are due to leave for the library and says Leave now, bring the essay notes, laptop charger is much more likely to produce action, because it names the behavior and arrives when the behavior is needed. Labeled alarms work the same way. An alarm at 8:40 that says Walk to Chem lab, goggles in bag does more than a vague sense that you have a lab this morning. Physical cues are just as powerful. Many people with ADHD benefit from a launch pad: a single spot near the door where everything needed for leaving the house lives, such as keys, ID card, wallet, headphones, and the bag already packed for tomorrow. Packing the bag the night before and putting it on the launch pad moves the work of remembering from the frantic morning to the calmer evening, and it means that walking out the door with the bag is enough to take everything. Similarly, if you need to return a library book, put it on top of your shoes, not on the shelf where you will not see it. The same logic applies to studying. If you plan to work on a particular assignment tomorrow, leave the materials open and ready on your desk tonight, with a sticky note on top that states the first action, for example Read pages 12 to 20 and underline the argument. When you sit down tomorrow, the cue is right there, the decision about what to do has already been made, and the barrier to starting is lower. This is a small thing, and small things are exactly where ADHD strategies earn their keep. Visibility also matters for longer horizons. A wall calendar or large whiteboard showing the whole term, with every major deadline and exam marked, makes distant dates visible in a way that a phone calendar, which usually shows a day or a week, does not. Seeing that three assignments cluster in week nine, while it is still week three, is the first step to doing something about it. The risk with wall displays is that the eye stops seeing them after a few weeks, so it helps to change them regularly, for example by crossing off each day or moving a marker, so that they remain alive. The daily look and the weekly reset Every external system decays. New commitments arrive, plans change, and within a week or two a system that is not maintained becomes out of date and untrustworthy. For a student with ADHD, this decay often happens without being noticed, and by the time it is noticed the system feels too messy to rescue. The remedy is two short routines: a daily look and a weekly reset. The daily look takes five minutes or less. At a fixed time, ideally attached to something you already do every day such as breakfast or the first coffee, open the calendar and look at today and tomorrow. Check the capture point and move anything new to the calendar or the list. Pick the one to three things that most need to happen today. That is all. It is not a planning session; it is a glance at the map before setting off. Attaching it to an existing routine matters because a new habit needs something to hang on. Planning to do the daily look at some point in the morning will fail. Doing it while the kettle boils has a chance. On days when the list is long and everything seems urgent, a quick way to choose is to ask of each item what happens if it is not done today. Some things carry a real consequence, such as a submission that closes at midnight or a form that loses you a place in a course. Others only feel urgent because they are visible. Do the consequential ones first, and give yourself explicit permission to leave the rest for tomorrow's list. The feeling that everything must be done at once is one of the fastest routes to doing nothing, because the brain, faced with twenty equal demands, often picks the twenty-first option of opening a video instead. The weekly reset takes longer, perhaps thirty to forty-five minutes, and happens at the same time each week. Sunday afternoon or Friday before leaving campus both work well. During the reset you go through the capture point and empty it, look at the whole coming week in the calendar, check the course pages and syllabi for anything due in the next two or three weeks, add time blocks for the work those deadlines require, update the short list, and tidy the physical launch pad and desk. Many students find this routine much easier to do with another person, such as a friend doing their own weekly reset at the same table, because the social commitment supplies the activation that an abstract intention does not. Finally, and perhaps most important, build in a restart rule. Your system will lapse. There will be a week of illness, a family emergency, or simply a stretch where everything slides. When this happens, the ADHD instinct is often all or nothing: the system is ruined, so abandon it and start a new one later, perhaps with a new app. Resist this. Decide in advance that when you notice the system has lapsed, you will do one emergency reset, only fifteen minutes long, in which you look at the next seven days, capture everything you can think of that is overdue, and pick one thing to do today. You do not need to reconstruct everything you missed. You need to get back on the map. A system that is restarted ten times in a term is working. A system that is abandoned after one lapse and replaced with a shinier one is the old cycle again. It is worth saying something about the lure of new tools, because it is one of the most reliable traps in ADHD. Setting up a new app or buying a new planner is interesting. It produces a burst of motivation and a satisfying sense of progress, and for a week or two the new system seems to work beautifully. What is actually working is the novelty. When it fades, the new system follows the old one into disuse. The practical defense is to choose tools that are boring and flexible, to commit to them for at least a full term before judging them, and to notice when the urge to switch is really an urge for stimulation. If you enjoy tinkering, set aside a fixed time once a term for system redesign and keep the rest of the term for using what you have. An external brain does not make ADHD go away. It does something more modest and more useful. It takes the job your working memory is worst at, holding commitments across time, and gives it to something that does it reliably. That frees your actual brain for the things it is good at: thinking, making connections, and working with energy once the work is underway. The next chapter turns to the moment where that underway so often fails to arrive: the moment of starting. Chapter 4: Starting Is the Hard Part Ask a group of students with ADHD which part of academic work is hardest, and very few will say the work itself. Once they are writing, the essay often goes reasonably well. Once they are in the problem set, they can usually get through it. The hardest part, by a wide margin, is the moment before: the stretch of time, sometimes minutes and sometimes days, in which the task sits there waiting and they cannot make themselves begin. That stretch is where hours are lost, where guilt accumulates, and where the rest of the day often goes to die. This chapter is about that moment. It explains why starting is so hard for ADHD brains, and it offers a set of practical techniques for getting across the gap, most of which work by making the first step smaller, making the start automatic, or borrowing activation from outside yourself. None of them requires you to feel motivated first. That is the point. Waiting to feel like it is a strategy that works when interest is high and fails reliably when interest is low, which is exactly when you need a strategy most. Why starting hurts From the outside, failing to start a task looks like choosing not to. From the inside it rarely feels like a choice. It feels more like pushing against a wall that nobody else can see. Understanding what that wall is made of makes it easier to get through. Part of it is the regulation of attention described in Chapter 1. Tasks that are not novel, urgent, interesting, or immediately rewarding generate very little pull, and for an ADHD brain very little pull can mean almost none. An essay due in a week has no urgency yet. Reading a dense article is not novel. The reward, a grade, is distant and abstract. With nothing drawing attention toward the task, attention drifts toward whatever does offer something: the phone, a conversation, a sudden conviction that the room needs tidying. Another part is emotional. Many tasks carry feelings with them, and those feelings are often unpleasant. A paper for a course you are struggling in carries anxiety about failure. An email to a professor about a missed deadline carries embarrassment. A long-avoided task carries the accumulated shame of all the previous days you did not do it. Avoiding the task avoids the feeling, and that brings immediate relief, which reinforces the avoidance. Over time, a task that was merely dull becomes genuinely aversive, because it has collected so much emotional residue. This is one reason that the longer something is put off, the harder it becomes to start. A third part is uncertainty. Tasks that are vague, such as work on the project, study for the exam, or sort out housing, are very hard to begin because there is no obvious first action. The brain has to do the planning work of figuring out what the first step is, and planning is itself an executive task that ADHD makes costly. Faced with a fuzzy task, the easiest thing to do is something else. Seeing these three components, low pull, unpleasant feelings, and unclear first steps, suggests three kinds of solution. You can make the first step so small and clear that it barely requires activation. You can decide in advance exactly when and where you will start, so that the decision is not left to the moment of weakness. And you can bring in outside sources of pull, such as other people, time pressure, or novelty, to supply what the task itself does not. Shrink the first step and decide in advance The single most useful technique for task initiation is to shrink the first action until it is almost embarrassingly small. Not write the essay, not even write the introduction, but open the document and type the title. Not study for the chemistry exam, but take out the chapter 4 notes and read the first page. The first step should be concrete enough that you could photograph yourself doing it, and short enough that you can finish it in two or three minutes. This works for a reason that is partly psychological and partly practical. Psychologically, the wall described above is highest at the very beginning. Once you are in motion, even slightly, the task becomes more familiar, the unpleasant feelings often shrink, and the next step becomes visible. Practically, a tiny first step removes the planning burden. You do not need to know how the whole essay will go in order to type the title. Many students find that once they have typed the title, they write a sentence, and once they have written a sentence, they write a paragraph. Not always, and that is fine. The agreement with yourself is only to do the tiny step. If you stop after it, you have still done more than yesterday, and tomorrow's start will be slightly easier. Related to this is defining what done looks like for the session. Study for two hours is an open-ended commitment that is easy to dread and hard to judge. Complete the practice questions for section 3.2 is finite. Knowing the finish line in advance reduces the sense of an endless task and gives you a clear point at which you are allowed to stop and feel that you succeeded. The second technique is to decide in advance exactly when and where you will begin. Research on what psychologists call implementation intentions, a form of planning that links a specific situation to a specific action, suggests this is one of the more effective simple tools for turning intentions into behavior. The format is an if-then plan: if it is two o'clock on Monday and I am at my library desk, then I will open the essay document and type the title. A widely cited 2006 meta-analysis by Peter Gollwitzer and Paschal Sheeran, pooling results from many studies of if-then planning, found a medium-to-large effect on goal attainment compared with simply forming the intention. Those studies were mostly not specific to ADHD, and results vary with the task and the person, but the logic suits ADHD particularly well. An if-then plan moves the decision from the moment of action, when self-regulation is weakest, to an earlier calm moment, and it attaches the start to a cue in the environment rather than to a feeling. A practical way to use this is to write your if-then plans into the calendar itself. Instead of a block labeled Essay, write Library, 2pm: open essay doc, type title and first heading. When the reminder fires, it does not ask you to decide anything. It tells you what the plan was. It also helps to anticipate the most likely obstacle and plan for it. If the usual problem is that you check your phone first and lose twenty minutes, the plan might add: if I reach for my phone, then I will put it in my bag and zip the bag. This sounds almost too simple to matter, and yet the evidence on if-then planning and the experience of many students suggest that spelling out the response to a predictable temptation in advance makes it more likely to happen. To see how these pieces fit together, consider a third-year psychology student facing a research proposal that has been sitting untouched for eleven days. Every time she thinks of it, she feels a wave of dread, partly because the task is large and vague and partly because she has already missed a check-in with her supervisor. On Sunday, during her weekly reset, she does three things. She writes the smallest possible first step on a sticky note and puts it on her closed laptop: open the proposal template and paste in the three articles I already found. She adds an if-then plan to her calendar for Monday at eleven, when she has a gap between classes and is usually still fresh: if it is eleven and I am at the third-floor library table, then I open the template. And she anticipates the obstacle she knows best, which is that she will open her email first, see the unanswered message from her supervisor, and spiral. So her plan includes a second line: if I feel pulled to check email, I will write the reply to my supervisor in two sentences first, then close the inbox. On Monday, the reply takes four minutes. It says she is behind, apologizes briefly, and proposes a meeting. The template takes another ten. By half past twelve she has a rough outline, and, more important, the task has stopped being a source of dread and has become an ordinary piece of work. Nothing about her motivation changed. What changed was the size of the first step, the certainty of when it would happen, and the removal of the obstacle she knew was coming. Borrow activation from outside The third family of techniques draws on outside sources of pull. Since ADHD brains are activated by urgency, novelty, interest, and the presence of other people, it makes sense to bring these into tasks that lack them. Other people are perhaps the most powerful source. Many people with ADHD find they can work far more easily when someone else is present, even if that person is doing something unrelated and says nothing. This is often called body doubling. It has not been studied much in controlled research, so claims about how and why it works should be held loosely, but it fits long-standing findings in social psychology that the presence of others can affect performance, and it is widely reported by adults with ADHD and by clinicians who work with them. In practice, body doubling can mean studying in a library or café rather than alone in your room, arranging to work with a friend at a set time, joining a study group that works silently together, or using a video call in which two people keep their cameras on while each does their own work. The social commitment gets you to the table, and the presence of the other person helps keep you there. Time pressure is another source, and it can be manufactured in small doses. A timer that counts down creates a mini deadline and turns an open-ended task into a bounded one. The best-known version is the Pomodoro Technique, devised by Francesco Cirillo, which alternates twenty-five minutes of focused work with a five-minute break. The specific numbers are not sacred. Some students with ADHD find twenty-five minutes too long to begin with and do better with ten or fifteen; others find that once they are in flow, the timer interrupts them unhelpfully, in which case they can let it run. What matters is the structure: a short, defined burst with a visible end and a planned break. A physical timer that shows time as a shrinking colored disc can be better than a phone, both because it makes time visible and because it avoids picking up the phone at all. Novelty can be injected deliberately. Moving to a different location for different subjects, working in a new café, switching from typing to handwriting, or using a different color of pen for a dull review task can all supply a small boost of interest. The economist Katherine Milkman and colleagues have described an approach called temptation bundling, in which something you enjoy is paired only with something you tend to avoid, for example listening to a favorite podcast only while doing laundry or allowing yourself a particular coffee only at the library. In their original study the approach increased gym attendance, at least for a while. It will not transform a hated task into a beloved one, but it can take the edge off. Interest can sometimes be found inside the task itself. Students with ADHD often do their best work when they can connect an assignment to something they genuinely care about. If a paper topic is open, choose one that sparks curiosity even if it seems slightly harder. If it is fixed, look for the angle that interests you, the argument you disagree with or the application to something in your own life. Asking yourself what you actually find interesting about the material, before starting, can create a thread to pull on. Finally, there are accountability structures that combine several of these at once. Telling a friend that you will send them your first paragraph by five o'clock creates a social deadline. Booking an appointment at the writing center for Thursday means you have to have something to show by Thursday. Asking a professor for feedback on an outline by a certain date turns one distant deadline into two closer ones. Each of these converts an internal intention, which ADHD handles poorly, into an external commitment, which it handles much better. A special word about the tasks that are small but emotionally heavy, such as the email to a professor, the phone call to the financial aid office, or the form for the housing office. These often sit untouched for weeks, not because they are hard but because each one carries a small dose of embarrassment or anxiety, and the delay makes the embarrassment grow. Two tactics help. The first is to batch them: set aside one short, regular slot each week, perhaps thirty minutes on Friday morning, as an administration hour in which you do nothing but these tasks, ideally with a friend doing their own. The second is to lower the stakes of the writing by using simple templates. An email to a professor about an extension needs only three parts: what happened, in one sentence without excessive detail or apology; what you are asking for; and a proposed new date. Having that structure in a saved note removes much of the agony of composing it, and professors generally respond better to a short, clear, early request than to a long apologetic one sent after the deadline. Staying started Starting is the hardest part, but it is not the only hard part. Many students with ADHD get going and then stall at the first point of confusion, or drift away after twenty minutes when the initial momentum fades. A few techniques help keep the work going once it has begun. Keep a parking lot. When a distracting thought arrives, such as a sudden memory that you need to email someone, a question you want to look up, or an idea for a different assignment, write it on a piece of paper beside you and return to the task. The thought is captured, so you do not need to act on it now or worry about losing it. At the end of the session, you deal with the list. Plan for the stuck point. When you hit a part of the task that is confusing, the ADHD reflex is often to switch to something else, which can mean abandoning the whole session. Instead, give yourself a rule: when stuck, write down precisely what you are stuck on, as a question, and then either skip to a different part of the task or set a five-minute timer to work on the question before seeking help. Writing the question down turns a vague feeling of being lost into a specific problem, and it gives you something concrete to ask a tutor or classmate. Stop at a good place. As Chapter 2 suggested, ending a session while you still know what the next step is makes the next start far easier. Some writers deliberately stop in the middle of a sentence so that picking up the thread requires no warm-up at all. At the least, before you stop, write a note to your future self stating the next action. The next session then begins with a tiny, obvious step rather than a blank. Forgive the gaps. There will be sessions where you sit down, do the tiny first step, and still cannot continue. That is not a failure of the technique. On some days, energy or mood simply is not available, and a few minutes of genuine effort followed by a planned break is a better outcome than an hour of guilt-filled half-work. Note what happened in your log, protect your next start, and move on. The long-term aim is not to win every session but to make starting less of an event: something that happens routinely at known times and places, by known small steps, until the wall that once seemed immovable becomes more like a door that sometimes sticks. Chapter 5: Designing Your Environment for Attention Willpower is a poor tool for managing attention, and it is a particularly poor tool for managing an ADHD brain. Every time you resist a distraction by effort alone, you spend something, and the distractions do not run out. The phone buzzes again. The roommate starts a conversation. A new tab offers something more interesting than the article. Over an afternoon, dozens of small acts of resistance add up, and the student who began the day determined ends it having done little except resist. A more reliable approach is to change the environment so that fewer acts of resistance are needed. This is not a trick specific to ADHD. Much of what is known about habit and behavior change points in the same direction: the easiest way to do something more is to make it easier, and the easiest way to do something less is to make it harder. For a student with ADHD, the payoff is larger, because the internal brakes are weaker and the external setting does more of the work. This chapter looks at how to use that principle in practice: in the design of friction, in the management of the phone, in the choice of study places, and in the handling of sound, movement, and the lecture hall. Friction as a design tool Friction is anything that adds effort, time, or steps between you and an action. A snack in the kitchen cupboard has more friction than a snack on the desk. A social media app that requires you to log in each time has more friction than one that opens instantly. Small amounts of friction make surprisingly large differences in behavior, because many actions are impulsive, and an impulse that meets a barrier often fades before the barrier is crossed. The design principle is simple: add friction to the things you want to do less, and remove friction from the things you want to do more. For studying, this means making the start of work as frictionless as possible, with materials laid out, the document already open, the charger already plugged in, and the water bottle already filled, while making distractions slightly harder to reach. Some concrete examples show how small the changes can be. Log out of social media on your laptop and do not save the password, so that opening it requires typing credentials. Move tempting apps off the phone's home screen into a folder on the last page. Put the gaming console controller in a drawer in another room during the week. Keep the television remote somewhere inconvenient. Install a website blocker on your laptop that blocks a short list of time-draining sites during study hours; several free and paid options exist, and the specific one matters less than setting it up once and then leaving it alone. None of these makes distraction impossible, and none needs to. The goal is to put a pause between impulse and action, long enough for the part of you that wanted to study to have a say. The same thinking applies in reverse to the things you want to do. If you want to review flashcards on the bus, put the flashcard app on the phone's home screen where the social media app used to be. If you want to go to the gym in the morning, sleep in your gym clothes or put them on top of your phone. If you want to read before bed, leave the book on the pillow. Each of these removes a small decision or a small search, and small decisions and searches are exactly where ADHD intentions tend to leak away. An important refinement is to design for your worst moments rather than your best. When you are rested and motivated, almost any setup works. The environment earns its value on the evening when you are tired, anxious, or bored, and the pull toward distraction is strongest. Ask yourself what your worst-moment self will do, and arrange things so that the path of least resistance for that self leads somewhere acceptable. The phone problem For most students, the phone is the single largest source of distraction, and for students with ADHD it is often something close to a constant companion that interrupts every task. This is not a personal failing. Smartphones and the apps on them are designed by skilled teams to capture and hold attention, using exactly the features that ADHD brains find most compelling: novelty, variable rewards, social feedback, and endless new content. A student trying to resist all of that by willpower is in an unfair fight. There has been research suggesting that the mere presence of a smartphone nearby might reduce available cognitive capacity, even when it is turned off. A 2017 study by Adrian Ward and colleagues reported this effect, which they called brain drain, and it attracted considerable attention. Later attempts to replicate the finding have produced mixed results, so it should not be treated as settled. What is well established from ordinary experience is simpler: a phone within reach is a phone that gets picked up, and each pickup costs time and focus beyond the moment itself, because returning to a demanding task after an interruption takes effort. The most effective intervention is also the most obvious: put the phone somewhere else. Not face down on the desk, which still leaves it one movement away, but in a bag, zipped, on the other side of the room, or in another room entirely. Many students resist this because they are worried about missing something important. It helps to be precise about what that could be. If you are expecting a genuinely urgent call, you can set the phone to allow calls from a small list of contacts while silencing everything else. Everything else can wait forty minutes. Beyond distance, a handful of settings make the phone much less grabby. Turn off all notifications except those from people and a few essential services such as your calendar; nearly every app sends far more notifications than you need. Use the phone's built-in focus or do-not-disturb modes, scheduled to switch on automatically during your study blocks so that you do not need to remember to activate them. Some people find that switching the screen to grayscale makes apps noticeably less appealing, though the effect varies from person to person. Setting daily time limits on particular apps adds a layer of friction, although these limits are easy to override and work best as reminders rather than barriers. Some students go further and use a separate device for study, such as an old laptop with no social media logged in or a basic timer instead of the phone's clock, so that the phone never needs to be touched during work. Others use apps that lock the phone for a set period. These heavier approaches can be very effective for students whose phone use is severely disrupting their work, and they are worth trying if lighter adjustments are not enough. The laptop is a phone in disguise. The same principles apply to browsers and computer apps: close everything not needed for the current task, use a single window where possible, turn off email notifications while you work, and consider a blocker for the few sites that most often take you away. If your course requires research online, it can help to separate searching from reading: collect the sources in one short burst, then close the browser and work from the downloaded files. Choosing and shaping places to study Where you study matters, and the right answer is different for different people and different tasks. Many students with ADHD default to studying in their bedroom, because it is convenient, and then discover that the bedroom is full of cues for everything except studying: the bed invites lying down, the console invites gaming, the room is associated with relaxing, and nobody else is there to provide the gentle pressure of shared work. Other students find libraries too quiet, and the silence makes every small sound unbearable. The practical approach is to treat location as a tool and to experiment deliberately. Try different places for a week or two each, and note in your log where work actually happened. Table 3 summarizes the typical strengths and drawbacks of common study settings, which can serve as a starting point for your own experiments. Table 3. Common study settings compared. Setting What it offers Main drawback Often best for Bedroom desk Convenience; no travel Cues for rest and play; isolation Short tasks; late planning Library quiet floor Low noise; others working nearby Silence can magnify small distractions Reading and focused writing Café or busy common area Background buzz; novelty Noise and interruptions vary Routine tasks; first drafts Study group room Social commitment; help on hand Can drift into socializing Problem sets; exam review Different spot for each subject Location cues the task More travel and planning Students taking varied courses Two ideas help in choosing. The first is to match the setting to the task. Deep reading or writing a difficult argument may call for the quietest place available, while working through routine practice problems or formatting references might go best in a busier setting where a little background stimulation keeps you engaged. The second is to link places to activities, so that the place itself becomes a cue. If a particular library table is where you write, and you do nothing else there, then sitting down at that table begins to trigger writing more easily over time. Mixing activities in one place, studying at the same desk where you watch films and scroll, weakens that link. Public study places also bring the benefit discussed in Chapter 4: other people. Being surrounded by people working creates a mild social pressure to work too, which is body doubling on a larger scale. It also adds a small cost to leaving, since packing up and walking out is more of a decision than wandering from a desk to a bed. For many students with ADHD, simply committing to work in a library rather than at home during set hours is one of the most effective changes they can make. If you must study at home, shape the space to support it. Create a work zone, even if it is only one end of a desk, that is kept clear of unrelated objects. Face a wall rather than a window or the door if movement pulls your eye. Keep only the materials for the current task on the desk, and put the others out of sight. At the end of each session, spend two minutes resetting the zone so that it is ready for next time. The goal is a space that says one thing: this is where work happens. Shared living adds another layer. Roommates and housemates are part of your environment, and a friendly roommate who drops by to chat every twenty minutes can undo the best-designed desk. It is worth having a direct conversation early in the term rather than resenting interruptions in silence. Agree on a simple signal, such as headphones on meaning please do not interrupt unless it is urgent, or a sign on the door during study blocks. Propose regular quiet hours that suit both of you. If you share a room, discuss where each of you will work at which times, so that one person's late-night study does not collide with the other's sleep. Most roommates are glad to have clear rules rather than guessing, and explaining that you have ADHD, if you are comfortable doing so, often makes the conversation easier rather than harder. You do not owe anyone that disclosure, but it can turn a request that might sound fussy into one that makes obvious sense. To see how the pieces combine, consider a second-year economics student who had been doing almost all his studying in bed with his laptop, the phone beside him and a streaming service open in another tab. He was putting in long hours on paper and getting little done. Over two weeks he made four changes. He moved all problem sets to a group study room booked with two classmates on Tuesday and Thursday afternoons. He claimed a particular carrel on the library's quiet floor for reading, and went there straight after his morning lecture so that there was no trip home in between. He put a blocker on his laptop covering three sites during those hours, and left his phone in his bag in a locker. At home, he stopped working in bed altogether and kept the bedroom for sleep and rest. None of this was dramatic, and he did not suddenly become a different student. But the number of hours he needed to finish the same work fell noticeably, and his evenings, which had been a guilty blur of half-study and half-scrolling, became genuinely free time. The environment had done the work that his willpower had been failing to do. Sound, movement, and the lecture hall Sound is personal. Some people with ADHD need near silence to concentrate. Others find silence oppressive and do better with background noise, which may mask small, irregular sounds that would otherwise capture attention. Research on music and white noise for attention in ADHD has produced mixed results, and individual differences are large, so the practical approach is to experiment. Try noise-canceling headphones in a busy space, instrumental music without lyrics, steady background noise such as rain or a fan, or silence, and notice which conditions go with productive sessions. Music with lyrics, especially songs you know well, tends to compete with reading and writing for the same mental resources, so save it for tasks that do not involve language. Movement is also personal and often misunderstood. Many people with ADHD think better when some part of the body is moving: tapping a foot, fidgeting with an object, pacing while reading, or standing. In the past, this restlessness was treated as something to suppress. A more helpful view is that small, controlled movement can help some people regulate attention, and that suppressing it takes effort that could go into the task. Quiet fidget objects, a standing desk or a high table in the library, a chair that allows some rocking, or the habit of reading while walking slowly in a hallway are all options. The test is simple: does it help you work, and does it avoid disturbing others? Lectures pose their own challenges, because the environment is set by someone else and you cannot pause it. A few adjustments help. Sit near the front, where there are fewer people and screens in your line of sight and where the lecturer's presence provides a stronger anchor. Take notes actively, by hand if possible or in a structured format on a laptop with other apps closed; the act of deciding what to write keeps attention engaged in a way that passive listening does not. Prepare briefly before each lecture, even five minutes spent skimming the slides or the reading, so that the material has something to attach to. If your university records lectures, use the recordings to fill gaps, but do not treat them as a substitute for attending, because the recording removes the structure that attendance provides and is easily postponed indefinitely. If you find that lectures are consistently difficult despite these adjustments, this is exactly the kind of challenge that formal accommodations can address, for example through permission to record, access to slides in advance, or the use of a note-taker. Chapter 8 discusses how to request these. A well-designed environment does not demand anything of you in the moment. That is its value. It does its work quietly, before you arrive, so that when your worst-moment self sits down to study, the easiest path already leads toward the work. Most of the changes described here take minutes to set up and then run on their own. The next chapter brings together the external brain, the starting techniques, and the environment into the largest organizational challenge of university life: a whole semester. Chapter 6: Organizing a Chaotic Semester A university semester arrives in pieces. In the first week you receive five or six syllabi, each with its own format, its own grading scheme, and its own set of deadlines scattered across fifteen weeks. There are readings, problem sets, quizzes, labs, midterms, papers, presentations, and a final exam or project, some of them listed with dates and some of them announced later. There are also the parts of life that no syllabus mentions: a job, rent, family obligations, friendships, health appointments, and the ordinary chores of keeping yourself fed and housed. For any student, pulling all of this into a workable plan is a real task. For a student with ADHD, whose sense of time tends to collapse into now and not now, it is the central organizational challenge of university life. The most common pattern is familiar. The first few weeks feel light, because the big deadlines are all in the not-now zone. Then, somewhere around week six or seven, several of them cross into the now zone at once, and the student discovers that two papers, a midterm, and a lab report all fall within the same ten days. The rest of the term becomes a sequence of emergencies, each one handled at the expense of the next, until the final exam period arrives as a wall of accumulated debt. This chapter describes how to prevent that pattern by doing a small amount of planning at the right times and in the right order: mapping the whole term at the start, working backward from big deadlines, giving each week a shape, and knowing what to do when things go wrong anyway. Week one: map the whole term The single most valuable hour of planning in any semester is the one you spend in the first week putting every known deadline into one place. It is tedious, which is exactly why it rarely happens, and it pays back many times over. Sit down with every syllabus and the course pages for each class. Go through them one at a time and enter every graded item into your calendar on the date it is due: every paper, problem set, quiz, lab, presentation, midterm, and final exam. Include the weight of each item as a percentage in the entry, if the syllabus gives it, so that you can see at a glance which deadlines matter most. Also enter things that are not graded but have dates, such as the last day to drop a class without penalty, registration windows for next term, and housing or financial aid deadlines. These administrative dates are exactly the kind of thing that ADHD lets slip, and missing them can be expensive. When everything is entered, step back and look at the whole term at once. A wall calendar or a single printed page showing all fifteen weeks is ideal for this, because it lets you see the shape of the semester rather than one week at a time. Look for pinch points: weeks where several major deadlines cluster. Almost every semester has at least two, usually around midterms and in the final weeks. Mark them clearly. These are the weeks where the term will be won or lost, and knowing about them in week one gives you time to do something about it. What you do about a pinch point is simple in principle: move work out of it. If a paper and a midterm fall in week eight, then the paper needs to be mostly written by the end of week seven, so that week eight can be devoted to exam preparation. This is not about being virtuous. It is about recognizing that the week-eight version of you will not have enough time or energy to do both, and that the week-five version of you has plenty of both but no sense of urgency. The planning in week one is what transfers some of week eight's urgency back to week five. It is also worth noticing, in this first look, whether the term as a whole is realistic. Students with ADHD often take on more than they can manage, partly out of optimism about time and partly because it is hard to feel the weight of future commitments. If the map shows a term with five demanding courses, a twenty-hour-a-week job, and a leadership role in a club, the problem is not planning. The problem is load, and it is better to see it in week one, when you can still drop a course or reduce your hours, than in week ten, when the only options are bad ones. In some countries and institutions, students with a documented disability can request a reduced course load while keeping full-time status for purposes such as financial aid or housing; this is common in the United States, but the rules vary widely by country and institution, so ask your disability services office what applies to you. Work backward from every big deadline Once the term is mapped, the next step is to break each large deadline into a chain of smaller ones. This is backward planning, and it is one of the most effective tools for time blindness, because it turns one distant deadline that feels unreal into several near deadlines that feel real. Take a research paper due in week ten. Working backward, you might decide that the final version should be finished by the end of week nine, leaving a few days of margin; a full draft by the end of week eight; an outline by the end of week six; sources gathered and read by the end of week five; and a topic chosen by the end of week three. Each of these becomes its own entry in the calendar, with its own reminder, and each one is small enough to feel achievable. The week-ten deadline has become a week-three deadline, which is close enough to feel. Two refinements make backward planning work better for ADHD. The first is to account for the planning fallacy, the well-documented tendency for people to underestimate how long tasks will take. People with ADHD often underestimate more severely, because the experience of doing a task is hard to project forward. A practical response is to look at your own history. Your breakdown log from Chapter 1 and your memory of past terms are better guides than your optimism. If papers have usually taken you longer than you planned, assume this one will too, and build the chain with that margin already in place. The second refinement is to attach the mini-deadlines to something outside yourself wherever possible. A self-imposed deadline that nobody else knows about is easy to let slide. A mini-deadline that involves another person is much harder to ignore. Book an appointment at the writing center for the week the outline is due. Ask the professor or teaching assistant during office hours whether you can bring a draft introduction by a certain date. Agree with a classmate to exchange drafts. These external anchors turn backward planning from a private intention into a structure with other people in it. Backward planning works at different scales, and it helps to see how the planning levels fit together. Table 4 sets out the four horizons that most students need to manage, from the whole term down to a single study session, along with the tool that suits each, when to review it, and the question to ask. Table 4. Four planning horizons for a semester. Horizon Main tool When to review Key question Term Wall calendar or one-page term map Week one; again at midterm Where are the pinch points? Week Digital calendar with time blocks Weekly reset What must move forward this week? Day Short list of one to three priorities Daily look What matters most today? Session Sticky note with first step and finish line At the start of each block What exactly am I doing now? The value of the table is not in its detail but in its reminder that planning happens at several levels, each with a different rhythm. The term map is done once and revisited occasionally. The weekly plan is built during the weekly reset. The daily priorities come from the daily look. The session plan is written just before you begin. Students with ADHD often plan intensively at one level, typically in a burst of term-long resolution, and neglect the others. The system works when each level feeds the next. The term map also deserves a second look at midterm. By week seven or eight, deadlines will have moved, new assignments will have been announced, and your sense of which courses are demanding will be far more accurate than it was in week one. Spend half an hour redrawing the remaining weeks: update dates, mark the pinch points still ahead, and check whether any course needs more time than you planned. This is also the moment to be honest about any course that is going badly. A mid-term review that reveals a failing grade in week seven leaves room to act, whether through extra help, a conversation with the professor, or a withdrawal before the deadline. The same discovery in week fourteen leaves almost none. Give each week a shape A semester is made of weeks, and the week is where planning becomes behavior. The aim is not a rigid schedule in which every hour is assigned. Rigid schedules tend to fail for ADHD, because a single disruption, a class that runs late or a morning lost to poor sleep, breaks the whole plan and triggers the all-or-nothing response. The aim instead is a week with a recognizable shape: fixed anchors, blocks of protected work time, a buffer, and deliberate recovery. Anchors are the fixed events of the week: classes, labs, shifts at work, regular meetings, and, importantly, meals and sleep. Put these in the calendar first. They are the skeleton on which everything else hangs, and they provide the external structure that ADHD needs. Work blocks come next. Look at the gaps between anchors and assign specific blocks to specific kinds of work, preferably at consistent times each week. If Tuesday and Thursday mornings are free and you are fresh then, make those your writing blocks. If Monday afternoon falls between two classes on campus, make it a library reading block. Consistency helps because repeated times and places become habits, and habits require less activation than decisions. Label each block with what it is for and, where possible, with the first step, as described in Chapter 4. Buffer is the element most often missing. Something will go wrong most weeks: an assignment will take longer than planned, a friend will need help, you will have a bad day. If every hour is already assigned, these events push everything else back and the week collapses. Leave at least one or two unassigned blocks, perhaps Friday afternoon, as a catch-up space. If the week goes well, the buffer becomes free time. If it goes badly, the buffer absorbs the damage. Recovery belongs in the calendar as deliberately as work. Chapter 2 made the case for pacing, and the week is where pacing happens. Schedule time that is genuinely off: exercise, social time, hobbies, or simply rest. Protect it the way you would protect a class. Students with ADHD often feel they have not earned rest until everything is finished, which, given how rarely everything is finished, means rest never comes, and the result is chronic depletion with lower output. Planned rest is not a reward for good behavior. It is maintenance. A common trap is the week that looks fine on paper but ignores energy. A calendar with four hours of lectures, a three-hour lab, a work shift, and two study blocks might fit within the available hours, but if all of these fall on the same two days, those days will be exhausting and the study blocks will not happen. When shaping a week, spread demanding commitments where you can, put the hardest cognitive work where your energy is highest, and assume that the hour after a draining activity will be less productive than the same hour on a quiet day. When things go wrong anyway No plan survives a whole semester intact. Illness, family crises, a bad stretch of mood, or simply the accumulated drift of a busy term will knock you off course at some point. What matters is how quickly and how calmly you recover. The first rule is to act early. The instinct, when an assignment starts to slip, is often to avoid thinking about it, which in practice means avoiding the professor, avoiding the course page, and sometimes avoiding the class itself. This makes everything worse. Most professors have far more flexibility, and far more sympathy, for a student who writes a week before a deadline to say they are struggling than for one who writes the day after. If you can see a problem coming, say so. A brief, clear message, as described in Chapter 4, is usually enough to open the conversation. The second rule is to triage. When several things are behind at once, you cannot catch up on all of them simultaneously, and trying to will leave you in the paralyzed state where everything is urgent and nothing moves. Sit down with the term map and rank what is overdue or imminent by consequence. Which items carry the most weight in your grade? Which have hard deadlines with no flexibility? Which could be negotiated, reduced, or, in the worst case, dropped? Then deal with them in order, one at a time, and accept that some lower-priority items may be done less well than you would like. A term in which you submit a good paper and a passable lab report is far better than one in which you attempt both perfectly and submit neither. The third rule is to use the institution's safety valves. Universities have formal mechanisms for students in difficulty, such as extensions, incomplete grades, late withdrawal from a course, or medical leave, although the names and rules vary widely by country and institution. These exist precisely for situations where a term has gone wrong for reasons that were hard to control. Students with ADHD sometimes feel that using them is cheating or giving up. It is neither. If you are registered with disability services, your adviser there can help you understand the options and speak to your professors. If you are not registered, an academic adviser or dean of students office can usually help. The important thing is to ask before the options close, since many of them have deadlines of their own. Group projects deserve a brief mention, because they combine several of the hardest things for ADHD: shared deadlines, coordination, and dependence on others. They also offer an advantage: other people. The most useful move is to volunteer early for a clearly defined part of the project, one with a concrete output and a date, and to ask the group to set internal deadlines for each part. This turns the group into an accountability structure. If you know you tend to fall behind, a quiet word with one reliable group member, asking them to check in with you a few days before your part is due, can prevent a great deal of stress for everyone. Finally, the end of term deserves its own planning. The final exam period often combines the heaviest workload of the semester with the most disrupted routine: classes end, the usual anchors disappear, and many students find themselves with long unstructured days that are hard to fill productively. Before the period begins, make an exam schedule that recreates structure. Fix a wake time. Assign specific subjects to specific blocks each day. Plan where you will study and with whom. Build in meals, movement, and sleep as non-negotiable. And plan a day or two of genuine rest after the last exam, before the next term begins, because the pattern of finishing one term exhausted and starting the next one already depleted is one of the quiet ways the boom-and-bust cycle extends itself across a whole degree. Chapter 7: Emotions, Shame, and the Burnout Cycle So far this book has treated ADHD mainly as a matter of attention, memory, time, and energy. Those are the parts that are easiest to see and easiest to design around. But many students with ADHD, if asked what makes university hardest, will talk about something else: the feelings. The flash of frustration that makes it impossible to continue after a small setback. The crushing embarrassment after a missed deadline. The certainty, after a critical comment on a paper, that they are not cut out for this. The low, persistent shame of always being behind. These emotional experiences are not side issues. For many students they do more damage than the attention problems themselves, because they drive avoidance, erode confidence, and feed the cycle of crisis and exhaustion that leads to burnout. This chapter looks at the emotional side of ADHD: why emotion regulation is often harder, how shame and perfectionism shape behavior, how burnout develops and how to recognize it early, and what recovery and prevention look like in practice. It is also the chapter where the line between self-help and professional help matters most, and it will be clear about where that line falls. Emotion is part of the picture Difficulty regulating emotion is not listed among the core diagnostic symptoms of ADHD, which focus on inattention, hyperactivity, and impulsivity. But it is common enough in people with ADHD that researchers increasingly treat it as a central feature of the experience. A 2014 review in the American Journal of Psychiatry by Philip Shaw and colleagues concluded that emotion dysregulation is prevalent in ADHD across the lifespan and is a major contributor to impairment. The authors discussed several ways of understanding the overlap, including the possibility that it is a core feature of the condition for many people, and noted that the question was not settled. In everyday terms, emotion dysregulation in ADHD often looks like feelings that arrive fast, run hot, and take a long time to settle. A small frustration becomes a large one before there is time to put it in perspective. Disappointment can feel overwhelming. Excitement can also run high, which is part of what makes new projects so compelling at the start. The same weakness in inhibition that makes it hard to resist a distraction can make it hard to pause between feeling something and acting on it, whether that means sending an angry message, walking out of a study session after one bad problem, or abandoning a course after one poor grade. One aspect of this deserves particular attention: sensitivity to rejection and criticism. Many adults with ADHD describe intense emotional reactions to perceived rejection, disapproval, or failure, reactions that feel out of proportion even to themselves. In popular writing this is often called rejection sensitive dysphoria. That term is not a formal diagnosis and has not been well studied as a distinct entity, so it is best understood as a description of a common experience rather than as a scientific category. The experience itself, however, is real for many people, and at university it has practical consequences. A critical comment from a professor can lead a student to avoid that professor's office hours for the rest of the term. A classmate's offhand remark can make a seminar feel unbearable. Fear of disappointing someone can lead to avoiding them altogether, which of course produces exactly the disappointment that was feared. Knowing this about yourself is useful, because it creates a small gap between the feeling and the response. When a wave of shame or anger arrives after feedback, it helps to recognize it as a familiar pattern, name it, and delay action. A simple rule such as never replying to difficult emails on the same day, or waiting until after a walk before deciding to drop a course, can prevent many decisions that feel right in the heat of the moment and look wrong a day later. So can reading critical feedback in two passes: the first time just to absorb the emotional hit, and the second time, perhaps the next morning, to extract what is actually useful. Shame, perfectionism, and the stories you tell By the time they reach university, most students with ADHD have received a great deal of negative feedback. Years of being told to pay attention, to try harder, to stop being careless, to live up to their potential, leave a mark. Many students absorb these messages into a story about themselves: that they are lazy, disorganized, unreliable, or simply not as capable as other people. This story is usually wrong, since ADHD is a well-documented neurodevelopmental condition rather than a character flaw, and the international consensus statement on ADHD is emphatic on this point. But being wrong does not make the story powerless. It shapes how students respond to every setback, and it often does so in ways that make things worse. Shame is the most corrosive of these responses. Guilt says I did something bad and focuses on the action, which can be repaired. Shame says I am bad and focuses on the self, which feels beyond repair. When a missed deadline triggers shame rather than guilt, the natural response is not to fix the problem but to hide from it. The student avoids the professor, avoids the course page, avoids thinking about it at all, and the avoidance creates more missed deadlines, which create more shame. This cycle is one of the most common routes by which a manageable difficulty becomes a failed course. Perfectionism is a less obvious partner in the cycle, and it surprises many people to learn how common it is in ADHD. After years of being criticized for careless work, some students overcorrect. They set impossibly high standards, check and recheck everything, and cannot submit work until it is perfect. Since nothing is ever perfect, they either miss deadlines or submit at the last moment after exhausting all-nighters. Perfectionism also makes starting harder, because the gap between the imagined perfect product and the first rough sentence is so painful that it is easier not to begin. And it feeds the all-or-nothing thinking that makes systems fragile: if the plan is not followed perfectly, it has failed and might as well be abandoned. Changing these patterns is slow work, and for many students it is best done with a therapist, as Chapter 8 discusses. But some shifts can begin on your own. The findings from the ACCESS trials of cognitive-behavioral treatment for college students with ADHD, discussed in the next chapter, are relevant here: improvements in what the researchers called adaptive thinking, meaning more balanced and less self-defeating ways of interpreting events, were strongly associated with better outcomes, even though they were harder to achieve than improvements in practical strategies. In other words, the stories matter, and changing them pays off. Three practices help. First, separate the behavior from the self. When something goes wrong, describe it in specific terms, such as I missed the deadline because I did not start until the night before and then my laptop crashed, rather than global ones, such as I always ruin everything. Specific descriptions point toward solutions; global ones point only toward despair. Second, adopt a standard of good enough for most work. Decide in advance what a satisfactory version of an assignment looks like, aim for that, and reserve extra polish for the few pieces that really matter. A submitted B paper is worth far more than an unsubmitted A paper. Third, treat lapses as data. When a plan breaks down, ask what happened and what you would change, the way a coach reviews a game, rather than cataloguing your faults. The breakdown log from Chapter 1 is useful here too, because it makes the review factual rather than emotional. It also helps to talk to other students with ADHD. Hearing someone else describe exactly the pattern you thought was your private failure can loosen shame more effectively than any argument. Many universities have peer groups or student organizations for neurodivergent students, and campus disability services or counseling centers can often point you to them. The burnout cycle Burnout is a term used loosely, and it is worth being careful with it. In the World Health Organization's International Classification of Diseases, eleventh revision, burnout is described as an occupational phenomenon resulting from chronic workplace stress that has not been successfully managed, rather than as a medical condition. It is characterized by exhaustion, mental distance or cynicism about one's work, and reduced effectiveness. The definition was written for the workplace, but many students recognize the same pattern in their academic lives, and it is a useful frame for understanding what happens when the boom-and-bust rhythm continues too long. For students with ADHD, burnout usually builds through a recognizable sequence. It begins with compensation: working harder, staying up later, and relying on anxiety and urgency to get things done. For a while this works, and the student may even be praised for results. Then the costs accumulate. Sleep erodes, exercise and social life are cut to make time for work, and recovery periods shrink. Tasks begin to take longer because the student is tired. More compensation is needed to keep up, which increases the costs further. Eventually a threshold is crossed. Something relatively minor, such as a cold, a breakup, or one bad grade, tips the balance, and the capacity to compensate simply disappears. The student who was holding everything together finds they cannot open their laptop, cannot face their email, cannot get out of bed for a morning class. From outside, it looks sudden. From inside, it was the end of a long overdraft. Early warning signs are worth knowing because burnout is much easier to prevent than to reverse. Common ones include persistent tiredness that does not lift after a weekend; loss of interest in subjects or activities you used to enjoy; increasing irritability or cynicism about your course; needing much longer than usual to do ordinary tasks; withdrawing from friends; sleeping much more or much less than usual; and a growing sense of dread about the week ahead. Any one of these can happen in a normal hard week. A cluster of them that persists for several weeks is a signal to change something. Consider how this might unfold for a nursing student in her second year. Her first year went well, largely because she worked in intense bursts before every exam and made up for disorganized weeks with long weekends in the library. In her second year the clinical placements begin, with early starts that her late body clock resists, and she keeps her part-time job because she needs the money. For the first half of the term she compensates as she always has: less sleep, more caffeine, no exercise, weekends consumed by catching up. By week eight she is irritable with friends, dreads every placement shift, and needs three hours to write an assignment that would once have taken one. She tells herself she just needs to push through to the end of term. Then she catches a respiratory infection, misses a week, and finds that when she recovers she cannot make herself return to her notes at all. Looking back, every warning sign was there from week five onward. What was missing was a moment in which she stopped and read them as signals rather than as obstacles to push past. Had she spoken to her adviser at week six, dropped a shift or two at work, and protected her sleep around placements, the term would still have been hard, but it would probably not have ended in collapse. It is important to recognize that burnout overlaps with depression and anxiety, and that these are more common in people with ADHD. Burnout is often tied to a particular situation and eases when the pressure is removed. Depression tends to be more pervasive, affecting mood, sleep, appetite, and pleasure across all areas of life, and it may not lift with rest alone. You do not need to diagnose yourself. If low mood, hopelessness, or loss of interest last more than a couple of weeks, or if they are affecting your ability to function, see a clinician at student health or the counseling center. These are treatable conditions, and getting help early makes a real difference. If at any point you have thoughts of harming yourself or feel that you cannot keep yourself safe, treat it as an emergency. In the United States, you can call or text 988 to reach the 988 Suicide & Crisis Lifeline, or call 911 for emergency services. In other countries the numbers differ, so it is worth finding your local emergency number and crisis line now, while you are well, and saving them in your phone. Campus security and residence staff can also help you get urgent support. Reaching out in a crisis is not an overreaction. It is exactly what those services exist for. Recovering and preventing If you recognize yourself in the description of burnout, the first step is to reduce the load, not to find a better way of carrying it. This is counterintuitive for students who are used to solving problems by working harder, but more effort is precisely the resource that has run out. Talk to your academic adviser or disability services about options for the current term: extensions, reduced course load, withdrawal from a course, or, in more serious cases, a leave of absence. None of these is a failure. They are ways of stopping the overdraft before it becomes bankruptcy. The second step is to rebuild the foundations described in Chapter 2: a regular wake time, regular meals, some daily movement, and time outdoors. These sound too simple to address something as heavy as burnout, but they are the base on which recovery rests, and they are often the first things lost when a student is struggling. Start with one, keep it steady for a week, and add another. The third step is to reintroduce work gradually, with deliberate limits. After a period of burnout, the temptation is to make up for lost time with a massive catch-up push, which is simply the boom-and-bust cycle starting again. Instead, set a modest daily work target, such as two focused blocks, and stop when you reach it even if you feel you could do more. Build up slowly. The goal is to rebuild trust in your ability to work steadily, not to prove that you can still sprint. Prevention is the same principle applied before the crisis. It means pacing by energy, capping hyperfocus sessions, planning buffer and recovery into every week, noticing the early warning signs, and treating them as reasons to adjust rather than to push harder. It also means building a life at university that is not entirely made of work: friendships, interests, physical activity, and rest are not distractions from your degree but part of what makes it possible to finish one. Finally, it means learning to recognize the specific emotional patterns that drive your own cycle. For some students it is perfectionism; for others it is shame-driven avoidance; for others it is saying yes to every opportunity because saying no feels like disappointing someone. Knowing which one is yours lets you put a check in place: a good-enough standard, a rule to email the professor within twenty-four hours of noticing a problem, a practice of waiting a day before agreeing to any new commitment. These are small interventions, but they sit at exactly the point where the cycle usually begins, and that is where interventions do the most good. Chapter 8: Getting Support Without Losing Yourself Everything in this book so far has been about what you can do: how to manage energy, build an external brain, start tasks, design your environment, organize a term, and handle the emotional side of ADHD. But the argument of the book has always included other people. ADHD is best managed by moving executive work outside your head, and some of the most powerful places to move it are into relationships and institutions: a disability services office that arranges accommodations, a therapist who helps you change patterns, a coach or mentor who checks in each week, a prescriber who manages medication if you use it, and friends who study beside you. Many students with ADHD are reluctant to use these supports. Some feel that asking for help is an admission of weakness, or that accommodations are an unfair advantage. Some have had bad experiences with services in the past. Some simply never get around to it, because contacting an office, gathering documents, and attending appointments are exactly the kinds of tasks that ADHD makes difficult. This chapter aims to make the supports concrete: what each offers, what the evidence says, and how to use them in a way that strengthens your own capacity rather than replacing it. Accommodations and disability services Most universities have an office responsible for disability services, sometimes called accessibility services, student access, or something similar. Its job is to arrange reasonable adjustments for students whose disabilities affect their access to education. ADHD is one of the most common conditions such offices support; the review by DuPaul and colleagues cited in the introduction noted that students with ADHD make up a substantial share of students registered with disability services. In the United States, the legal basis for these accommodations comes mainly from the Americans with Disabilities Act and Section 504 of the Rehabilitation Act, which require colleges to provide reasonable accommodations to qualified students with disabilities. Other countries have their own laws and systems, sometimes including funding for equipment or support workers, and practice differs even between institutions in the same country. The details below describe common patterns, but the only reliable source for your situation is your own institution's office. Registering usually involves submitting documentation of your diagnosis and how it affects you, and meeting with a staff member to discuss which adjustments are appropriate. Requirements for documentation vary; some offices ask for a recent evaluation, others accept a letter from a clinician. If you do not have documentation, or if yours is old, ask the office what they need before assuming you cannot register. Some universities offer or can refer you to assessment services. Common accommodations for ADHD include extended time on exams; taking exams in a room with fewer distractions; permission to record lectures; access to lecture slides or notes in advance; note-taking support; flexibility on some deadlines, typically arranged in advance with each professor; priority registration, which can help you build a schedule that suits your energy; and, in some systems, a reduced course load without loss of full-time status. Not all of these will be available everywhere, and not all will suit you. The best approach is to think about where your difficulties actually fall, using the breakdown log from Chapter 1 if you kept one, and to ask for adjustments that address those specific problems. Two points are worth emphasizing. First, accommodations are not an unfair advantage. They are designed to remove barriers that affect you because of a disability, so that your performance reflects what you know and can do rather than the effects of the condition on, for example, timed testing in a noisy room. Second, registering does not usually mean that everyone will know about your diagnosis. In most systems, professors are told which accommodations you are entitled to, not the underlying condition, and you decide what else to share. Whether and how to disclose to professors, classmates, or employers is your choice. If you suspect you have ADHD but have never been assessed, the first step is usually an appointment with student health or the counseling center, who can talk through your concerns and explain how assessment works locally. A proper adult ADHD assessment typically involves a detailed interview about current difficulties and childhood history, rating scales, and often input from someone who knew you as a child, along with checks for other conditions that can look similar, such as anxiety, depression, sleep disorders, or learning difficulties. Waiting lists can be long and costs vary. In the United States, whether an assessment is covered depends on your insurance plan and on whether your campus provides the service directly; in countries with public health systems, the route may run through a general practitioner and a specialist referral. It is worth starting the process early, since many of the supports in this chapter depend on it. In the meantime, nothing stops you from using the strategies in this book, which help with the difficulties whether or not they carry a label. Professors are also part of the picture, whether or not you have formal accommodations. A short conversation during office hours early in the term, in which you introduce yourself and mention briefly that you sometimes struggle with long deadlines and would appreciate knowing about any interim checkpoints, can make you a known person rather than a name on a list. You need not mention ADHD unless you want to. The point is to establish a relationship before there is a problem, so that if you later need to ask for flexibility, you are asking someone who has already met you. Office hours are also one of the most underused forms of external structure on any campus: a fixed time, a real person, and a natural deadline for having something to discuss. The main practical difficulty with accommodations is that they often require action from you each term: sending letters to professors, booking exam rooms, arranging notes. This is precisely the kind of administrative task that ADHD makes hard, and many students register and then fail to use what they have been granted. The solution is to treat these tasks as fixed items in the week-one planning described in Chapter 6. Put a calendar entry at the start of each term to send accommodation letters, and entries a week or two before each exam to confirm arrangements. If your office offers regular check-ins with an adviser, use them. The ACCESS trial described below found that students who received the intervention made greater use of disability accommodations, which suggests that support in using accommodations is itself valuable. Therapy, coaching, and the evidence Psychological treatment for adult ADHD has a growing evidence base, and some of the best of it concerns college students specifically. The most important recent work is the ACCESS program, short for Accessing Campus Connections and Empowering Student Success, developed by Arthur Anastopoulos and colleagues. ACCESS is a cognitive-behavioral program delivered over two semesters, combining group sessions with individual mentoring. It covers knowledge about ADHD, practical behavioral strategies, and adaptive thinking, and it actively connects students with campus resources. In a randomized controlled trial published in 2021, 250 college students with carefully diagnosed ADHD at two public universities were assigned either to receive ACCESS immediately or to wait a year. Those who received it showed greater improvements in ADHD symptoms, executive functioning, and use of disability accommodations, with effect sizes the authors described as medium to large. A related paper from the same trial reported moderate improvements in self-reported study skills, time management, daily functioning, and well-being, which were maintained or increased by the end of the second semester. The same paper found no significant difference in grade point average or credits earned, which is an honest reminder that better functioning does not always translate quickly into better grades, and that grades depend on many things besides ADHD. A later analysis found that students who rapidly improved their use of behavioral strategies showed robust gains in symptoms and functioning, and that improvements in adaptive thinking, though harder to achieve, were strongly linked to outcomes. Other research on cognitive-behavioral approaches for adults with ADHD points in a similar direction. A randomized trial by Steven Safren and colleagues, published in JAMA in 2010, found that adults with ADHD who were already taking medication but still had significant symptoms improved more with cognitive-behavioral therapy than with a relaxation-based comparison treatment. Mary Solanto and colleagues found benefits from a group-based meta-cognitive therapy focused on time management, organization, and planning. These studies involved specific, structured programs rather than general talk therapy, which is worth keeping in mind when looking for a therapist. Ask whether they have experience with adult ADHD and whether they use structured, skills-based approaches. Coaching is a related but distinct form of support. An ADHD coach typically works with you on goals, planning, and accountability, often through short weekly sessions, rather than on emotional or psychological issues. Coaching can be a very practical way of borrowing external structure, since the regular check-in itself creates a deadline and a relationship. The research base for coaching is smaller and less rigorous than for cognitive-behavioral therapy, and the field is not uniformly regulated, so qualifications vary. Some universities offer coaching or academic mentoring through disability services or learning centers, and these are often a good place to start. Table 5 summarizes the main sources of support available to students with ADHD, what each offers, and how students usually reach them. The availability and names of these services vary between institutions and countries. Table 5. Sources of support for students with ADHD. Source What it offers Usual route in Good to know Disability or accessibility services Formal accommodations; advising Register with documentation Accommodations often need renewing each term Counseling center or therapist CBT and skills-based therapy; support for mood and anxiety Campus counseling or referral Ask about experience with adult ADHD Student health or prescriber Assessment, medication decisions, sleep and health issues Appointment with a clinician Medication decisions belong here Coaching or academic mentoring Weekly planning and accountability Learning center or private coach Evidence base is smaller; qualifications vary Writing and tutoring centers Help with drafts and course content; external deadlines Book appointments Useful as mini-deadlines in backward plans Peers and study partners Body doubling; shared structure; understanding Study groups; student organizations Low cost and often underused A brief word on medication Medication is one of the most widely used and well-studied treatments for ADHD, and the international consensus statement published in 2021 describes a large body of evidence supporting its effectiveness in reducing symptoms for many people. It is not the subject of this book, and nothing here is intended to encourage or discourage its use. Whether medication is right for you, which one, at what dose, and how it fits with sleep, appetite, caffeine, and other conditions are decisions for you and a qualified prescriber. If you take medication, do not change the dose or timing on your own; if something is not working or side effects are bothering you, talk to your prescriber. Many people find that medication and behavioral strategies work best together, since medication can make it easier to use strategies, and strategies address problems that medication does not reach. One aspect of medication does need to be addressed plainly, because it arises on almost every campus: misuse and diversion. Stimulant medications prescribed for ADHD are sometimes used by students without prescriptions, usually in the mistaken belief that they will improve academic performance. A 2008 systematic review by Timothy Wilens and colleagues found that reported rates of past-year non-prescribed stimulant use among college-age individuals ranged widely across studies, and that between 16 and 29 percent of students with stimulant prescriptions reported having been asked to give, sell, or trade their medication. Those figures are from older studies and vary with how questions were asked, but they make clear that being asked is a common experience. If you are prescribed medication, never share, sell, or trade it. This is not a matter of etiquette. In many countries, including the United States, stimulant medications are controlled substances, and giving or selling them to someone else is a crime with serious consequences, including possible expulsion. It is also dangerous: the person taking it has not been assessed for heart conditions, interactions with other drugs, or other risks, and non-medical use, especially in high doses or combined with alcohol or energy drinks, can cause serious harm. Keep your medication somewhere secure rather than out on a desk in a shared room, and keep track of how much you have. If someone asks, a simple, firm refusal is enough; you might say that your prescriber monitors your supply closely, or simply that you do not share medication. If you are not prescribed medication and are tempted to take someone else's to get through a hard week, the strategies in this book, and a conversation with a clinician about your difficulties, are both safer and more likely to help. Building your team The final point of this chapter, and in some ways of the practical part of this book, is that support works best as a small, deliberate team rather than a scattered set of services used in emergencies. Think about who is on yours. For many students it might include a disability services adviser who arranges accommodations and checks in once or twice a term; a therapist or counselor, especially if emotional patterns or mood are a significant part of the picture; a prescriber, if medication is part of treatment; one or two regular study partners who provide body doubling and accountability; and a trusted person outside the university, such as a family member or old friend, who knows you well enough to notice when things are slipping. Using this team is itself a skill, and it follows the same principles as everything else in this book. Put the contacts in your calendar rather than relying on remembering. Schedule check-ins in advance rather than waiting for a crisis. Tell at least one person about the early warning signs of your personal burnout cycle, and ask them to say something if they notice. Consider how this looked for one final-year student who had spent three years managing on his own. He had a diagnosis from childhood, had stopped seeing anyone about it when he left home, and had never registered with disability services because it felt like admitting he could not cope. In his final year, with a dissertation and a job search on top of his courses, the old methods stopped working. Over a few weeks he registered with disability services and was granted extended time on exams and a weekly check-in with an academic mentor; he booked an appointment with a clinician to review his treatment, which he had not done in years; and he agreed with two classmates to work together in the library three afternoons a week. None of these supports was dramatic. Together they supplied the structure that his final year required: a weekly meeting that forced him to plan, a clinician who helped him address his sleep, and a routine of shared work that made starting almost automatic. Looking back, the problem had never been that he needed help. It was that he had waited until his last year to arrange it. And treat asking for help as a strategy rather than a failure. The students who do best with ADHD at university are rarely the ones who manage everything alone. They are the ones who have learned where their own capacities run out and have arranged, calmly and in advance, for something or someone to be there when they do. Conclusion: Measuring the Right Thing A student with ADHD who works through the chapters of this book will not end up with a single technique. They will end up with a set of shifts in where the work of self-management happens. Commitments move out of memory and into a calendar. The decision to start moves from the moment of weakness to an earlier, calmer moment. Resistance to distraction moves from willpower to the arrangement of the room and the phone. Urgency moves from the night before the deadline to a chain of smaller deadlines spread across the term. Accountability moves from private resolve to other people. Each of these is modest on its own. Together they amount to a different theory of how a person with ADHD gets things done: not by becoming more self-controlled, but by arranging a life that requires less self-control at the points where it most often fails. That theory carries a consequence that is easy to miss until the pieces have been laid side by side. If the work of self-regulation is being moved outward, then the right way to judge whether it is working also changes. Most students, and most of the people around them, judge an ADHD strategy by output: did the paper get written, did the grade go up, was the deadline met. Those things matter. But the chapters on energy and burnout show why output alone is a misleading measure. The all-nighter produces output. So does the anxious overcompensation that precedes collapse. A student can meet every deadline for two years on a system of crisis and recovery, and the numbers will look fine until the term when they suddenly do not. The measure that follows from this book is sustainability: how much the work costs to keep producing. A good week is not the week in which you did the most. It is the week in which you did enough without borrowing from the next one: you slept at roughly the same time, you ate, you moved, you started tasks on the days you planned to start them, and you ended the week with some energy left. A good term is one in which the pinch points were seen early and flattened, in which the lapses were followed by restarts rather than abandonment, and in which you finished tired but not depleted. Judged this way, a term with slightly lower grades and far less damage may be a better term, because it leaves you able to do the next one. The ACCESS trial offers a small echo of this point. Students who received the program improved in functioning, time management, and well-being, but their grades did not change significantly over the period studied. One reading of that result is disappointing. Another is that functioning and well-being are the foundation, and that grades are a lagging indicator that depends on much more than any single program can reach. What remains unsettled Honesty requires admitting how much of the practical advice for ADHD rests on thinner evidence than its popularity suggests. Body doubling is widely recommended and widely reported to help, but it has barely been studied in controlled research. Coaching has a smaller and less rigorous evidence base than cognitive-behavioral therapy. Much of the research on exercise and attention in ADHD involves children rather than university students. Studies of phone distraction have produced findings that did not always replicate. Even the better-supported tools, such as implementation intentions, were mostly tested in general populations rather than in people with ADHD specifically. None of this means the strategies do not work. It means that for any individual student, the evidence is a starting point rather than a guarantee, and the real test is personal experimentation. This is why the breakdown log, the two-week trial, and the habit of judging a strategy in week eight rather than week one run through the whole book. You are, in a practical sense, conducting a study with a sample size of one, and the most useful thing you can do is to run it carefully: change one thing at a time, keep a simple record, and believe what the record shows rather than what the novelty of a new tool makes you feel. There is a second unsettled question, and it concerns institutions rather than individuals. Much of what helps students with ADHD, such as frequent checkpoints on large assignments, clear deadlines published at the start of term, recorded lectures, flexible exam settings, and structured study spaces, would help many students without ADHD too. The more a university builds this kind of structure into its ordinary teaching, the less each student has to construct it alone, and the fewer students fall through the gap between the heavily scaffolded school years and the self-directed freedom of university. That is a debate for faculty and administrators more than for students, but it is worth knowing that the difficulties you face are partly a product of how university is organized, and not only of how your brain works. What to do differently If the argument of this book is right, then a few changes follow for the way you approach your next term. First, begin the term with a map rather than a resolution. Resolutions ask your future self to feel differently. A map, with every deadline, pinch point, and administrative date in one place, gives your future self something to act on. Spend one focused hour in the first week, with a friend if possible, building it. Second, choose one foundation and one system, not ten. A fixed wake time and a single trusted calendar, maintained through a weekly reset, will do more than any number of apps. Add anything else only after these have survived a month. Third, plan your recovery as deliberately as your work. Put rest, movement, meals, and at least one buffer block into the calendar before the term fills up. If this feels indulgent, remember the chapters on burnout: recovery is not a reward, it is the maintenance that keeps the whole system running. Fourth, arrange your support before you need it. Register with disability services if you are eligible. Book the first appointment with a counselor or clinician if you have been meaning to. Find one or two people to study with regularly. The best time to build a team is in week one, when nothing is on fire. Fifth, change the question you ask at the end of each week. Instead of asking whether you got everything done, which you will almost never be able to answer yes to, ask whether the week was sustainable and what one change would make next week easier. That question turns every week into information rather than a verdict, and it keeps your attention on the thing that actually determines how your degree will go: not the brilliance of your best weeks, but the steadiness of your ordinary ones. ADHD will not disappear when you close this book, and it does not need to. You are not trying to become someone else. You are trying to build a life at university in which the person you already are, with the mind you already have, can do good work without paying for it with your health. That is a practical goal, and it is within reach. Notes Introduction 1. DuPaul, G. J., Weyandt, L. L., O'Dell, S. M., & Varejao, M. (2009). College students with ADHD: Current status and future directions. Journal of Attention Disorders, 13(3), 234-250. https://doi.org/10.1177/1087054709340650 Chapter 1 1. Barkley, R. A. (2010). Taking Charge of Adult ADHD. New York: Guilford Press. (A second edition was published in 2021.) The ideas of ADHD as a disorder of doing rather than knowing, and of intervening at the point of performance, are developed throughout Barkley's work. 1. Faraone, S. V., Banaschewski, T., Coghill, D., et al. (2021). The World Federation of ADHD International Consensus Statement: 208 evidence-based conclusions about the disorder. Neuroscience and Biobehavioral Reviews, 128, 789-818. https://doi.org/10.1016/j.neubiorev.2021.01.022 Chapter 2 1. Bijlenga, D., Vollebregt, M. A., Kooij, J. J. S., & Arns, M. (2019). The role of the circadian system in the etiology and pathophysiology of ADHD: Time to redefine ADHD? ADHD Attention Deficit and Hyperactivity Disorders, 11(1), 5-19. https://doi.org/10.1007/s12402-018-0271-z 1. Cerrillo-Urbina, A. J., García-Hermoso, A., Sánchez-López, M., Pardo-Guijarro, M. J., Santos Gómez, J. L., & Martínez-Vizcaíno, V. (2015). The effects of physical exercise in children with attention deficit hyperactivity disorder: A systematic review and meta-analysis of randomized control trials. Child: Care, Health and Development, 41(6), 779-788. https://doi.org/10.1111/cch.12255 Chapter 3 1. Allen, D. (2015). Getting Things Done: The Art of Stress-Free Productivity (revised edition). New York: Penguin. (First published 2001.) Chapter 4 1. Gollwitzer, P. M., & Sheeran, P. (2006). Implementation intentions and goal achievement: A meta-analysis of effects and processes. Advances in Experimental Social Psychology, 38, 69-119. 1. Cirillo, F. (2018). The Pomodoro Technique. New York: Currency. 1. Milkman, K. L., Minson, J. A., & Volpp, K. G. M. (2014). Holding the Hunger Games hostage at the gym: An evaluation of temptation bundling. Management Science, 60(2), 283-299. Chapter 5 1. Ward, A. F., Duke, K., Gneezy, A., & Bos, M. W. (2017). Brain drain: The mere presence of one's own smartphone reduces available cognitive capacity. Journal of the Association for Consumer Research, 2(2), 140-154. Subsequent replication attempts have produced mixed results. Chapter 6 1. Buehler, R., Griffin, D., & Ross, M. (1994). Exploring the "planning fallacy": Why people underestimate their task completion times. Journal of Personality and Social Psychology, 67(3), 366-381. Chapter 7 1. Shaw, P., Stringaris, A., Nigg, J., & Leibenluft, E. (2014). Emotion dysregulation in attention deficit hyperactivity disorder. American Journal of Psychiatry, 171(3), 276-293. https://doi.org/10.1176/appi.ajp.2013.13070966 1. World Health Organization. International Classification of Diseases, 11th Revision (ICD-11), entry QD85, Burn-out, classified as a factor influencing health status rather than a medical condition. Chapter 8 1. Anastopoulos, A. D., Langberg, J. M., Eddy, L. D., Silvia, P. J., & Labban, J. D. (2021). A randomized controlled trial examining CBT for college students with ADHD. Journal of Consulting and Clinical Psychology, 89(1), 21-33. https://doi.org/10.1037/ccp0000553 1. Eddy, L. D., Anastopoulos, A. D., Dvorsky, M. R., Silvia, P. J., Labban, J. D., & Langberg, J. M. (2021). An RCT of a CBT intervention for emerging adults with ADHD attending college: Functional outcomes. Journal of Clinical Child and Adolescent Psychology, 50(6), 844-857. https://doi.org/10.1080/15374416.2020.1867989 1. Langberg, J. M., Dvorsky, M. R., Silvia, P., Labban, J., & Anastopoulos, A. D. (2023). Clinical change mechanisms in the treatment of college students with ADHD: Trajectories and associations with outcomes. Behavior Therapy, 54(3), 444-460. https://doi.org/10.1016/j.beth.2022.11.003 1. Safren, S. A., Sprich, S., Mimiaga, M. J., et al. (2010). Cognitive behavioral therapy vs relaxation with educational support for medication-treated adults with ADHD and persistent symptoms: A randomized controlled trial. JAMA, 304(8), 875-880. 1. Solanto, M. V., Marks, D. J., Wasserstein, J., et al. (2010). Efficacy of meta-cognitive therapy for adult ADHD. American Journal of Psychiatry, 167(8), 958-968. 1. Wilens, T. E., Adler, L. A., Adams, J., et al. (2008). Misuse and diversion of stimulants prescribed for ADHD: A systematic review of the literature. Journal of the American Academy of Child and Adolescent Psychiatry, 47(1), 21-31. https://doi.org/10.1097/chi.0b013e31815a56f1 Further Reading Russell A. Barkley, Taking Charge of Adult ADHD (Guilford Press, 2nd edition 2021).** A clear, practical guide by one of the most influential researchers in the field. Especially useful for its explanation of executive function and its emphasis on changing the environment at the point of performance. Mary V. Solanto, Cognitive-Behavioral Therapy for Adult ADHD: Targeting Executive Dysfunction (Guilford Press, 2011).** Written for clinicians, but readable by motivated students. It sets out the structured group program on time management, organization, and planning that was tested in Solanto's trial, and it is a good model of what skills-based therapy for ADHD looks like. J. Russell Ramsay and Anthony L. Rostain, The Adult ADHD Tool Kit: Using CBT to Facilitate Coping Inside and Out (Routledge, 2015).** A workbook-style guide drawing on cognitive-behavioral principles, with particular attention to the thoughts and emotions that drive procrastination and avoidance. Edward M. Hallowell and John J. Ratey, ADHD 2.0 (Ballantine, 2021).** An accessible and encouraging overview by two clinicians with long experience of ADHD. Read it for perspective and hope rather than for detailed evidence. Stephen V. Faraone and colleagues, "The World Federation of ADHD International Consensus Statement: 208 Evidence-Based Conclusions about the Disorder," Neuroscience and Biobehavioral Reviews 128 (2021).** Freely available online. The best single summary of what research firmly supports about ADHD, useful for correcting myths, including your own. David Allen, Getting Things Done (Penguin, revised edition 2015).** Not written for ADHD, and the full system is heavier than most students need, but its core ideas about capture and external systems underlie Chapter 3. CHADD (Children and Adults with Attention-Deficit/Hyperactivity Disorder), chadd.org. A US-based nonprofit with reliable educational material on adult ADHD, including information on college, accommodations, and treatment. Readers in other countries should look for their national ADHD organization as well, since services and legal frameworks differ. Hashtags: #ADHDStudents #ADHDStudentPlaybook #ADHDAwareness #ADHDInCollege #ADHDStudyStrategies #Neurodiversity #NeurodivergentStudents #ExecutiveFunction #ExecutiveDysfunction #FocusTechniques #AttentionManagement #TimeBlindness #TimeManagement #OrganizationStrategies #AcademicProductivity #TaskInitiation #OvercomingProcrastination #BodyDoubling #PomodoroTechnique #StudyHabits #ADHDTimeManagement #ExternalMemorySystems #ADHDBurnout #BurnoutPrevention #EnergyManagement #EmotionalRegulation #HyperfocusManagement #ADHDAccommodations #StudentWellbeing #SustainableProductivity

Download the Book (PDF): Introduction Every machine learning project eventually runs into the same wall. The model architecture is settled, the compute is booked, the evaluation plan is written, and then someone asks where the data will come from. Sometimes the data exists but cannot be moved: it sits in a hospital's electronic health record system, governed by consent forms, ethics approvals and data protection law, and the team that wants to build a sepsis predictor is three organisations and eighteen months of paperwork away from it. Sometimes the data exists and can be used, but the part that matters is vanishingly thin: a bank has ten million loan records, and the defaults that a credit model must learn to anticipate are a small fraction of them, thinner still in the segments where the bank most needs to be right. And sometimes the data simply does not exist in any quantity, because the event of interest is rare by nature: a child running between parked cars at dusk in heavy rain, a hairline crack in a turbine blade, a tumour presentation that a radiologist sees twice in a career. These are three different kinds of scarcity. The first is a scarcity of access: the information is there but locked. The second is a scarcity of balance: the information is there but the important class is drowned out. The third is a scarcity of events: the world has not yet produced enough examples for anyone to collect. Synthetic data generation is offered as the answer to all three, and over the past decade the offer has become far more credible. Generative adversarial networks, introduced by Ian Goodfellow and colleagues in 2014, showed that a neural network could learn to produce samples that another network could not tell apart from real ones. Denoising diffusion models, brought to prominence by Jonathan Ho, Ajay Jain and Pieter Abbeel in 2020, made that generation stable, controllable and astonishingly realistic. Tabular generators such as CTGAN and TabDDPM carried the same ideas to the spreadsheets and relational tables that most organisations actually run on. Simulation engines and, more recently, learned world models, such as the one Waymo described in February 2026 on top of Google DeepMind's Genie 3, can now render driving scenes that no fleet has ever recorded. The commercial world has noticed. Gartner-style forecasts about the share of AI training data that will be synthetic have circulated for years; more concretely, SAS bought the British synthetic data company Hazy in November 2024, and NVIDIA acquired Gretel in March 2025. Synthetic data is no longer a research curiosity. It is a product category, a line item in procurement, and increasingly a claim made to regulators. That last point is where this book begins to differ from much of the material written on the subject. A great deal of what is said about synthetic data treats generation as the hard part. Choose a powerful enough model, train it on the real data, sample as many rows or images as you like, and the scarcity problem dissolves: the privacy problem disappears because no synthetic record "is" a real person, the imbalance problem disappears because you can sample as many minority cases as you want, and the edge case problem disappears because you can render whatever the world has failed to supply. Each of those claims contains something true. Each of them, stated that baldly, is also wrong in a way that has cost real projects real money, and in the case of privacy has misled organisations about their legal exposure. The argument of this book is that synthetic data is a measurement problem far more than it is a generation problem. A synthetic dataset is worth exactly what you can demonstrate about it for a stated purpose, and no more. Three demonstrations matter. The first is fidelity: does the synthetic data reproduce the statistical structure of the real data closely enough that conclusions drawn from one transfer to the other? The second is utility: does a model trained or tested on the synthetic data actually perform better on real-world cases than it would have without it? The third is privacy: what can an adversary who holds the synthetic data learn about the specific people whose records trained the generator? These three properties pull against each other. Perfect fidelity is achieved by copying the training set, which offers no privacy at all. Perfect privacy is achieved by sampling noise, which has no utility. Every useful synthetic dataset sits somewhere on the surface between those extremes, and the work of a practitioner is to know where, to choose deliberately, and to be able to show the choice to someone else. Seen this way, the generative model is a component, not a solution. GANs and diffusion models matter enormously, and this book explains how they work in enough depth to reason about their failure modes: why GANs drop rare modes, why diffusion models memorise duplicated training examples, why a conditional tabular generator can produce a perfectly plausible loan applicant with an impossible combination of age and employment history. But the decisions that determine whether a synthetic data project succeeds are taken around the model, not inside it. They concern what claim the data is meant to support, which metrics can substantiate that claim, which attacks the release must withstand, which real data is held back to validate the result, and who signs off. The structure follows from that argument. The first chapter sets out what synthetic data is and is not, from its origins in statistical disclosure control at the US Census Bureau to the modern taxonomy of fully synthetic, partially synthetic and simulated data, and establishes the most important limit in the field: a generator trained on data cannot create information that the data did not contain, though it can combine that information with prior knowledge in useful ways. The second and third chapters explain the two model families named in this book's subtitle, generative adversarial networks and diffusion models, alongside the older and often underrated statistical alternatives that still win in some settings. The fourth chapter turns to fidelity and utility measurement, and the fifth to privacy: what synthetic data protects against, what it does not, what membership and attribute inference attacks have shown, and what differential privacy can and cannot guarantee. The next three chapters take the three scarcities in turn, each through a domain where the stakes are concrete. The sixth chapter covers healthcare and the law. It is written with care because the popular framing, that synthetic data lets organisations "bypass" privacy regulation, is both legally inaccurate and practically dangerous. Synthetic data can substantially reduce privacy exposure and can support compliance, and in well-governed programmes such as the UK Medicines and Healthcare products Regulatory Agency's synthetic datasets derived from primary care records, it has opened data to researchers who would otherwise never have seen it. But under the EU General Data Protection Regulation, the act of generating synthetic data from personal data is itself processing that needs a legal basis, and the output escapes the regulation only if it is genuinely anonymous by the regulation's own test. The European Data Protection Board's draft anonymisation guidelines of July 2026 say in terms that synthetic datasets and AI models are not automatically anonymous. Under HIPAA in the United States, synthetic data derived from protected health information leaves the Privacy Rule only by satisfying one of its de-identification standards. The chapter explains how those tests apply and how to build a programme that meets them, rather than how to avoid them. The seventh chapter addresses class imbalance in credit scoring, where synthetic oversampling is the oldest and most widely used application of synthetic data and also one of the most misunderstood. It explains why techniques from SMOTE to conditional GANs often improve a model's headline sensitivity while quietly ruining its probability estimates, why that matters more in lending than almost anywhere else, and how regulation of credit models in the United States and the European Union shapes what is acceptable. The eighth chapter turns to computer vision and the long tail of rare edge cases, from the game-engine datasets of the mid-2010s and domain randomisation to neural world models and diffusion-based augmentation, and to the persistent gap between simulated and real sensor data. The ninth chapter considers synthetic data as part of a larger system: what happens when models are trained on the outputs of other models, the risk of the degenerative process now called model collapse, and the governance and documentation that turn a one-off synthetic dataset into a dependable data product. The conclusion draws out what follows from the whole argument for practitioners, for the people who commission synthetic data, and for those who regulate it. The book is written for practitioners and for the technically literate people who direct them: data scientists, machine learning engineers, analytics leads, privacy officers, model risk managers and clinicians or researchers who work with data teams. It assumes familiarity with the basic vocabulary of machine learning, such as training and test sets, classifiers and loss functions, but it does not assume any prior knowledge of generative modelling or of privacy law. Where a short code example clarifies a point better than prose, one appears; the argument never depends on it. One commitment runs throughout. Synthetic data is a field with a strong promotional current, and it is easy to describe it in the language of breakthroughs. The aim here is the opposite: to state precisely what each technique does, to cite the evidence for and against it, and to be clear about where questions remain open. Synthetic data is genuinely one of the most useful tools available for working around data scarcity. It is useful in proportion to the honesty with which it is measured. CHAPTER 1 What Synthetic Data Is, and What It Cannot Be The phrase "synthetic data" covers so many different things that arguments about it often fail simply because the parties are talking about different objects. A procedurally generated city in a driving simulator is synthetic data. So is a table of ten thousand fictional patients sampled from a Bayesian network fitted to a hospital's records. So is a dataset in which the real incomes of survey respondents have been replaced with values drawn from a regression model while every other column stays real. So is a set of chest X-rays produced by a diffusion model conditioned on the word "pneumothorax". So, in the loosest usage, is a rotated and colour-shifted copy of a real photograph. These objects have different origins, different risks and different uses, and a practitioner who cannot say which kind they are holding cannot reason about what it is good for. This chapter does three things. It traces where the idea came from, because the original motivation still shapes the best practice. It sets out a working taxonomy. And it establishes the single most important limit on everything that follows: what a generator can and cannot add to the data it learned from. From disclosure control to deep generative models Synthetic data did not begin in machine learning. It began in official statistics, with a problem that is still the central problem of the field: how to let people analyse data about individuals without revealing anything about any particular individual. National statistical agencies collect census returns, labour force surveys and business registers under legal promises of confidentiality. For most of the twentieth century they honoured those promises by publishing aggregate tables and, for research use, microdata files in which records had been coarsened, swapped, top-coded or suppressed. Each of these techniques damaged the data in ways that were hard for analysts to account for, and each left residual risk. In 1993 the statistician Donald Rubin proposed a radical alternative in a short discussion paper in the Journal of Official Statistics: instead of releasing altered real records, release entirely synthetic records drawn from a statistical model of the population, generated using the same multiple imputation machinery Rubin had developed for missing data. Because no released record corresponded to a real respondent, the disclosure risk would be low; because the records were drawn from a model fitted to the real data, valid inferences could still be made from them, and the variability introduced by synthesis could be measured by releasing several synthetic copies and comparing them. In the same issue Roderick Little proposed the partial version: replace only the sensitive values, keep the rest. Over the following two decades Trivellore Raghunathan, Jerome Reiter and Rubin worked out the combining rules that let analysts get correct standard errors from multiple synthetic datasets, and agencies began to ship products built on the idea. The US Census Bureau's OnTheMap application, launched in 2008, published synthetic data about where people live and work at fine geographic detail; a paper describing it by Ashwin Machanavajjhala and colleagues was among the first to apply a formal privacy definition to a real government release. The Bureau's Synthetic Longitudinal Business Database followed, and so did a synthetic version of the Survey of Income and Program Participation linked to tax and benefit records, which researchers could analyse freely and then have their code validated against the confidential original. In the United Kingdom the synthpop package for R, published by Beata Nowok, Gillian Raab and Chris Dibben in 2016, made sequential regression-based synthesis available to any analyst. Three lessons from this lineage carry straight into modern practice. First, the purpose of synthesis was always to support valid inference about the real population, not to produce data that merely looked real. Second, synthetic data was never treated as automatically safe; agencies measured disclosure risk and often combined synthesis with other protections, including restricted access and validation servers. Third, the best programmes paired the synthetic release with a route back to the real data, so that a finding made on synthetic data could be checked before anyone relied on it. All three lessons are routinely forgotten when synthetic data is marketed as a replacement for real data rather than a carefully measured proxy for it. The deep learning era added power and changed the emphasis. Generative adversarial networks, variational autoencoders and later diffusion models could learn the joint distribution of high-dimensional data such as images, audio and long clinical time series, which classical sequential regression could not handle. At the same time, the dominant use shifted from publishing statistics to training models. A statistician releasing synthetic census microdata wants regression coefficients computed on the synthetic file to match those computed on the real one. A machine learning engineer generating synthetic pedestrians wants a detector trained partly on them to find more real pedestrians. These are different goals with different tests, and confusing them is one of the commonest errors in the field. A working taxonomy It helps to classify any synthetic dataset along three axes: where its information comes from, how much of each record is synthetic, and what it is for. The first axis distinguishes data-driven synthesis from model-driven synthesis. Data-driven synthesis fits a generative model to real records and samples from it. A GAN trained on credit applications, a diffusion model trained on retinal photographs, and a Bayesian network fitted to a primary care database are all data-driven. Everything such a generator knows came from the training data, plus whatever assumptions are built into its architecture. Model-driven synthesis, by contrast, starts from explicit knowledge of how the data is produced. A driving simulator renders camera images from three-dimensional geometry, material properties and a physically based model of light. Synthea, the open-source synthetic patient generator released by the MITRE Corporation and described by Jason Walonoski and colleagues in 2018, builds lifetime medical histories for fictional patients by running them through hand-authored disease modules derived from clinical guidelines and published incidence statistics, without training on any individual patient record. Model-driven data carries no privacy risk from individual records because no individual records were used, but it is only as faithful as the model: Synthea's patients follow the guidelines more neatly than real patients ever do, and simulated rain falls on the camera lens only in the ways the rendering engineer anticipated. Many modern systems are hybrids. A driving simulator may place hand-modelled vehicles into backgrounds reconstructed from real sensor logs. A learned world model such as Waymo's, built on a video model pre-trained on vast amounts of real footage, is data-driven in its origins but steered by explicit scenario descriptions. Knowing the mix matters because it tells you which failures to expect: data-driven components inherit the gaps and biases of their training data, while model-driven components inherit the blind spots of their designers. The second axis is the degree of synthesis. A fully synthetic dataset contains no real records; every value in every row was generated. A partially synthetic dataset keeps some real values and replaces others, typically the sensitive or identifying ones. Augmented data consists of real records transformed by label-preserving operations such as flipping, cropping, adding noise or changing brightness, and of interpolations between real records such as those produced by SMOTE, which Chapter 7 discusses at length. These categories differ sharply in privacy terms. A partially synthetic file in which only incomes were replaced still contains each person's real age, occupation and postcode, which may be enough to identify them; Latanya Sweeney's well-known finding that 87 per cent of the US population was likely to be uniquely identified by five-digit ZIP code, sex and date of birth is a reminder of how little that takes. Augmented images are transformed copies of real individuals' data and should be treated as such. The third axis is purpose, and it is the one that most often goes unstated. Synthetic data is used for at least five distinct jobs: • Training models where real data is scarce or imbalanced. • Testing models and systems, including generating stress scenarios and rare edge cases that real test sets lack. • Sharing data across organisational or legal boundaries for exploratory analysis, method development or education. • Software development and quality assurance, where engineers need realistic but non-real records to build and test pipelines. • Statistical inference about the real population, the original census use case. Each job demands different properties. Software testing needs schema validity and realistic edge cases in formats but not accurate joint distributions; a synthetic dataset of patients with plausible field formats and referential integrity is enough. Exploratory sharing needs approximate preservation of the main relationships, with a validation route back to real data before conclusions are relied upon. Training a model needs the synthetic data to improve performance on real cases, which is an empirical question answered only by testing on real held-out data. Statistical inference needs not only the right point estimates but the right uncertainty, which most machine learning generators do not attempt to provide. A synthetic dataset that is excellent for one job can be useless or actively misleading for another, which is why the first question in any synthetic data project should be "what is this for?" rather than "which generator should we use?" The information limit The most important fact about data-driven synthetic data is also the least advertised. A generator trained on a dataset cannot produce information about the world that the dataset did not contain. It can rearrange, interpolate and smooth that information, combine it with the inductive biases of its architecture, and present it in larger quantities, but it cannot conjure new evidence. Information theory makes this precise. The data processing inequality says that if a variable Z is computed from Y, which in turn depends on X, then Z cannot carry more information about X than Y does. Here X is the true population, Y is the training sample, and Z is the synthetic dataset generated from a model fitted to Y. However many synthetic rows are sampled, the synthetic dataset carries at most as much information about the population as the original sample did, and typically less, because fitting and sampling add their own error. This has direct practical consequences. If a hospital has 40 cases of a rare complication, a generator trained on its records can produce 4,000 synthetic cases, but those 4,000 cases encode what the generator inferred from 40. If the 40 real cases happened to share some incidental feature, say that most came from one ward with a particular charting habit, the synthetic cases will reproduce it faithfully and a downstream model may learn it as a signal. Sampling more synthetic cases does not dilute the quirk; it amplifies the model's confidence in it. Statistical power does not increase either. A study conducted on 4,000 synthetic patients derived from 40 real ones has the effective sample size of the 40, and any standard errors computed as though the 4,000 were independent observations are fiction. Why, then, does synthetic data help at all, and it demonstrably often does? There are four legitimate mechanisms, and it is worth keeping them distinct because each has its own conditions. The first is access. Synthetic data does not add information, but it can move information to places the original data cannot go. A synthetic copy of a dataset that has been rigorously shown to carry low disclosure risk may be shared with a university team, a vendor or the public when the real data cannot. The team receives less information than the real data contains, but much more than the nothing they would otherwise have. This is the census lineage, and it is the main justification in healthcare. The second is prior knowledge. Model-driven synthesis injects knowledge that is not in the training data: the physics of light, the geometry of road scenes, the clinical course of a disease as described in the literature, the knowledge implicit in a foundation model pre-trained on billions of images. When a driving simulator renders a child stepping out from behind a van in fog, it is not extracting that scene from a real dataset; it is combining the engineer's knowledge of what such a scene looks like with a renderer's model of how fog scatters light. This is genuinely new information relative to the fleet's logs, though its truthfulness depends on the quality of the prior. The third is regularisation and inductive bias. A downstream model trained on a small, imbalanced dataset can overfit in ways that a generator's smoother view of the data helps prevent. Oversampling the minority class, whether by interpolation or by a generative model, changes what the learner pays attention to. This can improve some measures of performance even though no information has been added; Chapter 7 shows that it can also damage others, and that simpler interventions often achieve the same effect. The fourth is invariance. Augmentation teaches a model that certain transformations do not change the label: a cat rotated ten degrees is still a cat; a lesion photographed under warmer light is still a lesion. The knowledge being added here is the invariance itself, which the practitioner supplies. Domain randomisation, discussed in Chapter 8, is the extreme form of this idea. Any claim that synthetic data improved a result should be traceable to one of these mechanisms. If it cannot be, the most likely explanation is a flaw in the evaluation: a test set contaminated by synthetic samples derived from it, a metric computed on synthetic rather than real data, or a comparison against a weak baseline. Chapter 4 returns to these errors in detail. Realism is not the goal One further distinction underpins the rest of the book. It is natural to judge synthetic data by how real it looks, and the history of generative modelling has been shaped by that instinct: GAN papers were long judged by how convincingly their faces fooled human observers, and the Fréchet Inception Distance, the most widely used image generation metric, measures how close synthetic images are to real ones in the feature space of a pre-trained classifier. But realism, in the sense of being indistinguishable from real samples, is neither necessary nor sufficient for usefulness. It is not sufficient because a generator can produce individually flawless samples while getting the distribution badly wrong. A face generator that produces only young, smiling, front-facing faces can produce perfect faces and a useless dataset. A tabular generator can produce rows that each pass every validity check while missing the correlation between income and default that a credit model needs to learn. Worse, the most realistic possible synthetic sample is a copy of a real training record, which is exactly what privacy demands must not happen. It is not necessary because some of the most effective synthetic data is plainly unrealistic. Josh Tobin and colleagues at OpenAI and Berkeley showed in 2017 that a robot's object detector trained entirely on crude simulated images, with random textures, lighting and camera positions so varied that no single image looked like the real world, could transfer to real camera images well enough to grasp objects. Jonathan Tremblay and colleagues at NVIDIA reported in 2018 that car detectors trained on domain-randomised synthetic images, again deliberately unrealistic, were competitive with detectors trained on more photorealistic synthetic data. The randomisation worked because it forced the model to rely on the features that survive every variation, which are the features that matter. The right question is therefore never "does this look real?" but "does this support the claim we need to make?" For data sharing, the claim is that analyses on the synthetic data reproduce analyses on the real data and that no individual is exposed. For model training, the claim is that the model performs better on real data. For testing, the claim is that the synthetic scenarios probe the failure modes that matter. Each claim needs its own evidence, and the generator, however impressive its samples, provides none of that evidence by itself. With that framing in place, the next two chapters turn to the generators themselves: first the adversarial approach that started the deep generative era, then the diffusion models and statistical methods that now share the field with it. CHAPTER 2 Generative Adversarial Networks and the Problem of Coverage In June 2014 Ian Goodfellow and seven co-authors at the Université de Montréal posted a paper with a simple and slightly mischievous idea. To teach a neural network to generate realistic data, set it against a second network whose only job is to tell real data from generated data. The generator improves by learning to fool the discriminator; the discriminator improves by learning not to be fooled. If both are trained well, the contest ends when the generator's output is so close to the real distribution that the discriminator can do no better than guess. Generative adversarial networks became the defining generative technique of the following six years. They produced the first synthetic faces that ordinary people could not distinguish from photographs, the first convincing synthetic medical images, and the first deep learning generators for tabular data that outperformed classical statistical synthesis on some benchmarks. They also exhibited, from the beginning, a characteristic failure that matters more for synthetic data than for almost any other application: they are very good at producing convincing samples and much less good at producing all the kinds of samples that the real data contains. This chapter explains why, and what that means for anyone using them to overcome data scarcity. The adversarial game A GAN has two components. The generator is a neural network that takes a vector of random noise, typically drawn from a standard Gaussian, and maps it to a sample: an image, a row of a table, a sequence of vital signs. The discriminator is a classifier that takes a sample and outputs the probability that it came from the real training data rather than from the generator. Training alternates between the two. In one step the discriminator is shown a batch of real samples and a batch of generated samples and updated to classify them correctly. In the next the generator is updated, with the discriminator held fixed, to make its samples more likely to be classified as real. The gradient signal that tells the generator how to improve flows backwards through the discriminator: the discriminator effectively tells the generator which direction in sample space would make each fake look more real. Goodfellow and colleagues showed that, in an idealised setting with unlimited capacity and perfect optimisation, this game has a unique equilibrium at which the generator's distribution equals the data distribution and the discriminator outputs one half everywhere. They also showed that, for a fixed generator, training the discriminator to optimality turns the generator's objective into minimising the Jensen-Shannon divergence between the real and generated distributions. That result explains both why GANs work and why they break. They work because the discriminator is a learned, adaptive critic. Older generative models were trained to maximise the likelihood of the data under the model, which for images tends to reward blurry averages that assign some probability to everything. A discriminator, by contrast, is unforgiving about any feature that distinguishes real from fake, so it pushes the generator towards sharp, specific samples. They break because the idealised conditions never hold. Real generators and discriminators have finite capacity, are trained by stochastic gradient descent on finite data, and do not reach the equilibrium. In practice the two networks can oscillate without converging, the discriminator can become so confident that its gradients vanish and the generator stops learning, and, most importantly for synthetic data, the generator can find that it fools the discriminator well enough by producing only a subset of the real data's variety. Mode collapse and why it matters for scarcity The last failure is called mode collapse. A "mode" here is loosely a cluster or type of example in the data: a digit class, a face pose, a kind of loan applicant. A collapsed generator maps many different noise vectors to the same or very similar outputs, concentrating on a few modes and ignoring the rest. In extreme cases it produces nearly identical samples regardless of input. In milder and more common cases it produces a convincing variety of common examples while under-representing or entirely omitting rare ones. The reason lies in the shape of the incentive. The generator is rewarded for each sample that the discriminator accepts. It is not directly rewarded for covering the whole distribution. If producing more examples of a common type is an easy way to keep the discriminator uncertain, the generator has no strong reason to venture into rarer regions where it is less sure of success. The discriminator does push back, because a generator that only produces one type becomes easy to detect, but the pressure is indirect and the dynamics are unstable: the generator can hop from mode to mode as the discriminator catches up, never covering all of them at once. For most image generation demonstrations, partial mode collapse is a cosmetic flaw. For synthetic data intended to overcome scarcity, it is a direct threat to the purpose. The rare cases are usually the reason for generating the data in the first place: the minority class in a credit dataset, the uncommon presentation of a disease, the unusual demographic subgroup whose under-representation in the real data is causing a model to perform badly for them. A generator whose natural failure mode is to drop rare examples is, without careful countermeasures, a tool that reproduces the majority and erases the minority. It is a striking irony that the technique most often proposed for fixing imbalance has an inbuilt tendency to worsen it. The research community developed many countermeasures. Martin Arjovsky, Soumith Chintala and Léon Bottou's Wasserstein GAN of 2017 replaced the Jensen-Shannon objective with an approximation of the Earth Mover's distance, which provides useful gradients even when the real and generated distributions do not overlap, and Ishaan Gulrajani and colleagues improved it later that year with a gradient penalty that made training markedly more stable. Conditional GANs, proposed by Mehdi Mirza and Simon Osindero in 2014, feed a label to both generator and discriminator so that the generator can be asked for a specific class; this does not remove collapse within a class, but it lets the user insist on samples from a rare class rather than hoping the generator will produce them. Minibatch discrimination, unrolled optimisation, spectral normalisation and progressive growing all helped. By 2019, Tero Karras and colleagues at NVIDIA could train StyleGAN on 70,000 high-resolution face photographs and produce faces of startling quality and diversity. None of these advances changed the fundamental point. A GAN's coverage of the data distribution is an empirical property that must be measured, not assumed. Chapter 4 describes the metrics that separate the two questions a GAN's samples raise: are they realistic (precision), and do they cover everything the real data contains (recall)? A generator can score highly on the first and poorly on the second, and the second is usually what matters for scarcity. Adapting the adversarial idea to tables and time series GANs were developed for images, which have a convenient property: every pixel is a continuous value on the same scale, and neighbouring pixels are strongly related. The data that most organisations want to synthesise looks nothing like that. A credit application table mixes continuous columns with skewed and multi-peaked distributions (income, loan amount), discrete columns with many categories (occupation, region), some of which are rare, and columns with hard logical constraints (a loan's end date must follow its start date; a person cannot be retired at nineteen). Electronic health records add high-dimensional sparse codes, irregular timing and long sequences. Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante and Kalyan Veeramachaneni addressed the tabular case in their 2019 NeurIPS paper introducing CTGAN, the conditional tabular GAN, and it remains the reference point for the field. The paper made three design choices worth understanding, because each addresses a specific way that naive GANs fail on tables. The first is mode-specific normalisation of continuous columns. A column such as transaction amount often has several peaks: many small purchases, a cluster of rent payments, a scattering of large transfers. Normalising such a column to a single mean and standard deviation squashes the peaks together and makes them hard for a neural network to reproduce. CTGAN instead fits a variational Gaussian mixture model to each continuous column, represents each value by which mixture component it belongs to plus its position within that component, and asks the generator to produce both. The second is the conditional vector. During training, CTGAN selects a discrete column at random, selects a category in that column, and asks the generator to produce a row with that category. The discriminator sees whether the row matches the condition. The third, called training-by-sampling, determines how those categories are chosen. Rather than picking categories in proportion to their frequency, which would mean rare categories were almost never requested, CTGAN samples them according to the logarithm of their frequency. Rare categories are therefore requested far more often than they occur, and the generator is forced to learn them. This is a direct, deliberate countermeasure to mode collapse on the dimension that matters for imbalance. The same paper introduced TVAE, a variational autoencoder adapted to tabular data with the same preprocessing, and a benchmark across simulated and real datasets comparing both against Bayesian networks and other baselines. The results were mixed in an instructive way: the neural methods performed well on many real datasets, but classical Bayesian network synthesisers remained competitive on some, and TVAE often matched or beat CTGAN. The broader lesson, repeated in many later benchmarks, is that deep generative models are not uniformly superior for tabular data, and that the choice of generator should be settled by measurement on the data at hand. CTGAN and TVAE were released as part of the Synthetic Data Vault, an open-source library from the MIT Data to AI Lab first described by Neha Patki, Roy Wedge and Veeramachaneni in 2016. The library's interface shows how short the path from real table to synthetic table has become, which is both its convenience and its danger. import pandas as pd from sdv.metadata import Metadata from sdv.single_table import CTGANSynthesizer real = pd.read_csv("loans.csv") metadata = Metadata.detect_from_dataframe(real) synth = CTGANSynthesizer(metadata, epochs=300) synth.fit(real) # Ask specifically for rare-class rows rather than hoping they appear from sdv.sampling import Condition defaults = synth.sample_from_conditions( [Condition(num_rows=5000, column_values={"defaulted": 1})] ) Ten lines produce five thousand synthetic defaulters. Nothing in those lines establishes whether those defaulters resemble real ones, whether a model trained on them will predict real defaults better, or whether any of them is an approximate copy of a real borrower. Those are the questions that the remaining chapters exist to answer. Clinical data posed further challenges. Edward Choi and colleagues' medGAN, presented in 2017, generated synthetic patient records consisting of high-dimensional binary and count variables such as diagnosis codes, by training an autoencoder to compress records and a GAN to generate in the compressed space. Cristóbal Esteban, Stephanie Hyland and Gunnar Rätsch proposed a recurrent conditional GAN for real-valued medical time series, using intensive care unit vital signs, and in the same 2017 paper introduced the evaluation protocol now known as "train on synthetic, test on real", which Chapter 4 discusses. Jinsung Yoon, Daniel Jarrett and Mihaela van der Schaar's TimeGAN of 2019 combined adversarial training with a supervised loss that pushed the generator to respect the step-by-step dynamics of sequences, not just their overall distribution. In medical imaging, Maayan Frid-Adar and colleagues at Tel Aviv University reported in 2018 that adding GAN-generated liver lesion images to a small training set of computed tomography scans improved a lesion classifier's sensitivity from 78.6 to 85.7 per cent and its specificity from 88.4 to 92.4 per cent compared with classical augmentation alone. The study was small, with a training set of 182 lesions, but it became one of the most cited demonstrations that synthetic data could help where real data was scarce. Its design also illustrates good practice: the generator was trained only on training data, and the improvement was measured on real test images. Diagnosing coverage in practice Because coverage is the property most likely to fail and the one most relevant to scarcity, it deserves its own routine checks before any of the broader evaluation of Chapter 4 begins. Several are cheap enough to run every time a GAN is trained. The first is a frequency audit of every categorical column, and of the cross-tabulation of the target with the categorical columns that define important segments. If the real training data contains 312 defaulters in the self-employed segment and the synthetic data, sampled at the same total size, contains a handful, the generator has collapsed on that segment regardless of how good its overall statistics look. CTGAN's training-by-sampling reduces this failure but does not eliminate it, especially for categories with very few examples, and the check costs seconds. The second is a comparison of conditional distributions for the rare class specifically. It is common to compare the distribution of income across all real rows with that across all synthetic rows and declare success. The relevant comparison for an imbalance application is the distribution of income among real defaulters against that among synthetic defaulters. Generators often reproduce the majority-class conditionals well and the minority-class conditionals poorly, simply because the minority contributed less to training. The third is a diversity check within the rare class. A collapsed generator asked for five thousand rare-class rows may produce five thousand variations on a few themes. Clustering the synthetic rare-class rows and the real rare-class rows with the same method, and comparing how many clusters each occupies and how evenly, gives a quick view of whether the synthetic data spans the real variety or concentrates on part of it. For images, the same check can be made in the feature space of a pre-trained encoder. The fourth is to watch the training dynamics. GAN losses are notoriously hard to interpret, because a falling generator loss may mean the generator is improving or that the discriminator has weakened. More useful is to sample from the generator at regular checkpoints during training and track the coverage checks above as training proceeds. It is common to find that coverage peaks at some point and then declines as the generator settles into a narrower set of modes, in which case an earlier checkpoint is the better generator. Several countermeasures from the research literature are now built into standard tools; CTGAN, for example, incorporates the "packing" technique proposed by Zinan Lin and colleagues in their 2018 PacGAN paper, in which the discriminator judges several samples jointly and can therefore detect when they are suspiciously similar. None of these checks is sophisticated. Their value lies in being aimed at the specific failure that the adversarial objective makes likely and that the purpose of the data makes costly. What GANs are still good for Diffusion models, the subject of the next chapter, have displaced GANs as the leading approach for image synthesis, and in many tabular benchmarks newer methods now outperform CTGAN. It would be a mistake, however, to treat GANs as obsolete. They have two practical advantages. The first is speed at generation time. A trained GAN produces a sample in a single forward pass through the generator, whereas a diffusion model typically requires many sequential denoising steps; for applications that need millions of samples, or generation inside a training loop, the difference is substantial. The second is a large body of engineering experience. CTGAN in particular has been used widely enough that its behaviour on common data types is well understood, and its implementations integrate with privacy techniques and evaluation tooling. GANs also have a privacy property that is sometimes cited in their favour. Because the generator never sees training data directly, only gradients passed back through the discriminator, it is sometimes claimed that GANs cannot memorise training records. That claim is false as stated. GANs can and do overfit, and membership inference attacks against them, beginning with Jamie Hayes and colleagues' LOGAN attack published in 2019, have shown that an adversary can often tell whether a given record was in the training set. What is true is that Nicholas Carlini and colleagues' 2023 study of training data extraction found diffusion models to be considerably less private than the GANs they compared them with, emitting near-exact copies of training images more readily. Neither family is private by construction; Chapter 5 explains what it takes to make either one private by design. The enduring lesson of the GAN era for synthetic data is the gap between realism and coverage. The adversarial game rewards a generator for being convincing, and convincing is not the same as complete. Every subsequent advance, from conditional sampling to the diffusion models that followed, can be understood in part as an attempt to close that gap. Whether a particular generator has closed it, on a particular dataset, for a particular rare class, remains something only measurement can tell. CHAPTER 3 Diffusion Models, Statistical Synthesisers and Choosing a Generator By 2021 the field of image generation had a new leader. Prafulla Dhariwal and Alex Nichol at OpenAI published a paper whose title stated the result plainly: "Diffusion Models Beat GANs on Image Synthesis". Within two years, diffusion models underpinned DALL-E 2, Imagen, Midjourney and Stable Diffusion, and researchers had adapted them to medical images, audio, video, molecules and tables. For synthetic data practitioners they brought two things GANs had struggled to provide: stable training and much better coverage of the data distribution. They also brought a new and sharper version of an old privacy problem. This chapter explains how diffusion models work, why their coverage is better, why they memorise, and then steps back to compare the full range of generators available today, including the statistical methods that deep learning was supposed to replace and that remain, on some data, the better choice. Learning to reverse noise The core idea is easier to state than the GAN game. Take a real sample, say an image. Add a small amount of Gaussian noise. Add a little more, and keep going for many steps until the image is indistinguishable from pure noise. This forward process requires no learning at all; it is a fixed recipe. Now train a neural network to undo one step of it: given a noisy image and the step number, predict the noise that was added, so that it can be subtracted. If the network can do this well at every noise level, then generation is simply the forward process run backwards. Start from pure noise, ask the network to remove a little, repeat, and after enough steps a clean sample emerges. Jascha Sohl-Dickstein and colleagues introduced the framework in 2015, drawing on ideas from non-equilibrium thermodynamics. It became practical with Jonathan Ho, Ajay Jain and Pieter Abbeel's 2020 paper on denoising diffusion probabilistic models, which showed that a simple training objective, essentially the mean squared error between the true and predicted noise, produced image quality competitive with the best GANs. Yang Song, Stefano Ermon and colleagues developed a closely related view in which the network learns the score, the gradient of the log-density of the data at each noise level, and showed in 2021 that both perspectives are discretisations of the same continuous stochastic process. Robin Rombach and colleagues' latent diffusion models, published in 2022 and the basis of Stable Diffusion, made the approach far cheaper by running the diffusion process in the compressed latent space of an autoencoder rather than on raw pixels. Two properties explain why diffusion models displaced GANs for many synthetic data tasks. The first is that training is a sequence of ordinary supervised regression problems. There is no adversary, no equilibrium to find, and no instability from two networks chasing each other. The loss goes down smoothly and is a meaningful indicator of progress, which makes diffusion models much easier to train reliably on new data. The second is coverage. A diffusion model's training objective is closely related to maximising the likelihood of the training data, which penalises the model for assigning low probability to any real example. A GAN that ignores a rare mode pays only an indirect price. A diffusion model that ignores a rare mode pays directly, because it will fail to denoise the examples from that mode. In practice diffusion models capture rare modes noticeably better, which is exactly the property that matters for scarcity. Diffusion models are also unusually controllable. Conditioning information, a class label, a text description, a segmentation map or a set of patient covariates, can be supplied to the denoising network at each step. Jonathan Ho and Tim Salimans' classifier-free guidance, introduced in 2021 and published in 2022, lets the user strengthen or weaken the influence of the condition at sampling time, trading diversity for adherence. For synthetic data this means a practitioner can ask for "chest radiograph, left-sided pneumothorax, elderly patient" or for "loan applicant, defaulted, self-employed, region North" and receive samples concentrated in that region of the distribution. That controllability is what makes diffusion models attractive for targeted generation of rare cases. It also makes them easy to misuse, because a strongly guided sample from a region with few training examples is the sample most likely to be a near copy of one of them. The main cost is speed. Generating one sample requires running the network many times, historically hundreds or a thousand steps, though improved samplers, distillation and consistency models have reduced this to a handful in many settings. For large synthetic datasets the sampling cost is now usually manageable, but it remains higher than for a GAN. Diffusion beyond images Medical imaging adopted diffusion models quickly. Walter Pinaya and colleagues at King's College London trained latent diffusion models on 31,740 T1-weighted brain MRI scans from UK Biobank, conditioned on age, sex and brain structure volumes, and in 2022 released a synthetic dataset of 100,000 brain images to the research community. Pierre Chambon and colleagues at Stanford fine-tuned Stable Diffusion on chest radiographs and their reports to produce RoentGen, which generates radiographs from text descriptions of findings. These projects show the attraction: high-quality, conditionally controllable images in a field where real data is hard to share and expert annotation is expensive. Tabular data required adaptation, since diffusion in its original form assumes continuous values. Akim Kotelnikov and colleagues' TabDDPM, presented at ICML in 2023, combined Gaussian diffusion for numerical columns with multinomial diffusion for categorical ones and reported better machine learning utility than CTGAN, TVAE and SMOTE across a benchmark of datasets. Later methods such as TabSyn, by Hengrui Zhang and colleagues in 2024, ran diffusion in a learned latent space for mixed-type tables. In public benchmarks these approaches have generally matched or outperformed GAN-based tabular generators on fidelity and utility measures. But the same benchmarks show something that matters as much: on several datasets the best diffusion models also produced synthetic rows lying very close to real training rows, which is good for fidelity scores and bad for privacy. Memorisation In January 2023 Nicholas Carlini and colleagues from Google, DeepMind, ETH Zurich, Princeton and Berkeley published "Extracting Training Data from Diffusion Models". Using a generate-and-filter pipeline, they extracted over a thousand training examples from state-of-the-art diffusion models, from photographs of individual people to trademarked logos. They also trained hundreds of diffusion models in controlled settings and concluded that diffusion models were much less private than the GANs they compared them with. Gowthami Somepalli and colleagues reached compatible conclusions the same year, finding that Stable Diffusion sometimes reproduced substantial portions of training images. The mechanism is intuitive once the training objective is understood. A diffusion model is rewarded for denoising every training example well. An example that appears many times in the training data, or that sits alone in a sparse region of the data space with no near neighbours, is one the model can best denoise by remembering it. Carlini and colleagues found that duplicated training images were far more likely to be extracted. The examples most at risk are therefore the duplicated and the unusual, and in the context of synthetic data the unusual are often precisely the rare cases the practitioner wants to generate more of. Medical imaging researchers have confirmed that the effect occurs in their domain. Studies of three-dimensional latent diffusion models trained on medical scans, including work by Salman Ul Hassan Dar and colleagues in 2023, found that a meaningful fraction of generated samples were near copies of training scans, especially with smaller training sets. A small training set is the normal condition in medical imaging, so this is not a corner case. The practical conclusion is not that diffusion models should be avoided but that their outputs must be audited for copying before any release, and that where privacy is a requirement, protection has to be designed into training rather than hoped for. Chapter 5 sets out the tools for both. A memorisation audit for a conditional image generator can be built from parts that are widely available, and it is worth describing concretely because it is so often skipped. Suppose a radiology department has trained a latent diffusion model on its chest radiographs and wants to generate examples of an uncommon finding to augment a classifier, with the possibility of sharing the synthetic images with an academic partner. The first step is to deduplicate the training data before training, since duplicated images are the most likely to be memorised. In medical archives duplicates are common: the same patient imaged repeatedly, the same image stored in several series, near-identical follow-up studies. Deduplication should operate at the level of the patient as well as the image, because a generator that memorises a patient's distinctive anatomy from several studies can reproduce it without copying any single image. The second step is to generate a large sample under the conditions that will actually be used, including the strongest guidance setting intended for production. Memorisation is most likely for rare conditions with strong guidance, so auditing only unconditional samples understates the risk. The third step is to embed every generated image and every training image with an encoder suited to detecting copies rather than to classification. Somepalli and colleagues used a model trained for image copy detection for this purpose; self-supervised encoders are a reasonable alternative. For each generated image, find its nearest training neighbour and record the similarity. The fourth step is calibration. Similarity scores mean little in isolation, so compute the same nearest-neighbour similarities between a held-out set of real images, which the generator never saw, and the training set. Generated images whose similarity to some training image exceeds what genuinely independent real images achieve are candidates for memorisation, and a clinician or imaging scientist should inspect the top candidates side by side with their nearest training images. The fifth step is to act on what is found: remove the near copies, investigate whether they cluster around particular patients or particular rare findings, and if they do, consider reducing guidance, enlarging or deduplicating the training data for those conditions, or excluding the affected patients from training. If the synthetic images are to leave the department, the audit becomes one input to the privacy assessment described in Chapters 5 and 6, not a substitute for it, because, as those chapters explain, the absence of visible copies does not establish the absence of leakage. The wider family of generators GANs and diffusion models are the most visible generators, but they are not the only ones, and for tabular data especially they are often not the best starting point. Statistical synthesisers fit an explicit probability model and sample from it. A Gaussian copula models each column's marginal distribution separately and captures dependence between columns through a correlation matrix on transformed values; it is fast, transparent and good at preserving marginals and pairwise correlations, but it cannot represent complex nonlinear interactions. Bayesian networks represent the joint distribution as a directed graph of conditional distributions; they are interpretable, handle mixed data well, and have a strong track record. The UK Medicines and Healthcare products Regulatory Agency used Bayesian networks, built with the bnlearn package in R, to generate its high-fidelity synthetic datasets from primary care records. Sequential regression or decision-tree synthesis, as implemented in synthpop, generates one column at a time conditional on the columns already generated, and remains a strong default for survey-style data. Under formal privacy constraints, methods that work by measuring a set of low-dimensional marginal distributions with added noise and then fitting a graphical model consistent with them, such as PrivBayes, MST and AIM, have repeatedly outperformed deep generative models, a point Chapter 5 develops. Variational autoencoders, introduced by Diederik Kingma and Max Welling in 2013, learn to compress data into a latent space with a known distribution and to decode samples from that space back into data. They are stable to train and cover the data well, but their samples are often blurrier or more averaged than those of GANs or diffusion models. TVAE showed that for tabular data this matters less than for images. Language-model-based generators treat each table row as a sentence. Vadim Borisov and colleagues' GReaT, presented at ICLR in 2023, converted rows into text such as "Age is 42, Occupation is nurse, Income is 38,000" and fine-tuned a pre-trained language model to generate such sentences. The approach benefits from the language model's prior knowledge of what plausible values look like and can condition on any subset of columns. Large language models are now also widely used to generate synthetic text data directly, a use that raises the questions about recursive training discussed in Chapter 9. Simulators and world models sit at the model-driven end of the spectrum: game engines, physics simulators and agent-based models, and more recently learned video world models such as NVIDIA's Cosmos platform, launched in January 2025, and the Waymo World Model announced in February 2026. Chapter 8 covers them in the context of edge cases. Table 1 sets out how these families compare on the attributes that usually decide a choice. Table 1. Main families of synthetic data generators compared. Family Strongest data types Coverage of rare modes Training stability Privacy notes Statistical (copula, Bayesian network, sequential trees) Tabular, survey, moderate dimension Good for modelled relationships; misses unmodelled interactions High Well suited to differential privacy via noisy marginals Variational autoencoder Tabular, simple images Good, but samples smoothed High No inherent protection; can be trained with DP-SGD Generative adversarial network Images, tabular, time series Weak without countermeasures (mode collapse) Low to moderate Memorises less readily than diffusion, but not private by default Diffusion model Images, video, audio, tabular Strong High Prone to copying duplicated and isolated examples Language-model-based Text, tabular as text Depends on fine-tuning data and prior High Can regurgitate training sequences Simulator or world model Sensor data, scenes, physical systems Whatever the scenario designer specifies Not applicable or high No record-level risk from hand-built models; learned world models inherit training data risks The table makes a point that is worth stating directly. There is no generator that is best on all attributes. Statistical methods are transparent and privacy-friendly but limited in what they can express. GANs are fast but drop modes. Diffusion models cover well and copy readily. Simulators provide control and no record-level privacy risk but only the realism their designers build in. How to choose A sensible selection process follows from the purpose, not from the novelty of the method. Start with the job the data must do, as defined in Chapter 1. If the data is for software testing, a schema-aware rule-based generator or a simple copula may be entirely sufficient, and anything more powerful adds privacy risk for no benefit. If it is for broad data sharing under a privacy requirement, a differentially private marginal-based method is often the strongest option for tabular data, because its privacy guarantee is formal and its utility under privacy constraints is well documented. If it is for training a model on a rare class in high-dimensional data such as images, a conditional diffusion model is likely the most capable generator, and it must be paired with memorisation audits. If it is for testing a perception system on scenarios that have never been recorded, simulation or a world model is the only option, and validation must focus on the gap between simulated and real sensor data. Then let measurement decide among the candidates. A realistic plan trains two or three generators of different families on the same training split, evaluates each with the fidelity, utility and privacy measures of the next two chapters, and chooses the one that best meets the stated purpose. The cost of training several tabular generators is usually small relative to the cost of being wrong. Finally, keep the simplest adequate baseline in every comparison. For class imbalance, that baseline is reweighting or threshold adjustment with no synthetic data at all. For data sharing, it is a properly de-identified real extract under a data use agreement. For image augmentation, it is classical augmentation. Synthetic data justifies its cost and risk only by beating these baselines on real-world evaluation, and the next chapter explains how to tell whether it has. CHAPTER 4 Measuring Fidelity and Utility A synthetic dataset arrives with a vendor's quality report. Every column's distribution matches the original to within a percentage point. The correlation matrix is nearly identical. A classifier trained to tell real rows from synthetic ones performs only slightly better than chance. A model trained on the synthetic data achieves an area under the ROC curve within 0.02 of one trained on real data. Is the dataset good? The honest answer is that it depends on what it will be used for, and that the report, while reassuring, may not have measured the thing that matters. Column distributions say nothing about three-way interactions. A good average discrimination score can conceal a subgroup on which the synthetic data is badly wrong. A downstream model's overall accuracy can hold up while its performance on the rare class, the reason the data was generated, collapses. And none of these figures says anything about privacy. This chapter sets out how to evaluate fidelity and utility rigorously. Privacy is treated separately in the next chapter, not because it is less important but because it requires a different way of thinking: fidelity and utility ask whether the synthetic data resembles the population, while privacy asks whether it resembles particular individuals too closely. From marginals to the joint distribution Fidelity measures how closely the synthetic data reproduces the statistical structure of the real data. It is natural to organise fidelity checks by how many variables they consider at once, because the failures of generators tend to appear at increasing levels of interaction. The first level is univariate. For each column, compare the real and synthetic distributions. For continuous columns the Kolmogorov-Smirnov statistic, the largest gap between the two cumulative distribution functions, is a common choice; for categorical columns, the total variation distance between category frequencies. Univariate checks are cheap and catch gross errors such as a generator that never produces incomes above a certain level, or a category that has vanished entirely. They are also the checks most generators pass easily, because matching marginals is the easiest part of the problem. A synthetic dataset in which every column was sampled independently from its real marginal would score perfectly at this level and be useless for almost any analysis. The second level is pairwise. Compare correlation coefficients for pairs of numeric columns, contingency tables for pairs of categorical columns, and appropriate association measures for mixed pairs. Pairwise checks catch the failure that independent sampling exhibits, and they are what most quality reports present as heatmaps. But they share a blind spot: many important relationships are interactions among three or more variables. In credit data, the relationship between income and default may differ by employment type; in clinical data, the effect of a drug on an outcome may depend on age and kidney function together. A generator can reproduce every pairwise relationship and get these interactions wrong. The third level addresses the full joint distribution, and the most useful tool here is the discriminator test, also called the propensity score method. Combine the real and synthetic rows into one dataset, label each by its origin, and train a classifier to predict the label. If the synthetic data were a perfect draw from the same distribution as the real data, no classifier could do better than chance. If the classifier succeeds, the features it relies on reveal where the synthetic data departs from the real. Joshua Snoke, Gillian Raab, Beata Nowok, Chris Dibben and Aleksandra Slavković formalised a version of this idea in 2018 as the propensity score mean squared error, or pMSE, which measures how far the classifier's predicted probabilities depart from the proportion expected under perfect synthesis, and derived its expected value under a correct synthesis model so that scores could be interpreted. The discriminator test is powerful precisely because it is adversarial: a flexible classifier such as a gradient-boosted tree ensemble will find whatever departures exist, including high-order ones. Its result depends on the classifier's capacity, so it should always be reported with the classifier specified, and the classifier's feature importances or misclassified examples should be inspected rather than just the score. A result that real and synthetic data are distinguishable mainly through a single column, such as a date field with an impossible spread, points to a fixable problem. A result that they are distinguishable through diffuse combinations of many columns points to a generator that has not learned the joint structure. Alongside these statistical checks, every synthetic dataset needs validity checks: rules that real data always satisfies and synthetic data must satisfy too. Discharge dates must follow admission dates. A diagnosis cannot be dated after the patient's recorded death, and a paediatric dosing code cannot attach to a patient aged seventy. A loan term must match the product type. Generators violate such constraints surprisingly often because they learn probabilities, not rules, and a small rate of violation becomes conspicuous at scale. The constraint list should come from domain experts, not be inferred from the data, and should be enforced either inside the generator, as several tabular libraries now allow, or by rejecting invalid samples. Utility: does it work for the job? Utility is fidelity judged by consequences. The question is not whether the synthetic data resembles the real data in general but whether it supports the specific use intended. Two families of utility measure correspond to the two main uses. For statistical analysis, the classic approach is specific utility: run the analyses the data is meant to support on both real and synthetic versions and compare the results. If researchers will fit logistic regressions of readmission on a set of covariates, fit that regression on both datasets and compare coefficients and confidence intervals. Alan Karr and colleagues proposed measuring the overlap of the real and synthetic confidence intervals for each estimate, which captures both bias and differences in uncertainty. The limitation is obvious: specific utility only tells you about the analyses you thought to test, and a synthetic dataset released for exploratory use will be subjected to analyses nobody anticipated. This is why the best data-sharing programmes, following the census model, pair synthetic releases with a route to validate important findings against the real data. For machine learning, the standard protocol is train on synthetic, test on real, abbreviated TSTR, introduced by Cristóbal Esteban, Stephanie Hyland and Gunnar Rätsch in their 2017 work on medical time series. Train a model on synthetic data and evaluate it on a held-out real test set. Compare with the same model trained on the real training data and tested on the same real test set, known as TRTR. The gap between TSTR and TRTR measures how much predictive information was lost in synthesis. Where the purpose is augmentation rather than replacement, the relevant comparison is between a model trained on real data alone and one trained on real plus synthetic data, both tested on real data. TSTR is conceptually simple and easy to get wrong. Four errors recur often enough to list. The first and most serious is leakage through the generator. If the generator was trained on data that includes the test set, synthetic samples can carry information about test records, and TSTR scores will be inflated. The generator must be trained only on the training split, and the real test set must be locked away before generation begins. This sounds obvious, but in practice synthetic data is often generated once, from the full real dataset, and then handed to modelling teams who split it later or test on a subset of the same real data. The second is testing on synthetic data. Evaluating a model on synthetic test data, sometimes called TRTS or TSTS, measures how well the model fits the generator's view of the world, not the world. Such numbers can be useful diagnostics of generator behaviour but should never be reported as model performance. The third is aggregate metrics that hide the target. If synthetic data was generated to improve performance on a minority class or a subgroup, the evaluation must report performance on that class or subgroup, with confidence intervals reflecting the small number of real test cases. An overall accuracy gain driven by the majority class says nothing about whether the synthetic minority cases helped. The fourth is weak baselines. A model trained on real plus synthetic data should be compared not just with the same model trained on real data, but with the best reasonable alternative that uses no synthetic data: class weighting, threshold tuning, classical augmentation, transfer learning from a related dataset. Many published gains from synthetic data shrink or vanish against such baselines. A related check, useful when synthetic data will be used to select between models rather than train them, is ranking preservation. Train several model types or hyperparameter settings on real data and on synthetic data, and check whether the synthetic data ranks them in the same order. A synthetic dataset that preserves rankings can safely support model development work, with final validation on real data; one that scrambles rankings will lead developers to the wrong model. Uncertainty: the measure most often forgotten There is a further property that matters whenever synthetic data will be used to estimate something about the real population, and it is almost never reported by machine learning tools: whether the synthetic data conveys the right amount of uncertainty. Consider a researcher who estimates, from a synthetic dataset of 50,000 patients, that a treatment is associated with a reduction in readmission, with a narrow confidence interval computed in the ordinary way. Two sources of uncertainty are missing from that interval. The first is the sampling uncertainty of the original real data: if the real dataset had 5,000 patients, the evidence available about the population is the evidence of 5,000, and the larger synthetic sample does not add to it, as the information limit of Chapter 1 requires. The second is the uncertainty introduced by synthesis itself: a different run of the generator, with a different random seed, would have produced a different synthetic dataset and a somewhat different estimate. The statistical disclosure control literature solved this problem for its own methods decades ago. Trivellore Raghunathan, Jerome Reiter and Donald Rubin's 2003 paper set out combining rules for fully synthetic data: generate several independent synthetic datasets, each the same size as the real data, perform the analysis on each, and combine the estimates using formulas that account for both the within-dataset variance and the variance between synthetic datasets. Reiter and others developed corresponding rules for partially synthetic data. The rules depend on the generator being a proper Bayesian model whose parameters are drawn from their posterior distribution for each synthetic dataset, which is how the census-style methods work, and which most deep generative models do not attempt. For practitioners using GANs or diffusion models, a pragmatic version of the same idea is still valuable. Train the generator several times with different seeds, or on bootstrap resamples of the real training data, generate a synthetic dataset from each, and repeat the analysis on each. The spread of results across synthetic datasets gives at least a lower bound on the extra uncertainty that synthesis introduces. Generate synthetic datasets of the same size as the real data unless there is a reason not to, so that naive standard errors are not deflated by an artificially large sample. And when synthetic data is shared for analysis, tell recipients plainly that standard errors computed from it do not reflect the uncertainty of the underlying evidence. A surprising amount of misinterpretation is prevented by that one sentence. For model training the analogous concern is variance in the utility result itself. A single train-on-synthetic, test-on-real experiment with one generator run and one model run can give a misleadingly favourable or unfavourable picture. Reporting the spread across several generator seeds and several model seeds, and computing confidence intervals on the real test metrics, especially for rare classes with few real test examples, is the minimum needed to claim that synthetic data helped. Measuring images, and the limits of a single number Image generation has its own metrics, and they have their own problems. The Inception Score, proposed by Tim Salimans and colleagues in 2016, and the Fréchet Inception Distance, proposed by Martin Heusel and colleagues in 2017, both pass images through an Inception network pre-trained on ImageNet and compare statistics of its outputs or internal features. FID, the more widely used, fits a Gaussian to the feature vectors of real images and another to those of synthetic images and computes the Fréchet distance between them. Lower is better. FID has three well-documented limitations. First, it measures distance in the feature space of a network trained on photographs of everyday objects. For chest radiographs, histopathology slides or infrared industrial images, those features may not capture what matters clinically or operationally, and a low FID is weak evidence of domain-relevant fidelity. Second, FID is biased by sample size, so scores computed on different numbers of images are not comparable; Min Jin Chong and David Forsyth showed this in 2020 and proposed corrections. Third, and most important for scarcity, FID collapses two different properties into one number. A generator that produces flawless images of only common types, and a generator that covers all types but produces some poor-quality images, can have similar FID. The response has been to separate the two properties. Mehdi Sajjadi and colleagues in 2018, and Tuomas Kynkäänniemi and colleagues in 2019, defined precision and recall for generative models. Precision is the fraction of synthetic samples that fall within the region of feature space occupied by real samples: are the samples realistic? Recall is the fraction of real samples that fall within the region occupied by synthetic samples: does the synthetic data cover the real variety? Muhammad Ferjad Naeem and colleagues refined these into density and coverage in 2020 to make them more robust to outliers. A mode-collapsed GAN shows high precision and low recall, which is exactly the diagnosis a synthetic data practitioner needs. George Stein and colleagues' 2023 study of generative model evaluation found that the choice of feature extractor matters greatly and that features from self-supervised models such as DINOv2 agreed with human judgements of image quality better than Inception features did. Ahmed Alaa, Boris van Breugel, Evgeny Saveliev and Mihaela van der Schaar extended these ideas in a 2022 ICML paper, proposing alpha-precision and beta-recall, which measure fidelity and diversity relative to the typical part of the distribution, and a third measure they called authenticity, which estimates the fraction of synthetic samples that are genuinely new rather than copies or near copies of training samples. Authenticity is an important conceptual bridge to the next chapter: it is the first fidelity-style metric that rewards a generator for not reproducing what it was trained on. Table 2 summarises the main families of measure, what each establishes and what each leaves out. Table 2. Families of fidelity and utility measure and their blind spots. Measure family Example metrics What it establishes What it misses Univariate Kolmogorov-Smirnov statistic, total variation distance Each column's distribution is reproduced All relationships between columns Pairwise Correlation difference, contingency table distance Two-way associations are reproduced Interactions among three or more variables Discriminator or propensity pMSE, classifier AUC for real versus synthetic Overall joint distribution is hard to tell apart Which differences matter for the intended use Specific utility Coefficient and confidence interval overlap Particular planned analyses give the same answers Unplanned analyses Machine learning utility Train on synthetic, test on real; augmentation gain Downstream models work on real data Privacy; subgroup effects if not disaggregated Precision and recall family Precision, recall, density, coverage, alpha-precision, beta-recall Realism and coverage separated Domain-specific relevance of the feature space An evaluation protocol that matches the purpose Pulling these elements together, a defensible evaluation for a synthetic dataset follows a fixed order. Before any generator is trained, write down the intended use and the claim the synthetic data must support. "Data scientists at the partner university will use it to develop and compare readmission models, with final models validated on real data by our team" is a claim. "It will be realistic" is not. The written claim determines which metrics are decisive. Split the real data and lock away the test set. If the data has a time dimension, split by time, because a generator trained on data from a later period than the test set can leak future information. Hold back a further set of real data for privacy evaluation, for reasons the next chapter explains. Train the candidate generators on the training split only. Run validity checks and reject or repair invalid samples, recording how many were rejected; a high rejection rate is itself a fidelity signal. Run the fidelity hierarchy: univariate, pairwise, discriminator. Disaggregate every result by the subgroups that matter for the use, including any protected characteristics and the rare classes that motivated generation. A synthetic dataset can match the overall population well and represent a minority ethnic group, an age band or a product line badly, and only disaggregation reveals it. Run the utility tests that correspond to the claim, with strong baselines and real test data, reporting performance on the target class or subgroup with confidence intervals. Finally, interpret the results against the claim and record them. The output of this process is not a single quality score but a short document stating what the synthetic data was evaluated for, how, with what results and with what known limitations. Chapter 9 describes how that document becomes part of the data product. This protocol is more work than generating data and looking at a correlation heatmap. It is also the only way to know whether the synthetic data has done anything at all, and it is the foundation on which the privacy evaluation of the next chapter is built. CHAPTER 5 What Synthetic Data Protects, and What It Does Not The most common argument for synthetic data runs as follows. Traditional anonymisation alters real records, and altered real records can be re-identified: researchers have repeatedly matched "anonymous" datasets to named individuals using a handful of quasi-identifiers. Synthetic data contains no real records. Every row is generated. There is no one-to-one mapping between a synthetic row and a real person, so there is nobody to re-identify. Therefore synthetic data is private. Each step of this argument sounds reasonable, and the conclusion is false. It is false not because synthetic data offers no protection, it often offers a great deal, but because the argument locates privacy risk in the wrong place. The risk was never that a particular row would be traced to a particular person. The risk is that the released data allows someone to learn something about a particular person that they could not otherwise have learned. A generator trained on personal data is a compressed representation of that data, and its samples can leak what it has absorbed. This chapter explains how that leakage happens, how it is measured, and what it takes to bound it. It is the technical foundation for the legal discussion in the next chapter, and the order is deliberate: the law asks whether individuals are identifiable by means reasonably likely to be used, and answering that question requires knowing what means are available. How synthetic data leaks Privacy researchers group the harms of a data release into three categories, which the European data protection authorities set out in the Article 29 Working Party's 2014 opinion on anonymisation techniques and which the European Data Protection Board's draft anonymisation guidelines of July 2026 restate in modernised form. Singling out means isolating the records that relate to one individual. Linkability means connecting records about the same individual across datasets. Inference means deducing the value of an attribute of an individual with significant probability. Synthetic data most obviously defeats singling out in its naive form, since no synthetic row is a person's record. It is far less obviously protected against inference, and inference is where the strongest attacks have focused. The first family of attacks is membership inference. The adversary holds the synthetic dataset and a target record, and wants to know whether the target was in the generator's training data. That may sound like a modest goal, but membership is often itself sensitive: if the training data was a registry of people with HIV, a psychiatric hospital's admissions or a list of fraud investigations, learning that someone was in it reveals the sensitive fact directly. Reza Shokri and colleagues demonstrated membership inference against classification models in 2017. Jamie Hayes, Luca Melis, George Danezis and Emiliano De Cristofaro extended it to generative models with their LOGAN attack, published in 2019, showing that the discriminator of a trained GAN, or a model trained to mimic one, tends to assign higher confidence to training records than to unseen ones. Against released synthetic tables, the most influential evaluation is Theresa Stadler, Bristena Oprisanu and Carmela Troncoso's "Synthetic Data – Anonymisation Groundhog Day", presented at USENIX Security in 2022. They used shadow modelling: the attacker trains many copies of the generator on datasets with and without the target, learns what the synthetic output looks like in each case, and uses that knowledge to judge the real release. Their conclusion, across a range of generators, was that synthetic data either did not prevent inference attacks or did not retain data utility, that it did not provide a better trade-off between privacy and utility than traditional anonymisation, and that its privacy gain was highly variable and hard to predict. Crucially, they found that the records at greatest risk were outliers: people whose combination of attributes was unusual. Those are precisely the people a rare-class generator is asked to model most carefully. Later work refined the tools. Florimond Houssiau and colleagues released TAPAS, a toolbox for adversarial privacy auditing of synthetic data, in 2022. Boris van Breugel and colleagues' DOMIAS attack of 2023 exploited the observation that generators tend to overfit locally, producing slightly too much density near their training points, and detected membership by comparing synthetic density with a reference estimate of the true population density. Shadow modelling is worth understanding step by step, because it shows why privacy depends on the generator and the data together rather than on the appearance of the output. Suppose an attacker wants to know whether a particular patient, whose record they know, was in the training data of a published synthetic dataset. They obtain or assemble a reference dataset drawn from a similar population, which in practice might be a public dataset, an earlier release or a purchased marketing file. From it they build many training sets, half of which include the target record and half of which do not. They train the same kind of generator, using the published method, on each, and sample a synthetic dataset from each. Now they have many labelled examples of what synthetic data looks like when the target was in training and when it was not. They summarise each synthetic dataset with features, such as counts of records near the target or statistics of the columns the target's record would influence, and train a classifier to tell the two cases apart. Finally they apply that classifier to the real published synthetic dataset. If it can tell, the release leaks membership. Two features of this attack are important. It requires the attacker to know the generation method, which is a reasonable assumption since methods are usually published and good security practice assumes that the adversary knows the system. And its success varies enormously from record to record. For a typical record in a dense part of the data, the generator's output barely changes whether the record is included or not, and the attack fails. For an outlier, whose inclusion pulls the generator noticeably towards an otherwise empty region of the data, the attack can succeed with high confidence. Averages over all records therefore hide the risk; it is concentrated in the people who are most unusual. The second family is attribute inference. The adversary knows some attributes of a target, perhaps age, postcode and sex from public sources, and wants to learn a sensitive attribute such as a diagnosis or credit default. Meenatchi Sundaram Muthu Selva Annamalai, Andrea Gadotti and Luc Rocher showed at USENIX Security in 2024 that a linear reconstruction approach could mount such attacks across entire synthetic datasets, not only against outliers, and that it worked against probabilistic graphical models, GANs and recent differentially private generators. They reported a trade-off that runs against the instincts of anyone generating data for scarcity: releasing more synthetic records improved statistical quality and also made inference substantially more effective. Their conclusion was that current generation methods could not consistently provide sufficient protection against inference attacks while retaining reasonable utility, and that differential privacy gave meaningful protection only in specific settings. The third family is extraction, in which the generator simply reproduces training records or close approximations of them. Carlini and colleagues' 2023 extraction of over a thousand training images from diffusion models, described in Chapter 3, is the clearest demonstration. In tabular data the equivalent is a synthetic row that matches a real training row on all or nearly all columns. For a dataset of common records this may be harmless, because many real people share the same combination, but for an unusual record it is a disclosure. Why similarity checks are not enough Many commercial and open-source tools assess the privacy of synthetic data with similarity metrics. The commonest is distance to closest record: for each synthetic row, compute the distance to the nearest real training row, and check that synthetic rows are not systematically closer to training rows than real held-out rows are. Related checks count exact matches and compare nearest-neighbour distance ratios. These checks are useful hygiene. A synthetic dataset that contains exact copies of training records, or rows much closer to the training data than held-out data would be, has a clear problem, and catching it is valuable. But passing these checks is not evidence of privacy, for three reasons. First, they test only for one kind of leakage, close copies, whereas membership and attribute inference attacks exploit statistical patterns that do not require any synthetic row to be close to any real one. A generator that slightly over-represents the neighbourhood of an unusual training record can reveal that record's presence without reproducing it. Second, the thresholds are arbitrary. There is no principled answer to how far a synthetic row must be from its nearest real neighbour to be safe, and the answer certainly differs between a common record in a dense region of the data and an outlier in a sparse one. Third, an adversary who knows the checks can be accommodated by a generator that passes them while still leaking. Georgi Ganev and Emiliano De Cristofaro demonstrated this concretely in a paper first circulated in 2023 and given a distinguished paper award at the IEEE Symposium on Security and Privacy in 2025, whose title states its argument: "The Inadequacy of Similarity-based Privacy Metrics: Privacy Attacks against 'Truly Anonymous' Synthetic Datasets". They showed attacks, including reconstruction of training records, succeeding against synthetic data that passed the kinds of similarity-based checks used in commercial products. The general lesson is that privacy cannot be established by inspecting the output for things that look like the input. It has to be established either by attacking the release with the strongest realistic adversary, which yields empirical evidence about specific threats, or by constraining the generation process mathematically, which yields a guarantee. Differential privacy The mathematical approach is differential privacy, introduced by Cynthia Dwork, Frank McSherry, Kobbi Nissim and Adam Smith in 2006. A randomised algorithm is differentially private if its output distribution changes very little when any single individual's data is added to or removed from the input. The parameter epsilon bounds how much: with a small epsilon, an observer of the output cannot tell with much confidence whether any particular person contributed. A second parameter, delta, allows a small probability of the bound failing and is usually set to be much smaller than one divided by the dataset size. Differential privacy has three properties that make it fit synthetic data unusually well. It is a property of the generation process, not of the output, so it holds whatever auxiliary information the adversary has. It composes: the privacy costs of several releases add up in a quantifiable way. And it is preserved by post-processing: anything computed from a differentially private output, including any number of synthetic samples drawn from a differentially private generator, carries the same guarantee. A generator trained with differential privacy can therefore be sampled indefinitely without spending further privacy budget. There are two broad routes to differentially private synthetic data. The first trains a deep generator with differentially private stochastic gradient descent, introduced by Martín Abadi and colleagues in 2016, which clips each example's contribution to the gradient and adds calibrated noise. Liyang Xie and colleagues' DPGAN of 2018 applied this to GANs. James Jordon, Jinsung Yoon and Mihaela van der Schaar's PATE-GAN, presented at ICLR in 2019, instead adapted the Private Aggregation of Teacher Ensembles framework, training several discriminators on disjoint subsets of the data and aggregating their votes with noise. The second route measures a selected set of low-dimensional marginal distributions, such as the joint distribution of each pair or triple of columns, adds noise to the measurements, and fits a model consistent with the noisy marginals from which to sample. Jun Zhang and colleagues' PrivBayes used a Bayesian network in this way. Ryan McKenna, Gerome Miklau and Daniel Sheldon developed the approach that won the 2018 differential privacy synthetic data challenge run by the US National Institute of Standards and Technology, later published as MST, and McKenna and colleagues' AIM of 2022 selected which marginals to measure adaptively. For tabular data, the marginal-based route has generally won. Yuchao Tao and colleagues' benchmark of differentially private synthetic data generators, published in 2021, found that marginal-based methods outperformed GAN-based ones across most datasets and privacy levels. The reason is that noise added to a few aggregate counts damages much less information than noise added to every gradient step of a large neural network. Differential privacy is not free, and the costs fall unevenly. Utility declines as epsilon falls, and the decline is steepest for the rare and the unusual, because their signal is the easiest for noise to drown. Eugene Bagdasaryan, Omid Poursaeed and Vitaly Shmatikov showed in 2019 that differentially private training reduced classifier accuracy most for under-represented groups. Georgi Ganev, Bristena Oprisanu and Emiliano De Cristofaro showed the same pattern for differentially private synthetic data in 2022, finding that it tended to distort the representation of minority subgroups. This creates a genuine tension for the applications in this book: the privacy mechanism that protects outliers does so by blurring them, and the scarcity applications exist to sharpen them. A related subtlety is the unit of privacy. Most differential privacy implementations for tabular data protect a single row. In many real datasets a person contributes many rows: an electronic health record holds a row per encounter, a transaction file a row per payment, a credit bureau file a row per account. A guarantee that protects any one row protects a person with fifty rows far less, because the privacy loss accumulates across all of their rows. Correct implementations either restructure the data so that each person contributes one record, bound the number of rows any person can contribute, or account for the group size in the privacy budget. A claim of differential privacy for longitudinal or relational data should always state whether the unit protected is a row or a person. There is also no universally agreed value of epsilon. Academic papers commonly report results for epsilon around 1 or below; deployed systems have used much larger values. An epsilon is meaningful only alongside the unit of privacy (a record, a person, a household), the time period it covers, and the rest of the threat model. A vendor's statement that synthetic data is "differentially private" without these details conveys little. Finally, differential privacy covers only what is inside the mechanism. Implementations commonly leak through preprocessing done on the raw data outside it: computing the minimum and maximum of each column to set value ranges, enumerating the categories that appear, or choosing hyperparameters by repeatedly evaluating on the real data. Each of these can reveal information about individuals, and a correct implementation either derives such metadata from public knowledge or spends privacy budget to compute it. A practical privacy evaluation For a practitioner, these findings translate into a defensible process. Begin with the threat model. Who will receive the synthetic data, and what else might they know? A synthetic dataset shared under contract with a vetted research team has a different threat model from one posted publicly. Write down the plausible adversary's auxiliary knowledge: public registers, social media, other releases from the same organisation. Decide whether a formal guarantee is required. For public release of data about sensitive populations, differential privacy with a stated epsilon and unit of privacy is the strongest available position. For controlled sharing, empirical evaluation combined with contractual and technical controls may be proportionate. Run empirical attacks regardless. Even a differentially private generator should be attacked, because implementation bugs are common. Hold back a set of real records that the generator never saw, and use it as the "non-member" population for membership inference. Run at least a shadow-model membership inference attack and an attribute inference attack for the sensitive attributes. Open-source tools including TAPAS and Anonymeter, the latter described by Matteo Giomi and colleagues in 2023 and designed explicitly around the singling out, linkability and inference criteria, make this practical without a dedicated research team. Look at the vulnerable records, not just averages. Average attack success across a dataset can be near chance while a handful of outliers are almost perfectly exposed. Report the worst cases, and consider whether those records should be removed, coarsened or handled separately before generation. Run the similarity checks as hygiene, not as proof. Remove exact and near-exact matches, and investigate why the generator produced them. Record the result. As with fidelity and utility, the output of this process is a short written assessment stating the threat model, the protections, the attacks run and their results, and the residual risk. That document is what a data protection officer, an expert determination under HIPAA or a regulator will want to see, and the next chapter explains why. CHAPTER 6 Healthcare, Privacy Law and the Limits of "Bypassing" Regulation Healthcare is the field where the access problem is most acute and the promise of synthetic data most often overstated. Clinical records are among the richest data any society holds. They are also, rightly, among the most protected. A researcher who wants to train a model to predict acute kidney injury, or a start-up building software to triage referrals, typically needs approvals from data controllers and ethics committees, a lawful basis under data protection law, agreements on security and onward use, and often access only inside a secure environment from which nothing leaves without checking. The process exists for good reasons, and it is slow. Synthetic data is frequently sold as the way around this. The pitch says that because synthetic patients are not real patients, synthetic health data is not personal data, and therefore the rules on personal data do not apply. Some vendors and commentators go further and describe synthetic data as a way to "bypass" privacy regulation. That framing is wrong, and an organisation that acts on it is likely to break the law and to mislead the patients whose data it holds. The accurate position is more useful than the inaccurate one. Synthetic data can substantially reduce privacy risk. Generated and evaluated properly, it can produce outputs that fall outside data protection law because they are genuinely anonymous, and it can support compliance in other ways even when they are not. But the law applies to the process of making synthetic data, it applies to the output unless the output meets the law's own test for anonymity, and that test is not met merely because the records were generated by a model. This chapter explains how the main regimes treat synthetic health data, and then describes what a well-governed programme looks like. What healthcare stands to gain The gains are real enough to be worth stating before the constraints. In the United Kingdom, the Clinical Practice Research Datalink, run by the Medicines and Healthcare products Regulatory Agency, holds anonymised primary care records for many millions of patients. With support from the government's Regulators' Pioneer Fund, its team developed a framework for generating and evaluating synthetic data and released high-fidelity synthetic datasets built with Bayesian networks: one on cardiovascular disease risk factors and one on COVID-19 symptoms and risk factors. They were designed for statistical analysis and for machine learning development, and CPRD commissioned an independent assessment of their status under UK data protection law. A researcher can use them to develop and debug methods, to train students, or to test whether an idea is worth a full application for the real data. Synthea, the model-driven generator described in Chapter 1, has supplied fictional but clinically structured patient records for testing health information systems and interoperability standards, with no privacy risk from real records because none were used. The 100,000 synthetic brain MRI scans released by Walter Pinaya and colleagues let researchers without access to UK Biobank experiment with brain imaging methods. GAN and diffusion augmentation, as in Maayan Frid-Adar's liver lesion study, has improved models trained on small imaging datasets inside a single institution, where the privacy question is different because the synthetic data never leaves. These are three distinct uses with three distinct privacy profiles. Model-driven data from published statistics carries essentially no individual-level risk. Data-driven synthetic data used inside the organisation that holds the real data raises few new issues because it stays within the same controls. Data-driven synthetic data released outside that organisation is where the legal questions concentrate, and it is the case the rest of this chapter addresses. The European position The EU General Data Protection Regulation applies to personal data, defined in Article 4(1) as information relating to an identified or identifiable natural person. Recital 26 adds the test that matters: to decide whether a person is identifiable, account should be taken of all the means reasonably likely to be used, by the controller or by another person, to identify them, considering the costs, the time required and the available technology. The same recital says the regulation does not apply to anonymous information, meaning information that does not relate to an identifiable person or personal data rendered anonymous in such a way that the person is no longer identifiable. Three consequences follow for synthetic data, and they hold however the synthetic data is produced. The first is that generation is processing. Training a generator on health records uses personal data, and health data is a special category under Article 9. The organisation doing it needs a lawful basis under Article 6 and a condition under Article 9, must respect purpose limitation, must carry out a data protection impact assessment where the processing is likely to be high-risk, and must meet transparency obligations to the people concerned. The European Data Protection Board's Guidelines 02/2026 on anonymisation, adopted in draft in July 2026 and open for consultation until the end of October 2026, state explicitly that anonymisation is itself processing requiring a legal basis. Generating synthetic data is a form of anonymisation attempt, so the same reasoning applies. Where the original data was collected for care, the controller must consider whether synthesis for research is a compatible purpose; Article 5(1)(b) treats further processing for scientific research as compatible when it meets the safeguards of Article 89, but that presumption has conditions and does not remove the need for a lawful basis. The second is that the output is personal data unless it is shown to be anonymous. There is no exemption for synthetic data in the GDPR, and none in the EDPB's draft guidelines, which warn that synthetic datasets and AI models are not automatically anonymous. The draft guidelines frame anonymity around three criteria that descend from the Article 29 Working Party's 2014 opinion: no isolation of an individual's records, no linkage of records relating to the same individual, and no inference of meaningful information about an individual. Each maps directly onto the attacks in Chapter 5. A synthetic dataset from which an attacker can infer, with significant confidence, whether a known person was in the training data, or what their diagnosis was, fails the inference criterion, whatever its rows look like. The EDPB's Opinion 28/2024 on AI models, adopted in December 2024, applied similar reasoning to trained models themselves: a model trained on personal data can be regarded as anonymous only if the likelihood of extracting personal data from it, directly or through queries, is insignificant. A generator is an AI model, so a decision to share the generator rather than its samples falls squarely under that opinion. The third is that the assessment is contextual. On 4 September 2025 the Court of Justice of the European Union decided EDPS v SRB (Case C-413/23 P), holding that pseudonymised data is not necessarily personal data for every recipient: data can be non-personal in the hands of a recipient who cannot reasonably re-identify individuals, while remaining personal data for the controller who holds the means to do so. The Court also held that the controller's transparency obligations are assessed from its own perspective at the time of collection. For synthetic data this cuts in two directions. It supports the view that a carefully generated synthetic dataset, shared with a recipient under conditions that make re-identification unrealistic, can be anonymous for that recipient. But it ties the assessment to the recipient and the conditions, so a dataset that is anonymous when shared with a vetted research team under contract may not be anonymous when posted publicly, where the means reasonably likely to be used include those of any motivated attacker. The EDPB's draft guidelines reflect this with two methods: a simplified approach that assumes the worst-case recipient, and a contextual approach that assesses the actual recipients and controls. Nothing here makes synthetic data legally unattractive. If a synthetic dataset is shown, by the kind of evaluation described in Chapter 5, to meet the no-isolation, no-linkage and no-inference criteria for its intended recipients, it is anonymous, and its onward use is outside the GDPR. That is a strong position for data sharing. But it is a position reached by demonstration, not by assertion, and the processing needed to reach it is fully within the regulation. The United Kingdom and the United States The United Kingdom retains the GDPR as UK GDPR, with the same structure. The Information Commissioner's Office published its guidance on anonymisation and pseudonymisation on 28 March 2025. It treats identifiability as context-specific and lying on a spectrum, and asks organisations to consider whether a "motivated intruder", a reasonably competent person with access to public resources but no specialist equipment, could identify individuals. The ICO's guidance on privacy-enhancing technologies discusses synthetic data as one such technology and is explicit that it can carry re-identification risk, especially where it closely reproduces outliers. The practical test in the UK is therefore the same as in the EU: demonstrate that the risk is remote for the context, and keep the assessment under review as attack techniques improve. In the United States, health data held by covered entities and their business associates is governed by the HIPAA Privacy Rule. Its structure is different from the GDPR's in a way that matters for synthetic data. The Privacy Rule permits a covered entity to use protected health information to create de-identified information, under 45 CFR 164.502(d), so generating synthetic data inside a covered entity, or by a business associate whose agreement permits it, is an allowed use. The output escapes the rule only if it meets one of the two de-identification standards in 45 CFR 164.514(b). The Safe Harbor method requires removal of eighteen listed identifiers, including names, most geographic subdivisions smaller than a state, all elements of dates except the year for dates directly related to an individual, and various numbers and codes, together with no actual knowledge that the remaining information could identify the individual. Safe Harbor fits synthetic data awkwardly. A synthetic record may contain full dates and five-digit ZIP codes that were generated rather than copied, and the rule's text does not say whether generated values count. The cautious reading, which most privacy counsel adopt, is not to rely on Safe Harbor for data-driven synthetic data that retains such fields, or to coarsen them to Safe Harbor levels before release, which sacrifices much of the value. The Expert Determination method requires a person with appropriate statistical and scientific knowledge to determine, and document, that the risk is very small that the information could be used, alone or in combination with other reasonably available information, by an anticipated recipient to identify an individual. This is the natural route for synthetic data. It is also, like the European test, contextual: it depends on the anticipated recipient and the information reasonably available to them, and an expert's determination for a controlled research release does not carry over to public release. The membership and attribute inference evaluations described in Chapter 5 are precisely the evidence an expert should require. HIPAA is not the only US law. Health data held outside covered entities, by many consumer health apps for example, falls outside HIPAA, but may fall under state laws such as the California Consumer Privacy Act, which has its own definition of de-identified information with contractual and technical conditions, and Washington's My Health My Data Act. Research using federal funding or conducted under the Common Rule raises separate questions about whether synthetic data derived from identifiable private information requires review. Table 3 sets the three regimes side by side. Table 3. How three privacy regimes treat synthetic data derived from health records. Law Is generation regulated? Test for output to leave scope Key reference points EU GDPR Yes: processing needing Article 6 basis and Article 9 condition Anonymous by means reasonably likely to be used; no isolation, linkage or inference Recital 26; EDPB Guidelines 02/2026 (draft); EDPB Opinion 28/2024; EDPS v SRB (2025) UK GDPR Yes: same structure as EU Identifiability remote in context; motivated intruder test ICO anonymisation and pseudonymisation guidance (March 2025) US HIPAA Privacy Rule Permitted use to create de-identified data (164.502(d)) Safe Harbor (18 identifiers) or Expert Determination of very small risk (164.514(b)) HHS de-identification guidance; state laws for non-covered data A further regulatory development points in a revealing direction. The EU Artificial Intelligence Act allows providers of high-risk AI systems to process special categories of personal data, such as health or ethnicity, to detect and correct bias, but Article 10(5) as originally adopted made this conditional on bias detection not being achievable with other data, including synthetic or anonymised data. The amendments agreed in 2026 broadened the scope of that permission while keeping a strict necessity test. The law, in other words, treats synthetic data as a preferred, less intrusive alternative where it is adequate for the purpose, which is close to the position this chapter takes. It rewards demonstrated adequacy; it does not treat synthetic data as outside the law's concern. Building a programme that holds up An organisation that wants to share synthetic health data, and to defend that decision to a regulator, an ethics committee or its patients, can follow a sequence that reflects all of the above. Establish the legal basis for generation before generation. Treat synthesis as a processing purpose in its own right, document the lawful basis and special category condition, carry out a data protection impact assessment, and update privacy notices where needed. Keep the real data and the generator inside a secure environment. Train generators in a trusted research environment or equivalent, and never export the trained generator itself unless it has been separately assessed as anonymous under the reasoning of EDPB Opinion 28/2024. A shared generator lets recipients sample without limit and probe the model directly, which makes inference attacks easier than against a fixed synthetic sample. Tier the outputs by fidelity and risk. Many programmes offer a low-fidelity tier, preserving formats and simple marginals, suitable for software development and widely shareable, and a high-fidelity tier, preserving complex relationships, released only to approved users under contract, with conditions prohibiting re-identification attempts and linkage. This mirrors the contextual reasoning of both EDPS v SRB and the ICO: the same synthetic data can be adequately anonymous in one setting and not another. Prefer formal guarantees for wide release. For any release that will be public or very widely shared, use a differentially private generator with a stated epsilon and unit of privacy, and accept the utility cost, particularly for rare conditions, as the price of a release that does not depend on predicting what every future attacker can do. Evaluate as Chapter 5 describes, and give the evaluation to the person making the anonymity decision, whether a data protection officer under the GDPR or an expert under HIPAA's Expert Determination standard. Record the decision, its context and its review date. Provide a route back to the real data. Following the census model, offer a validation service in which researchers who find something in the synthetic data can have their analysis run on the real data inside the secure environment, with results checked before release. This protects patients, because nobody draws clinical conclusions from synthetic data alone, and it protects the science. Tell patients. Public trust in health data use is fragile, and synthetic data is novel enough that its use should be explained plainly in the organisation's public materials, including what it is, what it is used for, and what protections apply. None of this amounts to bypassing regulation. It amounts to using a technique that, applied with care, lets an organisation meet the law's requirements while sharing more than it otherwise could. That is a better outcome for patients and researchers than the shortcut, and it is the only one that survives scrutiny. CHAPTER 7 Credit Scoring and the Minority Class Credit risk modelling has a peculiar relationship with scarcity. Lenders hold enormous amounts of data: every application, every repayment, every missed payment, recorded over years. Yet the outcome they most need to predict, default, is usually rare. In a healthy economy most borrowers repay. For prime mortgage books, the share that defaults in a given year can be well under one per cent, and some portfolios, such as lending to large corporates or sovereigns, have so few defaults that the Basel banking framework treats them as a distinct problem; Katja Pluto and Dirk Tasche's 2005 paper on estimating default probabilities for "low default portfolios" was written precisely because standard methods break down when there is almost nothing to count. Fraud is rarer still. The widely used public dataset of card transactions made by European cardholders over two days in September 2013, released by researchers at the Université Libre de Bruxelles and Worldline, contains 492 frauds among 284,807 transactions, about 0.17 per cent. This is the second kind of scarcity introduced at the start of the book: not a shortage of data but a shortage of the class that matters. It is also the oldest application of synthetic data in machine learning, and the one where the gap between common practice and the evidence is widest. Many credit and fraud teams routinely oversample the minority class with SMOTE or a generative model, report improved sensitivity, and ship models whose probability estimates are badly wrong. In most fields that would be a technical flaw. In lending, where probabilities set prices, determine capital and drive loss provisions, and where decisions must be explained to regulators and applicants, it is a business and compliance problem. What imbalance does, and what it does not do It is worth being precise about the harm imbalance causes, because much of the remedy follows from the diagnosis. A classifier trained on imbalanced data is not, in general, a bad estimator of probabilities. Logistic regression fitted by maximum likelihood on data with a one per cent default rate will, if the model is well specified, estimate default probabilities that average about one per cent and are calibrated across the range. Gradient-boosted trees trained with a log-loss objective behave similarly. The problem that practitioners observe, a model that "never predicts the minority class", arises almost entirely from the decision rule applied to those probabilities. If the model's outputs are converted to labels with a threshold of 0.5, and very few applicants have a predicted default probability above 0.5, then very few will be labelled as likely defaulters, even if the model ranks applicants well. Accuracy, which rewards labelling everyone as a non-defaulter, makes the problem look worse still. The direct remedy for a decision-rule problem is to change the decision rule. If missing a defaulter costs the lender twenty times as much as wrongly declining a good applicant, the cost-minimising threshold is not 0.5 but roughly 1/21, and Charles Elkan's 2001 paper on cost-sensitive learning set out how to derive such thresholds and how to correct for changed class priors. Class weighting during training achieves a similar effect within the learning algorithm. Neither requires a single synthetic record. There are genuine cases where imbalance damages learning itself rather than just the decision rule. With very few minority examples, a flexible model may have too little signal to learn the minority class's structure, and may overfit the few examples it has. Some learning algorithms, particularly those with greedy splitting criteria or those trained with small minibatches in which minority examples rarely appear, can underweight the minority class in ways that weighting does not entirely fix. It is in these cases that synthetic minority examples might add something, through the regularisation and inductive bias mechanism described in Chapter 1. But they do not add information about defaulters that the real defaulters did not contain. From SMOTE to generative models Nitesh Chawla, Kevin Bowyer, Lawrence Hall and Philip Kegelmeyer published SMOTE, the Synthetic Minority Over-sampling Technique, in the Journal of Artificial Intelligence Research in 2002. Its mechanism is simple. For each minority example, find its nearest minority-class neighbours in feature space, pick one, and create a new example at a random point on the line segment between them. Repeat until the classes are as balanced as desired. Compared with duplicating minority examples, which encourages a model to memorise them, SMOTE spreads the synthetic examples through the region the minority class occupies, which can encourage broader decision regions. SMOTE has become one of the most cited techniques in machine learning and has spawned dozens of variants. Borderline-SMOTE, proposed by Hui Han and colleagues in 2005, generates new examples only near the class boundary. ADASYN, by Haibo He and colleagues in 2008, generates more examples where the minority class is hardest to learn. SMOTE-NC handles mixtures of numeric and categorical features. All share SMOTE's basic assumption that the space between two minority examples is also minority territory. In credit data that assumption is often false. Defaulters are not a compact cluster; they include young applicants with thin files, established borrowers hit by job loss, small businesses in a declining sector, and fraudsters posing as any of these. Interpolating between a young renter and a middle-aged homeowner who both defaulted produces an applicant who resembles neither and may sit squarely in a region dominated by good payers. Categorical features make it worse, because the midpoint between two occupations or two regions is not meaningful. Generative models were meant to fix this by learning the minority class's actual distribution rather than interpolating linearly. Ugo Fiore and colleagues, in a 2019 paper in Information Sciences, trained a GAN on the minority class of a credit card fraud dataset and used its samples to augment training, reporting improved sensitivity over the unaugmented baseline. Conditional tabular generators such as CTGAN, and diffusion-based ones such as TabDDPM, can generate defaulters conditioned on segment, product or other attributes, and their samples respect categorical structure far better than SMOTE's. Kotelnikov and colleagues reported that TabDDPM gave better downstream classifier performance than SMOTE on most of their benchmark datasets. These are genuine improvements in the realism of synthetic minority examples. The question that matters, however, is not whether the synthetic defaulters are realistic but whether training on them produces better credit decisions than the simpler alternatives. And here the evidence is sobering. In 2022 Ruben van den Goorbergh, Maarten van Smeden, Dirk Timmerman and Ben Van Calster published a study in the Journal of the American Medical Informatics Association with a blunt title: "The harm of class imbalance corrections for risk prediction models". Using Monte Carlo simulations and a case study on ovarian cancer diagnosis, they examined random undersampling, random oversampling and SMOTE with standard and penalised logistic regression. Every correction method led to poor calibration, with strong overestimation of the probability of belonging to the minority class, and none improved discrimination as measured by the area under the ROC curve. The corrections did improve sensitivity and specificity at the default threshold, but the same results were obtained simply by shifting the threshold. Their conclusion was that outcome imbalance is not a problem in itself and that correcting it may worsen model performance. The study concerns clinical risk prediction, but its logic transfers to credit without modification. A probability of default is a risk prediction. Oversampling the minority class during training changes the class prior the model learns from; a model trained on balanced data believes that half of all applicants default, and its probability estimates are inflated accordingly. That distortion is invisible if the model is evaluated only on ranking measures such as AUC or the Gini coefficient that credit teams commonly report, and on classification metrics at a tuned threshold. It becomes very visible when the probabilities are used as probabilities: to price a loan, to compute expected losses under IFRS 9 or CECL, or to estimate the probability of default inputs to regulatory capital under the internal ratings-based approach. A model whose probabilities have been inflated by oversampling will overstate expected losses, misprice risk and, if used for capital, produce numbers that a supervisor will reject. Machine learning benchmarks point the same way for stronger learners. Yotam Elor and Hadar Averbuch-Elor's 2022 study "To SMOTE, or not to SMOTE?" compared balancing methods across many imbalanced datasets and found that for strong classifiers such as gradient-boosted trees, with the decision threshold properly optimised, balancing generally did not improve performance; it helped mainly weaker learners. Gradient-boosted trees are the workhorse of modern credit scoring, so this matters. None of this means synthetic minority data never helps. It means the burden of proof lies with the synthetic data. The baseline to beat is a well-tuned model trained on the real imbalanced data, with class weights or threshold adjustment set to reflect the actual costs of errors, and with probabilities calibrated on real data at the real class ratio. If synthetic augmentation improves on that baseline, on real held-out data, in the metric that matters for the use, it has earned its place. Many claimed improvements in the literature and in practice are measured against weaker baselines. If synthetic augmentation is used, two disciplines are essential. The first is to do it inside cross-validation, never before. Oversampling the full dataset and then splitting it into training and test folds places synthetic examples derived from test-fold records into the training folds, so the model is effectively trained on interpolations of the cases it will be tested on. Miriam Santos and colleagues documented in 2018 how common this error is in the imbalanced learning literature and how much it inflates reported performance. The second is to recalibrate on real data at the real class ratio after training, using a held-out real calibration set and a method such as Platt scaling or isotonic regression, or to apply an explicit prior correction. A minimal correct pipeline in Python looks like this. from imblearn.pipeline import Pipeline from imblearn.over_sampling import SMOTENC from sklearn.calibration import CalibratedClassifierCV from sklearn.ensemble import HistGradientBoostingClassifier # Oversampling happens only inside each training fold pipe = Pipeline([ ("smote", SMOTENC(categorical_features=cat_idx, random_state=0)), ("gbm", HistGradientBoostingClassifier()), ]) # Calibrate on real, imbalanced held-out folds so probabilities # reflect the true default rate rather than the balanced one model = CalibratedClassifierCV(pipe, method="isotonic", cv=5) model.fit(X_train, y_train) The same structure applies with a generative model in place of SMOTE: the generator must be trained only on the training fold, and the final probabilities must be calibrated against real data that the generator never saw. Calibration should then be checked directly, with calibration curves and measures such as the Brier score, alongside discrimination. When a separate calibration set is not available, the distortion introduced by rebalancing can at least be corrected analytically, because it is a change in the class prior. A model trained on data in which the default rate has been raised from its true value to some artificial value produces odds of default that are inflated by the ratio of the artificial prior odds to the true prior odds. Dividing that ratio out restores the correct scale. A worked example shows the size of the effect. Suppose the true default rate is 1 per cent and the training data was rebalanced to 50 per cent. The prior odds are about 0.0101 in reality and 1.0 in training. A model trained on the rebalanced data that outputs a default probability of 0.30 for an applicant is expressing odds of 0.30/0.70, about 0.43. Multiplying by 0.0101 gives corrected odds of about 0.0043, a default probability of roughly 0.43 per cent. The uncorrected model has overstated this applicant's risk by a factor of about seventy. Marco Saerens and colleagues described this correction, and how to estimate new priors when they are unknown, in 2002. The correction is exact only when the oversampling has changed nothing but the class proportions; synthetic minority examples that differ in distribution from real ones introduce further distortion that only empirical calibration on real data can detect. The selection problem synthetic data cannot fix Credit data has a deeper scarcity than the rarity of defaults, and it illustrates the information limit of Chapter 1 more sharply than any other example in this book. Lenders observe repayment outcomes only for applicants they accepted. Applicants who were declined never received a loan, so whether they would have repaid is unknown. A model trained on accepted applicants learns the relationship between characteristics and default within the population the previous policy chose to lend to, and that population differs systematically from the full applicant population the new model will score. The credit risk literature calls the attempt to correct for this reject inference, and it has resisted easy solutions for decades. It is tempting to think that a generative model could fill the gap by producing synthetic outcomes for rejected applicants. It cannot, at least not from the lending data alone. A generator trained on accepted applicants with known outcomes learns the joint distribution of characteristics and outcomes among accepted applicants. Asked to generate outcomes for applicants with the characteristics of rejected ones, it extrapolates from that distribution into regions where it has little or no data, and its outputs encode the assumption that the relationship in the accepted population holds unchanged in the rejected population. That is precisely the assumption that reject inference exists to question. The synthetic outcomes look like data but are an assumption expressed in data form, and a model trained on them will inherit the assumption with an unwarranted appearance of empirical support. Genuine information about rejected applicants has to come from outside the generator: from controlled experiments in which a small random share of marginal applicants are accepted, from credit bureau data on how declined applicants performed on credit obtained elsewhere, or from explicit modelling assumptions stated as such. Synthetic data can then be used to explore the consequences of those assumptions. It cannot replace the evidence. Fairness, regulation and the uses that do make sense Imbalance in credit is not only about defaults. Applicant populations are also imbalanced across demographic groups, regions and product types, and models can perform worse for groups that are under-represented in the training data. It is tempting to oversample under-represented groups with synthetic applicants to fix this, and synthetic data vendors sometimes market "fairness" features that do exactly that. The same cautions apply with added force. Synthetic applicants for an under-represented group encode only what the generator learned from the few real ones, and a generator's errors tend to be largest for small groups; the Ganev, Oprisanu and De Cristofaro finding that differentially private generators distort minority subgroups is one example. Synthetic augmentation can make a model appear fairer on synthetic test data while changing little on real applicants. Any fairness claim must be tested on real data disaggregated by group. Regulation sharpens all of this. In the United States, the Equal Credit Opportunity Act and Regulation B prohibit discrimination on protected characteristics and require lenders to give applicants the specific principal reasons for adverse action; the Consumer Financial Protection Bureau's Circular 2022-03 stated that creditors using complex algorithms are not exempt from that requirement. Supervisory guidance on model risk management, the Federal Reserve and Office of the Comptroller of the Currency's SR 11-7 of 2011, expects banks to document the data used to develop models and to justify its relevance and quality. A credit model trained partly on synthetic defaulters must be able to explain why that data was appropriate, how it was generated, how it was validated against real outcomes, and why the model's probability estimates remain accurate. A model risk team will rightly ask why a weighted model on real data was not sufficient. In the European Union, the Artificial Intelligence Act lists AI systems used to evaluate the creditworthiness of natural persons or establish their credit score as high-risk, with an exception for fraud detection. Article 10 requires that training, validation and testing data be relevant, sufficiently representative and, to the best extent possible, free of errors and complete in view of the intended purpose, and that data be examined for possible biases. Following the Digital Omnibus amendments that entered into force in July 2026, the obligations for such stand-alone high-risk systems apply from 2 December 2027. Synthetic data used to train a credit model will fall within the data governance documentation that the Act requires: its origin, the generation process, the assumptions it encodes and its examination for bias. That is not a prohibition on synthetic data. It is a requirement to be able to say what it is and show that it is fit for purpose, which is the argument of this book expressed as law. Given all this, where does synthetic data earn its place in credit risk? Four uses stand out. The first is data access and collaboration. Banks often cannot share real applicant data with vendors, academic partners or even other internal teams in different jurisdictions. Synthetic credit data, evaluated for privacy as Chapter 5 describes, can let a vendor build and test a pipeline, or a research partner prototype a method, before any real data changes hands. Final models are trained and validated on real data inside the bank. The second is testing systems rather than training models. Synthetic applicants with specified characteristics are valuable for testing that decision systems apply policy rules correctly, that adverse action reasons are generated correctly, and that a model's behaviour does not change in unacceptable ways when a characteristic that should not matter is varied. This counterfactual testing uses synthetic data as a probe, where realism matters less than control. The third is stress and scenario analysis. Default experience in a benign period is a poor guide to default experience in a recession. Generators conditioned on macroeconomic variables, or simulation models of borrower behaviour, can produce plausible stressed portfolios for testing how models and processes respond. Such data is model-driven in the sense of Chapter 1, and its value depends on the quality of the economic assumptions, which must be stated. The fourth is genuinely data-starved learners. Where a team has evidence, from properly designed experiments on real held-out data, that synthetic minority examples improve a specific model over the best non-synthetic baseline, and the probabilities have been recalibrated on real data, synthetic augmentation is a legitimate tool. Such cases exist, particularly in fraud detection with neural sequence models and very few labelled fraud cases. They should be demonstrated rather than assumed. The broader lesson of this chapter is that class imbalance is often a decision problem disguised as a data problem. Synthetic data is a data solution. It should be reached for after the decision problem has been addressed directly, and judged by whether it improves real outcomes in the metrics that lending actually uses. CHAPTER 8 Computer Vision and the Long Tail of Rare Events On the night of 18 March 2018, a test vehicle operated by Uber's self-driving unit struck and killed Elaine Herzberg as she walked a bicycle across a four-lane road in Tempe, Arizona. The National Transportation Safety Board's investigation found that the vehicle's sensors had detected her about 5.6 seconds before impact, but that the perception system's classification of her alternated between vehicle, bicycle and "other", and that it never classified her as a pedestrian, because she was not at a crosswalk. The system, the Board found, did not include a consideration for jaywalking pedestrians. Each time the classification changed, the system discarded its prediction of her path. By the time it determined that a collision was imminent, about 1.2 seconds remained. The causes of the crash went well beyond perception, including the disabling of the vehicle's own emergency braking and the inattention of the safety driver. But the perception failure illustrates the third kind of scarcity with painful clarity. A person pushing a bicycle across an unlit multi-lane road outside a crossing is not a bizarre event; it happens every night somewhere. It was rare in the data that shaped the system's assumptions, and the system's designers had not imagined it as a case to handle. Machine perception systems fail disproportionately on situations like this: combinations of object, pose, lighting, weather, occlusion and context that are individually unremarkable and jointly rare. Collecting enough real examples of every such combination is impossible, because the combinations are too many and most of them are too rare. Synthetic data is the main response the industry has developed, and computer vision is where it has the longest and most successful track record. This chapter traces how the approaches developed, why the gap between simulated and real images has been so persistent, and how synthetic edge cases should be used for training and for testing. From game engines to randomised worlds The modern history begins with the realisation that video games had already solved much of the rendering problem. In 2016 Stephan Richter, Vibhav Vineet, Stefan Roth and Vladlen Koltun published "Playing for Data", in which they extracted about 25,000 images from the game Grand Theft Auto V along with pixel-level semantic labels, obtained by intercepting the communication between the game and the graphics hardware. Labelling a real street scene at pixel level takes a human annotator a long time; their method produced labels in seconds per image. They reported that semantic segmentation models trained on the game data plus just one third of the real CamVid training set outperformed models trained on the complete CamVid training set. The same year German Ros and colleagues released SYNTHIA, a dataset rendered from a purpose-built virtual city, and Adrien Gaidon and colleagues released Virtual KITTI, which recreated real driving sequences from the KITTI benchmark in a game engine so that models could be compared on real and synthetic versions of the same scenes. In 2017 Alexey Dosovitskiy and colleagues released CARLA, an open-source driving simulator that became a standard research platform. These datasets showed two things at once. Synthetic images could substantially reduce the amount of real labelled data needed. And models trained on synthetic images alone performed noticeably worse on real images than models trained on real ones, because rendered images differ from camera images in textures, lighting, noise, lens effects and countless small statistical regularities that a neural network can learn to depend on. This is the domain gap, or in robotics the sim-to-real gap, and much of the subsequent history is the history of attempts to close or circumvent it. There were two broad strategies. The first was to make synthetic images look more real. Better rendering helped. So did learned image-to-image translation: Jun-Yan Zhu and colleagues' CycleGAN of 2017 could learn to translate images from one domain to another without paired examples, and Judy Hoffman and colleagues' CyCADA of 2018 applied such translation to make synthetic driving images resemble real ones while preserving their labels, improving the transfer of segmentation models. The second strategy, already introduced in Chapter 1, was to stop trying. Josh Tobin and colleagues' domain randomisation, published in 2017, rendered training images with random textures, colours, lighting and camera positions, so varied that the real world became, from the model's point of view, just one more variation. Their object detector, trained only on simulated images, located real objects accurately enough for a robot to grasp them. Jonathan Tremblay and colleagues at NVIDIA applied the idea to car detection in 2018, placing car models in scenes with random distractor objects, textures and lighting, and found that detectors trained on this deliberately unrealistic data were competitive with those trained on more photorealistic synthetic data, and improved further when fine-tuned on real images. Aayush Prakash and colleagues' structured domain randomisation of 2019 added context, placing objects according to the plausible structure of scenes, so that cars appeared on roads rather than floating in space. The lesson of domain randomisation is subtle and important for edge cases. A model trained on a narrow, realistic synthetic distribution learns the regularities of that distribution, including the renderer's artefacts. A model trained on a broad, randomised distribution learns only the features that survive all the variation, which are more likely to be the features that matter in the real world. For rare conditions, this suggests that the goal of synthetic data is not to reproduce the frequency of real conditions but to cover the space of conditions broadly, including combinations the real world rarely produces. Simulators, world models and generated images Autonomous driving companies made simulation central to their development. In 2018 MIT Technology Review reported that Waymo's software was driving ten million miles a day in simulation and had accumulated more than six billion virtual miles, and described how Waymo engineers used "fuzzing", borrowed from computer security, to rerun real scenarios with random variations in the positions, speeds and behaviours of other road users to see whether any variation caused a failure. This is synthetic data in a form slightly different from the rest of this book: not a dataset handed to a training algorithm but a scenario generator used to probe a system. It addresses edge cases by taking real situations and pushing them towards their boundaries. Hand-built simulators have an obvious limitation. Every object, material, weather effect and behaviour must be modelled by someone, and the simulator can only produce what its builders thought to include. Learned world models aim to remove that bottleneck by learning what the world looks like from enormous amounts of real video. NVIDIA launched its Cosmos platform of world foundation models in January 2025, trained, the company said, on 20 million hours of video, and intended for generating synthetic training data for robots and vehicles. In February 2026 Waymo announced the Waymo World Model, built on Google DeepMind's Genie 3 video world model and adapted to generate camera and lidar data. Waymo described using it to simulate exceedingly rare events, from tornadoes to an encounter with an elephant, and stated that its driver had by then travelled nearly 200 million fully autonomous miles in the real world and billions in simulation. The model can be steered with language prompts and driving inputs, so engineers can ask for variations of a scene that the fleet has never recorded. World models change the economics of edge cases. A scenario that once required an artist to model a new object and an engineer to script its behaviour can now be described in a sentence. They also change the validation problem. A hand-built simulator's errors are at least systematic and inspectable: its rain looks the way the renderer makes rain look. A learned world model's errors are statistical and harder to anticipate: it may render a plausible elephant whose motion is subtly wrong, or produce lidar returns from a tornado that no real sensor would produce. Because the world model is itself a data-driven generator, it inherits every limitation discussed in Chapters 1 to 3, including the information limit. A world model has seen many elephants in videos and many roads, so combining them is a reasonable interpolation; whether its rendering of an elephant at night in rain, through a particular lidar's scan pattern, is faithful enough to train or test a safety-critical system is an empirical question. Outside driving, text-to-image diffusion models have become a general-purpose source of synthetic training images. Ruifei He and colleagues asked in an ICLR 2023 paper whether synthetic data from generative models was ready for image recognition, and found it helpful particularly in zero-shot and few-shot settings, where real data for a class is absent or minimal. Mert Bülent Sarıyıldız and colleagues showed at CVPR in 2023 that models trained on synthetic clones of ImageNet, generated from class names with Stable Diffusion, learned representations that transferred well, though they trailed real-data training on ImageNet itself. Lijie Fan and colleagues' 2024 study of scaling laws for synthetic images found that synthetic images showed a scaling trend similar to, but slightly less effective than, real images for training CLIP-style models, that off-the-shelf text-to-image models' inability to generate certain concepts badly hurt supervised classifiers, and that synthetic data was most valuable when real images were limited, when test data differed substantially from training data, or when synthetic images were combined with real ones. That pattern, most valuable where real data is scarce and where the test domain is shifted, is exactly the profile of rare edge cases. It is also matched by the most serious practical problem with generated images: label fidelity. A rendered image from a simulator comes with perfect labels, since the simulator knows where every object is, down to the pixel. A diffusion model asked for "a cracked weld seam under low light" produces an image that may or may not contain a crack, in a location not explicitly known, of a type not guaranteed to match the request. At high guidance strengths the model adheres more closely to the prompt but produces less diverse images, and, as Chapter 3 noted, is more likely to reproduce training examples. Teams using diffusion models to generate edge cases typically need a verification step, such as a separate model or human review that confirms the requested content is present and correctly labelled, and conditioning on spatial inputs such as segmentation maps or bounding boxes rather than text alone. Beyond driving: inspection and medicine Driving dominates public discussion of synthetic edge cases, but the same problem appears wherever visual systems must catch rare events, and two other fields show the pattern clearly. In industrial visual inspection, the events of interest are defects, and a well-run production line produces few of them. A model inspecting castings, welds, circuit boards or pharmaceutical packaging may have thousands of images of good parts for every image of a particular defect type, and some defect types may have been seen only a handful of times. Inspection teams have used every approach described in this chapter: rendering defects onto computer-aided design models of the part, cutting and pasting real defects onto images of good parts with blending, and training generative models on the few real defects to produce more. They have also used the opposite strategy, anomaly detection, which trains only on good parts and flags anything that departs from them; this avoids the need for defect examples altogether, at the cost of flagging harmless variation. The experience of inspection teams reinforces the lessons of driving. Synthetic defects help most when their appearance is driven by an understanding of how the defect physically arises, such as how a crack propagates through a material or how a solder joint fails, and least when a generator is asked to invent defects from a few examples, in which case it tends to produce variations on those examples rather than the defects that will actually occur next. In medical imaging, rare presentations of disease create the same long tail. A classifier for skin lesions trained mostly on common lesions in lighter skin, or a fracture detector trained mostly on adult radiographs, will perform worse on the uncommon cases. Generative augmentation has been proposed for both, and the Frid-Adar study of Chapter 2 showed it can help. But medicine adds two complications to the general picture. The first is that, as Chapter 3 described, a generator trained on a handful of real examples of a rare presentation is at high risk of memorising them, which matters for privacy if the synthetic images leave the institution. The second is that the features distinguishing a rare presentation are often subtle, and a generator that produces a convincing overall image may not reproduce them faithfully. A radiologist's review of synthetic rare-case images is therefore not a formality but a core part of validation, and evaluation of any model trained on them must be on real cases of the rare presentation, assembled where necessary across several institutions. Using synthetic edge cases responsibly Choosing a source is the first decision. Table 4 compares the main sources of edge-case images on the attributes that typically decide among them. Table 4. Sources of training and test images for rare edge cases compared. Source Control over scenario Realism of sensor data Label quality Main failure mode Targeted real collection Low Real Requires human annotation Too slow and costly for very rare combinations Hand-built simulation High Limited by renderer; domain gap Exact Covers only what designers modelled Domain randomisation High Deliberately unrealistic Exact Needs real fine-tuning for best results Image-to-image translation of synthetic scenes High Improved, with translation artefacts Mostly preserved Translation can alter or remove labelled content Text- or layout-conditioned diffusion Moderate High visually Uncertain without verification Missing or wrong content; memorised examples Learned world model Moderate to high High, but errors hard to predict Varies by system Plausible but physically or sensor-inaccurate output The common thread in this history is that synthetic data has proved most valuable in computer vision as a complement to real data and most dangerous as a substitute for real evaluation. Several principles follow. Use synthetic data to train broadly and real data to validate narrowly. Synthetic edge cases can expand a model's training distribution into regions the real data does not reach. But the evidence that the model now handles those regions must come, wherever possible, from real examples of them, even if only a few. A small curated real test set of rare conditions, collected deliberately over time, is worth more for validation than any volume of synthetic examples, because it is the only evidence that is not contaminated by the generator's assumptions. Measure the domain gap directly. Train the same model on real data and on synthetic data for the same task and compare their performance on real test data. Use the feature-space precision and recall measures of Chapter 4, with domain-appropriate feature extractors, to check whether synthetic images cover the real variation. Track how performance on real edge cases changes as synthetic data is added, since the relationship is not always monotonic. Separate training scenarios from test scenarios. If the same generator, with the same assumptions, produces both the training edge cases and the test edge cases, good test results may show only that the model has learned the generator's version of the world. Test scenarios should come from a different source wherever possible: a different simulator, real data, or at least scenarios specified independently of those used in training. Treat scenario coverage as a safety argument, not a statistic. Safety standards for automated driving have moved in this direction. ISO 21448, on the safety of the intended functionality, is concerned with hazards that arise not from component failures but from functional insufficiencies, such as a perception system's inability to handle a situation, and asks developers to identify and reduce the space of unknown hazardous scenarios. UL 4600, a safety standard for autonomous products, similarly requires a structured argument that the system is acceptably safe, including evidence about how edge cases have been identified and handled. Synthetic data contributes to such arguments, but the argument has to explain why the synthetic scenarios cover the hazards that matter, which in turn requires a systematic process for identifying hazards from real-world experience, incident data and expert analysis. Simulation can only test the edge cases somebody has thought of or that a world model happens to produce; it cannot by itself discover the jaywalking pedestrian that nobody considered. That last point returns to the Tempe crash. The failure there was not that the system lacked synthetic images of pedestrians outside crosswalks. It was that its design did not consider them at all. Synthetic data multiplies the examples of situations that have been identified. It does not identify them. The organisations that use synthetic edge cases best treat the generator as the second step in a process whose first step is the disciplined, continuing search for what the system has not yet been shown. CHAPTER 9 Synthetic Data in the Loop: Collapse, Verification and Governance The previous chapters treated synthetic data as a one-off product: train a generator, sample a dataset, evaluate it, use it. In practice synthetic data increasingly lives inside larger systems. Models are trained on data produced by other models. Generated images and text flow onto the web and into the next round of training data. Organisations regenerate synthetic datasets as the real data changes, release new versions to new partners, and build pipelines in which synthetic data is created, filtered and consumed automatically. These loops create risks and opportunities that a single dataset does not, and they make governance, the routine discipline of knowing what data exists, where it came from and what it can be trusted for, as important as any generator. When models learn from models In July 2024 Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson and Yarin Gal published a paper in Nature titled "AI models collapse when trained on recursively generated data". They examined what happens when each generation of a model is trained on data produced by the previous generation. Across language models, variational autoencoders and Gaussian mixture models, the result was the same in kind: over successive generations the models progressively lost information about the tails of the original distribution, the rare events and unusual examples, and eventually converged towards a narrow distribution bearing little resemblance to the original. They called the process model collapse. Sina Alemohammad and colleagues reached compatible conclusions for image generators in a paper presented at ICLR in 2024 under the title "Self-Consuming Generative Models Go MAD", finding that without enough fresh real data in each generation, either the quality or the diversity of generated images progressively declined. The mechanism is a compounding of the errors discussed in earlier chapters. Every generator approximates its training distribution imperfectly, and the errors are not random: generators tend to under-represent low-probability regions, because those regions contribute little to the training objective and because finite samples from them are sparse. A second generator trained on the first's output sees an already thinned tail and thins it further. Repeated, this erodes the tails entirely. For this book's subject the implication is pointed. The rare cases are what disappear first, and the rare cases are why most synthetic data is generated. The finding has been widely discussed as a threat to the future of large language models, whose training data is drawn from a web that increasingly contains generated text. But it applies with equal force to any organisation that builds synthetic data on synthetic data: a vendor whose generator was itself trained partly on synthetic samples; a team that augments a dataset with synthetic minority cases, trains a model, uses the model to label more data, and trains a generator on the result; a simulation pipeline whose scenarios are produced by a world model trained on logs that already include simulated scenarios. Collapse is not inevitable. Matthias Gerstgrasser and colleagues showed in 2024 that when synthetic data is accumulated alongside the original real data, rather than replacing it, the test error of successive models stays bounded rather than growing without limit. The practical rule is straightforward: keep the real data, keep it in the training mix, and never let a pipeline discard real examples in favour of generated ones. Track provenance so that it is always possible to tell which records are real and which are generated, and at which generation. Verification as the source of value There is an apparent paradox here. If training on generated data causes degradation, why has synthetic data become central to training today's most capable language models? Since Suriya Gunasekar and colleagues at Microsoft trained the small code model phi-1 largely on synthetic "textbook quality" data in 2023, developers have used model-generated text extensively for instruction tuning, reasoning and specialised skills, with clear gains. The resolution lies in what distinguishes useful synthetic data from collapse-inducing synthetic data, and it applies far beyond language models. Useful synthetic data adds something that its generator alone does not contain. In language model training the additions are typically a stronger teacher model, so that a smaller model learns from a better one; external knowledge supplied in prompts, such as documents or curricula; and, most importantly, verification. Generated code can be executed against tests. Generated mathematical solutions can be checked against known answers or formal proof checkers. Generated answers can be filtered by other models or by human reviewers. The synthetic data that survives verification carries information from the verifier, not just from the generator, and this is what prevents the narrowing that recursive training otherwise produces. The same logic runs through every successful use in this book. The census's synthetic microdata were paired with validation against the confidential file. Domain-randomised training data for robots succeeded because the robot's real-world performance verified it. A diffusion-generated defect image is useful only after something confirms the defect is present and correctly labelled. A synthetic credit dataset helps only if a model trained on it is shown to perform better on real loans. In each case the synthetic data's value comes from a check against something outside the generator, and the absence of such a check is the warning sign. This suggests a design principle for synthetic data pipelines: every synthetic dataset should be connected to at least one verifier that draws on information the generator did not have. For training data, the verifier is evaluation on real held-out data. For edge cases, it is real examples of the edge case, physical constraints, or independent review. For data shared under privacy constraints, it is validation of key findings on the real data. Where no such verifier exists, the synthetic data should be treated as a hypothesis, not as evidence. The rise of large language models has made one shortcut especially tempting, and it deserves a specific warning. A team wanting synthetic clinical notes, customer complaints or loan application narratives can now prompt a general-purpose model with a few real examples and ask for thousands more "like these". The output is fluent and varied, and there is no generator to train. But the approach combines several of the risks described in this book. The real examples placed in the prompt are personal data disclosed to whoever operates the model, which is processing that needs its own legal basis and, if the model is a third-party service, a suitable contract. Language models reproduce their inputs readily, so a synthetic note produced from a handful of real notes may contain verbatim phrases, names or distinctive details from them; Nicholas Carlini and colleagues showed in 2021 that large language models could be made to emit memorised training sequences, including personal information, and in-context examples are even more exposed than training data. And the information limit applies with full force: thousands of synthetic notes generated from five real ones carry the information of five notes and the language model's general prior, which may be fluent and wrong about the specific clinical setting. Used with a model deployed inside the organisation's own controls, with real examples screened and outputs audited for copying and verified by domain experts, prompted generation can be a useful source of test and training text. Used casually, it is an unrecorded disclosure that produces data of unknown quality. A brief illustration shows how verification and governance fit together in practice. Imagine a hospital analytics team asked to give a university partner data for developing early-warning models for patient deterioration. The team states the claim: the synthetic data will support method development, and any model the partner proposes will be retrained and validated by the hospital on real data inside its secure environment. That claim determines the verifier, which is the hospital's own real-data validation, and the privacy standard, which is a contextual anonymity assessment for a named recipient under contract rather than for the public. It determines the evaluation priorities: ranking preservation across candidate model types matters more than exact reproduction of every marginal. It determines the documentation, which records the claim, the evaluation and the conditions of use, and which tells the partner plainly that results obtained on the synthetic data are provisional. And it determines what happens when the partner later asks to publish the synthetic data alongside a paper: that is a different claim, with a public recipient, and it triggers a new privacy assessment, very likely with a differentially private generator, rather than an informal yes. Nothing in that sequence is technically difficult. What makes it work is that each decision follows from a stated claim, and each claim is tied to something outside the generator that can check it. Governing synthetic data as a product The most common failures in synthetic data programmes are not technical but organisational. A synthetic dataset is generated for one purpose, evaluated for that purpose, and then reused for another. A file labelled "synthetic" is assumed to be safe to share and ends up somewhere its privacy evaluation never considered. A generator is retrained on new data without re-running the evaluation. Synthetic records are mixed into a training set and their provenance is lost, so that years later nobody can say how much of a production model's training data was real. These failures have the same remedy: treat each synthetic dataset as a data product with documentation that travels with it. Timnit Gebru and colleagues' "Datasheets for Datasets", published in Communications of the ACM in 2021, proposed that every dataset be accompanied by a document recording its motivation, composition, collection process, preprocessing, intended uses, distribution and maintenance. For synthetic data, a datasheet needs additional fields that follow directly from the earlier chapters. It should state: • The intended use and the claim the synthetic data is meant to support, as defined in Chapter 1. • The source data, its time period, and the legal basis on which it was processed to train the generator. • The generator family, version, configuration and any constraints enforced, so that the dataset can be reproduced or regenerated. • The privacy mechanism, if any, with the differential privacy parameters and unit of privacy where applicable. • The fidelity, utility and privacy evaluation results, disaggregated by the subgroups and rare classes that matter. • The known limitations, including relationships the evaluation found to be poorly preserved and uses for which the data is not suitable. • The permitted recipients and conditions of use, consistent with the contextual anonymity assessment of Chapter 6. • A review date, after which the evaluation should be repeated in light of new attack techniques or changes in the real data. Some of these fields matter more than they first appear. The review date reflects the fact that privacy assessments decay: the ICO's guidance and the EDPB's draft guidelines both stress that anonymisation decisions must be revisited as technology develops, and the attacks of Chapter 5 have improved markedly within a few years. The source data period matters because synthetic data inherits the world as it was when the real data was collected; a synthetic credit dataset generated from records during a period of low interest rates will not reflect borrower behaviour after rates rise, and a synthetic clinical dataset generated before a change in treatment guidelines will encode the old practice. Synthetic data goes stale exactly as real data does, and needs regenerating and re-evaluating on a schedule. Provenance also needs to survive at the level of individual records and images, not only datasets. A column or metadata flag marking each record as synthetic, with the generator version, costs almost nothing and prevents the most damaging confusions. For images and video, emerging content provenance standards such as those of the Coalition for Content Provenance and Authenticity offer a way to attach that information in a form that travels with the file. Repeated releases raise a privacy problem of their own. An organisation that publishes a synthetic version of its data each quarter, regenerated from the latest real records, is making a series of releases, and an adversary can compare them. If a person's record enters the real data between two releases, the difference between the two synthetic datasets may carry a trace of that record, much as the difference between two aggregate statistics computed with and without one person can reveal that person's value. Differential privacy accounts for this explicitly: the privacy loss of successive releases from overlapping data adds up, and a programme that releases repeatedly must budget for the total, not just for each release in isolation. Programmes that rely on empirical privacy evaluation should evaluate the series as well as each release, for example by running membership inference attacks that use two consecutive releases together. The same reasoning applies when several synthetic datasets with different column subsets are generated from one real source for different recipients who might combine them. A different kind of loop affects evaluation. As synthetic data becomes common, it finds its way into benchmark and test sets, sometimes knowingly and sometimes because a dataset assembled from many sources includes generated material without saying so. A model evaluated on a test set that contains synthetic examples produced by a generator similar to the one used to create its training data will look better than it is, for the reason Chapter 8 gave for separating training and test scenarios. Test sets used to make decisions about real-world deployment should be checked for synthetic content and should, as far as possible, consist of verified real data. Provenance marking at the point of generation is what makes such checks possible later. When synthetic data is bought rather than built, the same questions become procurement questions. A buyer should ask a vendor what claim its synthetic data has been evaluated to support; whether the evaluation used real held-out data the generator never saw; which privacy attacks were run, against which records, with what results; whether any formal privacy guarantee applies, with what parameters and unit of privacy; how rare classes and small subgroups were evaluated; and whether the vendor's own generators were trained on any synthetic data. A vendor that answers with distance-to-closest-record statistics and a correlation heatmap has not answered. The market has matured, through consolidation such as SAS's acquisition of Hazy in November 2024 and NVIDIA's acquisition of Gretel in March 2025, and through open-source tools such as the Synthetic Data Vault, synthpop and SynthCity, a benchmarking library released by Zhaozhi Qian, Bogdan-Constantin Cebere and Mihaela van der Schaar in 2023. Buyers can now reasonably expect evaluation evidence, not just samples. Who decides A final governance question is who has authority to approve the use of synthetic data, and the answer should follow the claim. If synthetic data will be released outside the organisation, the privacy claim is decisive, and the decision belongs with whoever is accountable for data protection, informed by the evaluation and, in the United States, by the expert making any HIPAA determination. If synthetic data will train a model used in decisions about people, such as credit or clinical decisions, the utility and fairness claims are decisive, and the decision belongs within model risk management or clinical governance, with the evidence that the synthetic data improved performance on real cases. If synthetic data will test a safety-critical system, the coverage claim is decisive, and it belongs within the safety case. In none of these cases should the decision rest with the team that built the generator alone. That is not a criticism of generator builders; it is the same separation of development and validation that model risk management has required of statistical models for decades. Synthetic data is a model output, and it deserves the same independent challenge. Organisations that build this governance find that it pays off beyond compliance. Documented synthetic datasets can be reused with confidence for the purposes they were evaluated for. Regulators and partners accept them more readily. Engineers spend less time rediscovering what earlier datasets could and could not do. And the habit of stating, for every synthetic dataset, the claim it supports and the evidence behind it turns out to be the single most effective protection against the failures described throughout this book. Conclusion The chapters of this book have examined three kinds of scarcity and the generators built to overcome them, and a pattern runs through all of them that is only visible once they are set side by side. In every domain, the cases that synthetic data is most often asked to supply are the cases about which it is hardest to be sure. Consider where each difficulty concentrated. Generative adversarial networks tend to drop rare modes, so the minority class is the first casualty of mode collapse. Diffusion models capture rare modes better but memorise the isolated and the duplicated, so the unusual training example is the one most likely to be copied. Membership and attribute inference attacks succeed most against outliers, so the people whose records matter most to a rare-class generator are the people most exposed by it. Differential privacy protects those people by adding noise that drowns their signal first, so the formal remedy for the privacy problem degrades the rare cases most. Real test data is thinnest for exactly the rare classes and edge cases that synthetic data was generated to address, so the evidence for whether it worked is weakest where the claim is strongest. And model collapse, when synthetic data feeds further generation, erodes the tails of the distribution before anything else. Call this the rare-case paradox. Synthetic data is in greatest demand at the edges of the distribution, and the edges are where every property that makes synthetic data trustworthy, coverage, fidelity, privacy and verifiability, is hardest to achieve and hardest to measure. This is not a reason to abandon synthetic data. It is a reason to change where effort and money go. The first consequence concerns what is scarce. Once a generator is trained, synthetic samples cost almost nothing; a team can produce a million synthetic defaulters or ten million simulated miles in an afternoon. What remains expensive is real, verified evidence about the rare cases: a curated set of genuine edge-case images, a validation cohort of patients with the uncommon presentation, a sample of declined credit applicants whose outcomes were actually observed. The binding constraint on synthetic data programmes is not generation capacity but verification capacity. Organisations that understand this invest deliberately in small, carefully assembled real datasets of rare cases, held back from all training and from all generators, and they treat those datasets as among their most valuable data assets. Organisations that do not understand it produce ever larger volumes of synthetic data whose value nobody can establish. The second consequence concerns how synthetic data is described. The question "is this synthetic data good?" has no answer. The question "what claim does this synthetic dataset license, and on what evidence?" does. A synthetic dataset may license the claim that a software pipeline handles realistic records correctly, or that a planned regression gives the same answer as on the real data, or that a model trained with it performs better on real cases of a rare class, or that no individual in the training data is exposed beyond a stated level of risk to a stated kind of recipient. These are different claims, they require different evidence, and a dataset that supports one may not support another. Every synthetic dataset should carry its claim with it, in the documentation described in Chapter 9, and every use of it should be checked against that claim. Most of the failures described in this book, from the oversampled credit model with inflated probabilities to the "anonymous" release that leaked membership, came from using synthetic data for a claim it had never been tested against. The third consequence concerns the law, and it corrects the most persistent misconception in the field. Synthetic data does not bypass privacy regulation, and the regulators have now said so in terms. The European Data Protection Board's draft anonymisation guidelines treat synthetic datasets and AI models as not automatically anonymous and treat anonymisation itself as processing; HIPAA's de-identification standards apply to synthetic data like any other derived data; the Court of Justice's judgment in EDPS v SRB makes anonymity depend on the recipient and the context. Read together, these positions are not hostile to synthetic data. They describe a route by which carefully generated and rigorously evaluated synthetic data can be shown to be anonymous for particular recipients under particular conditions, and so be shared in ways the real data cannot. That route runs through the evaluation methods of Chapters 4 and 5. The law, in effect, asks for the same evidence that good engineering practice requires anyway. Several questions remain genuinely open, and a reader should watch them. The EDPB's anonymisation guidelines were still in consultation at the time of writing, and their final text, together with how national authorities apply the contextual reasoning of EDPS v SRB to synthetic data, will shape what European organisations can share. Differential privacy works well for moderate-dimensional tabular data, but whether it can protect high-resolution medical images or long patient histories without destroying their usefulness for rare conditions is not settled. Learned world models have made it cheap to generate edge cases for perception systems, but how to establish that a world model's rendering of a rare event is faithful enough to count as evidence in a safety case is a problem the safety standards have only begun to address. And the extent to which model collapse affects systems trained on web-scale data that increasingly contains generated content, as opposed to controlled experiments, is still being measured. What should a reader do differently? A practitioner should begin every synthetic data project by writing down the claim it must support, lock away real test data before any generator is trained, beat the simplest non-synthetic baseline before reaching for a generator, evaluate fidelity, utility and privacy disaggregated by the rare classes and subgroups that motivated the work, and keep real data in every training mix. Someone commissioning or buying synthetic data should ask for evidence against the specific claim, refuse similarity statistics as proof of privacy, and fund the collection of real rare-case validation data as part of the same budget. A privacy officer or regulator should judge synthetic data by the same tests as any other attempt at anonymisation, while recognising that, properly done, it can make more data safely available than older techniques. And everyone involved should resist the temptation to let the realism of generated samples stand in for evidence, because realism is the one property that a generator can deliver without delivering anything else. Synthetic data is one of the most useful tools available for working around data scarcity. Used with discipline, it opens locked data, rebalances training, and lets machines practise on situations the world has not yet produced. Its value is set not by the generator that produced it but by what can be shown about it, and the scarce resource, in the end, is not data but proof. Hashtags: #SyntheticDataGeneration #SyntheticData #DataScarcity #MachineLearning #ArtificialIntelligence #GenerativeAI #GenerativeAdversarialNetworks #GANs #DiffusionModels #TabularDataGeneration #CTGAN #TabDDPM #DataAugmentation #ClassImbalance #MinorityClassOversampling #SMOTE #PrivacyPreservingAI #DifferentialPrivacy #DataAnonymization #MembershipInferenceAttacks #SyntheticHealthcareData #MedicalDataPrivacy #CreditRiskModeling #ComputerVision #RareEventSimulation #DomainRandomization #SimToRealTransfer #DataFidelity #SyntheticDataUtility #ModelCollapse

Download the Book (PDF): Introduction Every organization that adopts GraphQL at any scale eventually has the same meeting. Someone from the mobile team arrives with a list of fields they need for a new screen. Someone from the platform team explains that the GraphQL server is owned by a group of four engineers who are already three sprints behind. Someone from the payments domain points out that the resolvers for invoices were written by a person who left eighteen months ago and nobody wants to touch them. Someone else asks, quietly, why the company moved to microservices if every change still has to squeeze through one repository, one deploy pipeline, and one review queue. That meeting is the subject of this booklet. GraphQL was designed to give clients a single, typed, self-describing entry point to an organization's data. It succeeds at that so well that the entry point becomes the most contended piece of software in the company. The schema grows to thousands of types. The server that implements it accumulates every domain's business logic, or at least every domain's glue code. Its release cadence becomes the slowest release cadence of any team that depends on it. What began as an elegant contract between frontend and backend turns into a monolith sitting in front of a fleet of services that were supposedly independent. GraphQL federation is the architectural answer that the industry converged on for this problem. Instead of one server implementing one schema, each domain team runs its own GraphQL service, called a subgraph, that defines and implements the part of the schema it owns. A composition step merges those subgraph schemas into a single supergraph schema and verifies that they fit together. A router sits at the edge, accepts client queries against the supergraph, works out which subgraphs can answer which parts of each query, calls them, and stitches the results into one response. Clients still see one endpoint and one schema. Behind it, ownership is distributed. What this booklet argues The controlling claim of this booklet is simple to state and harder to practice: federation is an organizational tool before it is a technical one. The directives, the router, the query planner, and the composition algorithm are all real engineering and all matter. But the reason federation exists is to let the shape of the API follow the shape of the organization, so that the team that owns a business capability also owns its slice of the graph, can change it on its own schedule, and is held to a contract that machines verify rather than meetings negotiate. When federation works, it works because three things are true. Schema boundaries map onto genuine domain ownership, so that most changes touch one subgraph and one team. Composition runs continuously as a contract check in every subgraph's delivery pipeline, so that breakage is caught before deploy rather than discovered in production. And the router is treated as critical infrastructure with its own performance budget, security posture, and operational discipline, not as a thin proxy that someone set up once. When federation fails, it usually fails because one of those conditions does not hold. Teams split the schema along technical lines instead of business lines and end up with every query touching five subgraphs. Composition happens only in a central team's environment and becomes the new bottleneck. Or the router is treated as plumbing until a query plan that fans out to forty subgraph calls takes down a checkout flow on the busiest day of the year. A federated graph built without these disciplines is a distributed monolith with a router in front of it: all the coupling of the original system, plus network hops. Who this is for This booklet is written for engineers, architects, and engineering managers who already understand the basics of GraphQL, meaning types, queries, mutations, resolvers, and the idea of a schema as a contract, but who have not yet built or operated a federated graph. It assumes familiarity with microservices in general, with continuous integration, and with the usual vocabulary of distributed systems such as latency, fan-out, and idempotency. It does not assume any particular language or vendor. Where configuration is shown, it is shown because the specific syntax teaches something about the underlying idea. Readers who are evaluating whether to adopt federation will find the early chapters most useful, especially the account of why monolithic API layers break down and the description of the supergraph model. Readers who have already decided and are now designing their first subgraphs should spend time on the chapters on entities, ownership, and composition. Readers operating a federated graph in production will get the most from the chapters on the router, security, and migration. The final chapter, on choosing a stack, is for anyone who has to sign a contract or defend a technology choice. The ground this covers The term "federation" in GraphQL has, for most of its history, meant Apollo Federation, the specification and tooling that Apollo introduced in 2019 and substantially redesigned as Federation 2 in April 2022. That remains the dominant dialect and the one this booklet uses for most examples, because it is what the large majority of production federated graphs speak. As of late 2026, Apollo's Federation specification has reached version 2.15, released in July 2026 as a long-term support release, and Apollo's router is in its second major version, written in Rust, with the older version 1 having reached end of support on March 31, 2026. But the ecosystem is no longer one vendor. WunderGraph's Cosmo offers an Apache 2.0 licensed router written in Go with a full control plane. The Guild's Hive offers a schema registry, a JavaScript gateway, and since September 2025 a Rust router, all under the MIT license. Grafbase and ChilliCream's Fusion offer their own gateways. And within the GraphQL Foundation, a Composite Schemas Working Group formed in 2024 has been drafting a vendor-neutral specification for exactly this problem, borrowing the strongest ideas from existing federation approaches. The licensing terms of the different tools differ in ways that matter: Apollo's router and its Federation 2 composition libraries are under the Elastic License v2, which permits internal use but forbids offering the software as a managed service, while several competitors are under permissive open-source licenses. These differences shape real decisions, and the last chapter treats them directly. The chapters run in an order that follows the life of a federated graph. Chapter 1 explains the gateway bottleneck: why a single GraphQL server or a hand-built aggregation layer stops scaling, and what earlier attempts such as backends-for-frontends and schema stitching taught the industry. Chapter 2 introduces the supergraph model in full: subgraphs, the supergraph schema, composition, and the router, and follows a single query from client to response through query planning and execution. Chapter 3 covers entities, the mechanism at the heart of federation, which allows one type to be defined and extended by several subgraphs. It explains keys, reference resolution, and the directives that govern shared fields. Chapter 4 turns from mechanics to design. It addresses how to draw subgraph boundaries, how to decide who owns a type, and the schema design patterns that keep a federated graph coherent rather than a patchwork. Chapter 5 treats composition as a contract. It covers schema registries, checks against real client traffic, the delivery pipeline for subgraph changes, and how to move fields between subgraphs without downtime. Chapter 6 is about the router in production: query plan cost, the cross-service equivalent of the N+1 problem, caching, demand control, subscriptions, and the observability needed to understand where a slow query spends its time. Chapter 7 addresses security and governance at the edge, including authentication and authorization in a distributed schema, hiding internal fields, persisted queries, and graph variants for different audiences. Chapter 8 is a migration playbook for organizations that already have a monolithic GraphQL server or REST aggregation layer and need to get to a supergraph without stopping the world. Chapter 9 compares the available stacks, their licensing, their tradeoffs, and the emerging open standard, and ends with the cases where federation is the wrong choice. The conclusion draws out what follows from all of this: what an organization should do differently once it understands federation as a way of distributing ownership, and which questions remain genuinely unsettled. A note on what federation is not Federation is not a performance optimization. A federated query that touches three subgraphs will, all else equal, be slower than the same query answered by one server with direct database access, because it involves more network calls and more serialization. Federation buys organizational throughput, the ability of many teams to ship independently, at a cost in runtime complexity. Good router design and good schema boundaries make that cost small, sometimes negligible. They do not make it zero. Federation is also not a replacement for service design. If the underlying services are tangled, with shared databases, circular dependencies, and unclear ownership, federation will expose that tangle in the schema rather than fix it. Many teams discover the true shape of their domain model only when they try to federate it and find that every type wants to live in every subgraph. And federation is not mandatory. A company with one backend team and one frontend client almost certainly does not need it. A company with a dozen product teams, several client applications, and a history of fighting over a shared GraphQL server very likely does. Most organizations sit somewhere between, and part of the purpose of this booklet is to give them the means to judge where. Chapter 1: The Gateway Bottleneck Microservices promised that teams could build, deploy, and scale their parts of a system independently. For backend-to-backend communication, that promise is often kept. A pricing service can ship a new version without coordinating with the inventory service, provided the contract between them holds. But clients do not want to talk to forty services. A mobile app rendering a product page needs the product's name and images from the catalog service, its price from pricing, its stock level from inventory, its reviews from a reviews service, and personalized recommendations from a machine learning service. Asking the app to make five calls over a cellular connection, handle five failure modes, and join the results itself is a poor design, and every organization that tries it eventually builds something in between. That something is the API gateway, or more precisely the aggregation layer: a piece of software that presents clients with a convenient, client-shaped API and translates their requests into calls against the backend services. This chapter is about why that layer, whatever form it takes, tends to become the bottleneck that microservices were supposed to remove, and what the industry learned from the successive attempts to fix it. How aggregation layers become monoliths The first aggregation layer in most organizations is written by whichever team needs it first. Often it is the web team, writing a Node.js service that calls internal REST endpoints and returns JSON tailored to their pages. It is small, fast to change, and owned by the people who consume it. For a while, this works well. Then a second client arrives. The mobile team wants the same data in a slightly different shape, with fewer fields to save bandwidth. They can either add endpoints to the web team's service or build their own. If they add to the existing service, they now share a codebase and a release schedule with a team whose priorities are different. If they build their own, they duplicate the logic for calling the backend services, handling their errors, and applying business rules that inevitably leak into the aggregation code. GraphQL appeared to resolve this tension. Instead of one endpoint per screen, the aggregation layer exposes a typed schema, and each client asks for exactly the fields it needs. The web team and the mobile team can share one server because neither forces its response shape on the other. Facebook built GraphQL internally in 2012 for precisely this reason, to serve its mobile applications from a single flexible API, and released it publicly in 2015. Within a few years it had become a common choice for exactly this aggregation role. But a single GraphQL server does not remove the centralization; it relocates it. The schema is now the shared resource. Every domain that wants to expose data to clients must add types and resolvers to the one server. The resolvers call the domain services, but they also contain decisions: how to map a backend's error codes to GraphQL errors, how to paginate, which fields to compute, which permissions to check. Those decisions accumulate in the GraphQL server, and the server's owners become responsible for code whose business meaning they do not understand. The symptoms are recognizable. Pull requests to the GraphQL repository wait days for review because the owning team is small and the contributors are many. A change to the orders types breaks a test in the user types because of some shared helper. The deploy pipeline takes forty minutes and must be green for every domain before any domain can ship. Incidents in the GraphQL server take down every client feature at once, and the on-call engineer for the server is paged for failures that originate in services they have never seen. Domain teams stop adding fields because the friction is too high, and instead build side-channel REST endpoints that clients call directly, which defeats the purpose of having a unified API. Netflix described this trajectory publicly in 2020. Its studio engineering organization, which builds the software used to produce films and series, had adopted a single GraphQL API, internally called Studio API, to unify access to many backend services. It was successful enough that it became a central bottleneck: one team maintained a schema that represented the work of many domain teams, and that team became the gate through which every change had to pass. Netflix's response was to move to a federated architecture, which it called Studio Edge, in which domain teams owned and operated their own portions of the graph behind a shared gateway. The account is useful because it describes the problem in the terms most organizations experience it: not as a technical scaling limit, since the server could handle the traffic, but as an organizational one, where the coordination cost of a single schema grew faster than the number of teams contributing to it. The underlying force: Conway's law In 1968 Melvin Conway observed that organizations which design systems are constrained to produce designs that copy their own communication structures. The observation, now called Conway's law, is usually invoked as a warning, but it is better read as a description of a force that can be worked with rather than against. A monolithic API layer violates the structure of a microservices organization. The organization has many teams, each owning a domain. The API layer has one owner. Every time a domain team needs to change what clients can see, it must communicate with the API team, and that communication path is the bottleneck. No amount of process improvement removes it, because the bottleneck is structural: the system's design requires a conversation that the organization's design makes expensive. The inverse is also true. If the API layer is split so that each domain team owns the part of the client-facing API that corresponds to its domain, then most changes require no cross-team communication at all. The team that adds a "estimated delivery date" field to orders is the orders team, which owns both the field and the service that computes it. The only coordination required is ensuring that the new field does not conflict with anything other teams have defined, and that check can be automated. This is the essential insight behind federation. The goal is not to make the API layer faster or more flexible. The goal is to align the API layer's ownership structure with the organization's ownership structure, so that the communication paths the system requires are the ones the organization already has. Earlier attempts and what they taught Federation did not appear from nowhere. It was the product of several earlier approaches, each of which solved part of the problem and failed in instructive ways. Backends for frontends In 2015 Sam Newman popularized the backend-for-frontend pattern, abbreviated BFF, which had been used at SoundCloud and elsewhere. Instead of one general-purpose aggregation layer, each client experience gets its own: a web BFF, an iOS BFF, an Android BFF, each owned by the team that builds the corresponding client. The pattern resolves the conflict between clients with different needs by refusing to share a layer at all. BFFs work well when the number of clients is small and the client teams have the backend skills to maintain a service. They solve the problem of client teams waiting on a central API team. But they do not solve the problem of domain logic leaking into the aggregation layer; they multiply it. Each BFF reimplements the calls to the catalog service, the handling of its errors, and the rules for which products are visible to which users. When the catalog service changes its contract, every BFF must change. The domain team, which understands its own logic best, has no ownership of how that logic is exposed to clients. Schema stitching Schema stitching was the first serious attempt to distribute a GraphQL schema. The idea, implemented in the graphql-tools library maintained first by Apollo and later by The Guild, was to take several independent GraphQL schemas, each served by its own service, and merge them into one schema at a gateway. The gateway would delegate parts of each incoming query to the appropriate underlying service. Early stitching had a significant weakness: the logic for connecting types across services lived in the gateway. If the reviews service had a Review type with a productId field and the catalog service had a Product type, linking them so that a client could query product { reviews { ... } } required someone to write resolver code in the gateway that knew about both services. That gateway code became, once again, a central place where cross-domain knowledge accumulated and a central team became the bottleneck. Apollo deprecated its support for schema stitching in 2019 when it introduced federation, arguing that the connection logic belonged in the services rather than the gateway. The Guild later revived and substantially improved stitching with a feature called type merging, which moved much of the configuration closer to the services. Stitching remains a legitimate approach for some situations, particularly when integrating schemas from services you do not control. But the lesson from its early form was clear: any design that requires a central component to understand the relationships between domains will recentralize ownership. Hand-built gateways Many organizations built their own solutions. Some wrote GraphQL gateways that loaded schema fragments from each service at startup and merged them using custom conventions. Others used REST-based API gateways with response composition features. These efforts often worked within a single company but carried large maintenance costs, and each reinvented the same hard problems: how to plan a query across services efficiently, how to detect conflicts between schema fragments before deployment, and how to evolve shared types without breaking clients. The comparison across these approaches is worth laying out before turning to federation itself. Table 1 summarizes where each places the knowledge of how domains relate, and who owns it. Table 1. Where cross-domain knowledge lives in each approach. Approach Who owns client-facing schema Where cross-domain joins live Main failure mode Single GraphQL server Central API team Central server resolvers Review and deploy bottleneck Backends for frontends Each client team Duplicated in every BFF Domain logic duplicated and drifting Early schema stitching Domain teams, plus gateway team Gateway configuration code Gateway team becomes bottleneck Federation Domain teams Declared in subgraph schemas Poor boundaries cause excessive fan-out The last row anticipates the argument of the rest of this booklet. Federation's distinctive move is to put the declaration of how types connect across domains into the subgraph schemas themselves, using directives that a composition tool can read and verify. The gateway, which federation calls the router, does not contain domain logic. It reads a composed schema that tells it which service can resolve which field and how to look up an entity in each service, and it plans queries mechanically from that information. What a solution must provide Drawing these lessons together gives a set of requirements that any architecture for a distributed client-facing API must meet. They are worth stating plainly because they become the criteria for judging every design decision later in this booklet. First, clients must see one coherent API. A single endpoint, a single schema, consistent naming and conventions, and the ability to query across domains in one request. If clients have to know which service owns which data, the aggregation problem has been pushed back onto them. Second, domain teams must own their slice of that API end to end. They must be able to add, change, and deprecate the types and fields that correspond to their domain without waiting on another team, and they must own the code that resolves those fields. Third, the connections between domains must be declared by the domains, not implemented centrally. When the reviews team wants to add a reviews field to the Product type, it should be able to do so in its own service, without asking the catalog team to change anything and without a gateway team writing glue. Fourth, conflicts must be caught automatically and early. When two teams' changes are incompatible, the system must say so before either change reaches production, and it must say so in the delivery pipeline of the team making the change. Fifth, the runtime cost must be bounded and observable. Splitting one server into many introduces network hops. The architecture must plan queries efficiently, batch requests to each service, and make it possible to see where time is spent when a query is slow. Federation, in the form that has matured since 2019, is an attempt to meet all five requirements. The next chapter describes its components and follows a query through them. The chapters after that address how well it meets each requirement in practice, and what an organization must do to close the gaps. The cost of doing nothing It is worth pausing on the alternative, because many organizations live with the gateway bottleneck for years before addressing it, and the costs are easy to underestimate because they are diffuse. The most visible cost is lead time. If adding a field to the client-facing API requires a pull request to a shared repository, a review from a busy central team, and a deploy through a slow pipeline, then the lead time for any client-visible change has a floor set by that process. Domain teams that could ship backend changes in hours wait days or weeks for those changes to reach users. A less visible cost is the erosion of the API itself. When the shared server is hard to change, teams route around it. They expose REST endpoints directly to clients, or they overload existing fields with new meanings, or they add generic escape hatches such as a metadata field of type JSON that holds whatever the domain team wants without schema review. Each workaround makes the API less coherent and less self-describing, which undermines the reasons for adopting GraphQL in the first place. A third cost is concentrated operational risk. A single server that implements every domain's resolvers is a single failure domain. A memory leak in a rarely used resolver for one domain can exhaust the process and take down every client feature. A slow dependency in one domain can tie up the server's connection pool and degrade unrelated queries. Teams that have moved to microservices for fault isolation find that their client-facing layer has reintroduced exactly the shared fate they were trying to escape. The final cost is the hardest to measure: the ownership gap. When the people who write the resolvers for a domain are not the people who own that domain, the resolvers are always slightly wrong. They miss edge cases the domain team would know about. They encode assumptions that were true when they were written and have since changed. They are maintained reluctantly by engineers who would rather be working on their own domain. Every organization that has run a large shared GraphQL server has resolvers that nobody understands and nobody wants to delete, and those resolvers are a direct product of the ownership gap. None of these costs alone forces a change. Together, at sufficient scale, they make the case for an architecture in which the client-facing API is owned, piece by piece, by the teams that own the underlying domains. That is the case for federation, and it is why the next chapter begins not with directives or routers but with the idea of a graph assembled from independently owned parts. Chapter 2: The Supergraph Model A federated GraphQL architecture has four moving parts: subgraphs, composition, the supergraph schema, and the router. Each is simple in isolation. The design becomes powerful, and occasionally confusing, in the way they interact. This chapter describes each part, then follows a single query from a client through the router to several subgraphs and back, so that the mechanics are concrete before the following chapters turn to design and operation. The examples use a small retail domain that recurs throughout the booklet. There are three subgraphs. The products subgraph, owned by the catalog team, knows about products: their identifiers, names, descriptions, and images. The inventory subgraph, owned by the fulfillment team, knows how many units of each product are in stock and where. The reviews subgraph, owned by a customer-experience team, stores reviews that customers have written about products. A client wants to render a product page showing the product's name, its stock status, and its three most recent reviews. Subgraphs A subgraph is an ordinary GraphQL service with two additions. First, its schema uses a set of federation directives to declare how its types relate to types in other subgraphs. Second, it implements a small number of extra fields that the router uses to discover its schema and to look up entities by key. Here is a minimal schema for the products subgraph, written in Federation 2 syntax: extend schema @link(url: "https://specs.apollo.dev/federation/v2.12", import: ["@key", "@shareable"]) type Query { product(id: ID!): Product topProducts(first: Int = 10): [Product!]! } type Product @key(fields: "id") { id: ID! name: String! description: String imageUrl: String } The @link directive at the top declares which version of the federation specification this subgraph uses and which directives it imports. This mechanism, introduced with Federation 2, allows subgraphs to opt into newer features one at a time and lets composition tools know exactly which semantics each subgraph expects. Different subgraphs in the same supergraph may link different federation versions, and composition handles the combination. The @key directive on Product is the central idea of federation. It declares that Product is an entity: a type that can be identified by a key, here its id field, and that other subgraphs may contribute fields to. Chapter 3 is devoted to entities. For now the essential point is that @key tells the router how to ask this subgraph for a specific product when it already knows the product's identifier. The inventory subgraph looks like this: extend schema @link(url: "https://specs.apollo.dev/federation/v2.12", import: ["@key"]) type Product @key(fields: "id") { id: ID! inStock: Boolean! quantityAvailable: Int! } Notice that the inventory subgraph also defines a Product type with the same key. It does not define name or description; it defines only the fields it owns. It also has no root Query field for products at all. It does not need one, because it never serves as the entry point for a product query. It only adds fields to products that some other subgraph has already found. The reviews subgraph adds a reviews field to Product and defines its own Review type: type Product @key(fields: "id") { id: ID! reviews(first: Int = 5): [Review!]! } type Review { id: ID! rating: Int! body: String! createdAt: String! } Each subgraph is deployed and operated by its owning team. Each can be written in a different language and use a different GraphQL server library, provided that library supports the federation subgraph specification. Libraries exist for most mainstream languages, including Apollo Server and other JavaScript servers, Netflix's DGS framework and Spring for GraphQL in the Java ecosystem, Strawberry and Graphene in Python, gqlgen in Go, async-graphql in Rust, and Hot Chocolate in .NET. Under the hood, a federation-capable library adds two fields to the subgraph's Query type. The service field returns the subgraph's schema as a string of SDL, the schema definition language, so that tools can retrieve it. The entities field accepts a list of entity representations, each an object containing a type name and key fields, and returns the corresponding entities. When the router needs inventory data for products with identifiers 1 and 2, it calls _entities on the inventory subgraph with two representations, and the subgraph's reference resolver for Product looks each one up. Client applications never call these fields; they exist only for the router. Composition Composition is the process of taking all the subgraph schemas and producing a single supergraph schema. It is a build-time step, not a runtime one, and that distinction is the source of much of federation's reliability. During composition, the composition tool performs three jobs. It merges the type definitions, so that the three partial definitions of Product become one type with id, name, description, imageUrl, inStock, quantityAvailable, and reviews. It validates that the merge is coherent: that types with the same name have compatible definitions, that key fields exist and have matching types, that shared fields are marked as shareable, and, most importantly, that every field in the merged schema can actually be resolved by some path through the subgraphs. And it annotates the result with metadata recording which subgraph can resolve which field and how entities are keyed in each. That third validation, sometimes called satisfiability checking, is what makes federation trustworthy. Composition does not merely check that the schemas are syntactically compatible. It checks that for every field a client might query, starting from any root field, the router can construct a plan to fetch it. If the reviews subgraph adds a field to Product but the Product type in reviews uses a key that no other subgraph can provide, composition fails with an error explaining which field is unreachable and why. The error appears at build time, in the pipeline of the team that introduced it, rather than as a runtime failure for a client. Composition is performed by a library. In the Apollo ecosystem, that library was historically written in JavaScript and wrapped by the Rover command-line tool and the GraphOS platform. With Federation 2.15, released in July 2026, Apollo rewrote composition in Rust. Other vendors maintain their own composition implementations: WunderGraph Cosmo and The Guild's Hive both compose Apollo Federation subgraphs with their own code, and both publish compatibility test results against the Apollo specification. Locally, a developer can compose a supergraph with Rover from a configuration file listing the subgraphs: federation_version: =2.12.0 subgraphs: products: routing_url: http://products:4001/graphql schema: file: ./products.graphql inventory: routing_url: http://inventory:4002/graphql schema: file: ./inventory.graphql reviews: routing_url: http://reviews:4003/graphql schema: file: ./reviews.graphql Running rover supergraph compose --config supergraph.yaml produces the supergraph schema. In production, composition usually happens in a schema registry rather than on a laptop, for reasons Chapter 5 explains. The supergraph schema The output of composition is the supergraph schema, a single SDL document that contains every type and field from every subgraph, annotated with routing metadata. It is not meant to be read by humans regularly, but looking at a fragment once demystifies what the router knows. The merged Product type in a supergraph schema looks roughly like this: type Product @join__type(graph: PRODUCTS, key: "id") @join__type(graph: INVENTORY, key: "id") @join__type(graph: REVIEWS, key: "id") { id: ID! name: String! @join__field(graph: PRODUCTS) description: String @join__field(graph: PRODUCTS) inStock: Boolean! @join__field(graph: INVENTORY) quantityAvailable: Int! @join__field(graph: INVENTORY) reviews(first: Int = 5): [Review!]! @join__field(graph: REVIEWS) } The @join__type directives say that each of the three subgraphs knows about Product and can look it up by id. The @join__field directives say which subgraph resolves each field. Elsewhere in the document, an enum lists the subgraphs and their URLs. Everything the router needs to plan a query is in this one file. From the supergraph schema, the router derives a second schema, the API schema, which is what clients see. The API schema strips out all federation metadata and any fields marked as inaccessible. When a client introspects the router or a developer browses it in a GraphQL IDE, they see a clean schema with a Product type that has seven fields and no indication that it spans three services. That opacity is deliberate. Clients should not know or care which team owns which field. The router The router is the runtime component. It accepts client requests over HTTP, validates them against the API schema, plans how to execute them across subgraphs, executes the plan, and assembles the response. Apollo's original gateway was a Node.js library, @apollo/gateway, that ran inside Apollo Server. In 2022 Apollo released a standalone router written in Rust, which became the recommended runtime. The router's second major version, released in early 2025, replaced the query planner, which had until then been a JavaScript library executed inside the Rust process, with a native Rust implementation, and Apollo reported meaningful reductions in CPU and memory use as a result. Version 1 of the router reached end of support on March 31, 2026, and Apollo now operates a long-term support policy in which LTS releases arrive roughly every six months. Other routers are fully capable alternatives. WunderGraph's Cosmo Router is written in Go and licensed under Apache 2.0. The Guild offers Hive Gateway, written in JavaScript and running on Node.js, Bun, Deno, or edge runtimes, and since September 2025, Hive Router, written in Rust. All of them read an Apollo Federation supergraph schema, or an equivalent configuration produced by their own composition, and all of them perform the same basic job. A router is usually configured with a YAML file and given the supergraph schema either as a local file or by polling a registry. A minimal Apollo router configuration might set the listen address, enable query deduplication toward subgraphs, and set a timeout for each subgraph call: supergraph: listen: 0.0.0.0:4000 traffic_shaping: all: deduplicate_query: true subgraphs: reviews: timeout: 2s include_subgraph_errors: all: false The router contains no business logic. It does not know what a product is or how reviews are stored. It knows only what the supergraph schema tells it: which fields live where, and how to move between subgraphs using keys. Following a query Now consider the product page query: query ProductPage($id: ID!) { product(id: $id) { name inStock reviews(first: 3) { rating body } } } When the router receives this query, it first parses and validates it against the API schema. Then it builds a query plan, a small program describing which subgraph calls to make, in what order, and how to merge their results. The router reasons as follows. The root field product is only defined in the products subgraph, so the plan must begin with a fetch to products. That fetch can also include name, since products resolves it, and must include id, because the router will need the key to fetch the remaining fields elsewhere, plus __typename, so that the router knows the concrete type of the returned entity. The fields inStock and reviews live in inventory and reviews respectively. Neither depends on the other, so once the router has the product's key it can fetch from both in parallel. The resulting plan has a sequence of two steps. The first step is a single fetch to products: query { product(id: $id) { __typename id name } } The second step runs two fetches in parallel, each calling _entities with the representation obtained in the first step. To inventory, the router sends a query asking for inStock on Product entities. To reviews, it sends a query asking for reviews(first: 3) { rating body }. Each subgraph's reference resolver loads the product by id and resolves the requested fields. When both responses arrive, the router merges them into the result of the first fetch, a step called flattening, and returns a single JSON response shaped exactly like the client's query. From the client's point of view, nothing about the response reveals that three services were involved. Two properties of this process are worth emphasizing. First, the query plan for a given operation is deterministic and depends only on the operation and the supergraph schema, so routers cache plans and reuse them for every request with the same operation. Planning is expensive for complex operations, sometimes taking tens of milliseconds, so plan caching and pre-warming the cache when a new supergraph is deployed matter in production. Second, the plan always uses _entities calls for the second and later steps, and those calls accept a list of representations. If the root field had returned twenty products instead of one, the router would still make exactly one call to inventory and one to reviews, each with twenty representations. Batching at the entity level is built into the protocol. When a subgraph fails The walk-through above assumes every subgraph answers. In production, some will not. A subgraph may time out, return an error, or be unreachable during a deploy. How the router behaves in that case is one of the most consequential and least discussed properties of a federated graph, and it is determined largely by the nullability choices in the schema. When a fetch fails, the router does not abandon the whole response. It records an error, sets the fields that fetch was meant to supply to null, and continues with the rest of the plan. If the reviews subgraph times out on the product page query, the client receives the product's name and stock status, a null for reviews, and an entry in the response's errors array explaining which path failed. The page can render without its reviews section. This partial-response behavior is a large part of what makes federation resilient: one domain's outage degrades one part of the experience rather than all of it. But GraphQL's null propagation rules can undo this resilience. If a field is declared non-null and its value comes back null, the null propagates upward to the nearest nullable ancestor. In the example schema, reviews is declared as [Review!]!, a non-null list. If the reviews fetch fails, reviews cannot be null, so the null propagates to product, which is nullable, and the client receives product: null. The name and stock status that were successfully fetched are discarded. A failure in one subgraph has erased data from two others. The lesson is that fields contributed to an entity by a subgraph other than its origin should usually be nullable, even when the contributing subgraph would never return null under normal conditions. Nullability in a federated schema is not only a statement about the data; it is a statement about failure isolation. A non-null field contributed by a secondary subgraph ties the availability of the whole parent object to that subgraph. Teams accustomed to monolithic schemas, where marking fields non-null is often encouraged for client convenience, need to unlearn that habit for cross-subgraph fields. Many organizations encode this as a lint rule: fields added to an entity by a subgraph that is not the entity's origin must be nullable. The router's own configuration also matters here. Per-subgraph timeouts determine how long the router waits before declaring a fetch failed. A timeout that is too generous lets one slow subgraph hold every response hostage; a timeout that is too tight produces spurious failures under ordinary load. Setting timeouts per subgraph, based on each subgraph's observed latency distribution, is far better than a single global value. Some routers also support retries for idempotent fetches and circuit breaking, which stops calling a failing subgraph for a short period so that it can recover. All of these settings belong to the central team that operates the router, but they should be agreed with the subgraph owners, since only they know which of their fields are critical and which can safely be dropped. Why the build-time step matters It would have been possible to design federation so that the router discovered subgraph schemas at startup and merged them dynamically. Early versions of Apollo's gateway did exactly this, using a mode in which the gateway queried each subgraph's _service field and composed on the fly. That mode still exists for local development, but it is strongly discouraged in production, and the reason illuminates the whole design. If composition happens at runtime, a subgraph that deploys an incompatible schema change breaks the router the next time it restarts or polls. The failure surfaces in production, possibly hours after the deploy, and possibly in a different team's on-call rotation. The router might fail to start at all, taking down every client. If composition happens at build time, in a registry that every subgraph publishes to, the incompatible change is rejected when the team tries to publish it. The router keeps serving the last successfully composed supergraph. The team that made the change sees the composition error in their own pipeline and fixes it. No client is affected, and no other team is paged. This shift, from discovering compatibility at runtime to verifying it at build time, is the reason federation can deliver independent deployment without sacrificing reliability. Every later chapter that discusses operational practice depends on it. The registry and the checks it runs become, in effect, the contract enforcement mechanism between teams, and Chapter 5 treats them in detail. Before that, Chapter 3 examines the mechanism that makes cross-subgraph types possible: entities, keys, and the directives that govern who resolves what. Chapter 3: Entities and Keys Everything distinctive about federation rests on one concept: the entity. An entity is a type whose instances can be uniquely identified, and which more than one subgraph can contribute fields to. Products, users, orders, accounts, shipments, and invoices are typical entities. They are the nouns of a business that several teams care about, each from its own angle. Without entities, federation would be nothing more than a router that sends different root fields to different services. With them, it becomes possible for a client to start at a product and walk to its reviews, its stock levels, its seller, and the seller's ratings in one query, with each hop resolved by a different team's service. This chapter explains how entities work at the protocol level, how keys are chosen, and how the family of federation directives controls which subgraph resolves which field. It is the most technical chapter in the booklet, and it is worth reading carefully, because nearly every design problem in a federated graph traces back to a decision about entities. Keys and reference resolvers A type becomes an entity in a subgraph when the subgraph annotates it with @key. The directive's fields argument names one or more fields that together identify an instance uniquely. A product might be keyed by id. A user might be keyed by id or, in another subgraph, by email. A line item might be keyed by a compound key of orderId and lineNumber. Declaring a key creates an obligation. The subgraph must be able to take a representation, a small object containing typename and the key fields, and return the full entity. The code that does this is called a reference resolver. In Apollo Server and many other JavaScript libraries it is written as a special resolver named resolveReference: const resolvers = { Product: { __resolveReference(ref, { dataSources }) { return dataSources.inventory.byProductId(ref.id); }, inStock: (product) => product.quantity > 0, quantityAvailable: (product) => product.quantity, }, }; When the router sends the inventory subgraph an entities query with a list of representations, the library calls _resolveReference once per representation and then resolves the requested fields on each result. In Java with DGS, the equivalent is a method annotated as an entity fetcher. In Go with gqlgen, it is a generated function on the entity resolver. The names differ; the obligation is the same. The reference resolver is where federation's runtime cost most often goes wrong. Because entities receives a list, a naive implementation that calls the database once per representation will issue one query per product. The router has batched twenty product lookups into one HTTP call, and the subgraph has unbatched them into twenty database queries. The fix is to use a batching layer, such as the DataLoader pattern common in GraphQL servers, so that all representations in one entities call are loaded with a single query. This is the most important performance rule for subgraph authors, and Chapter 6 returns to it. Multiple keys An entity can have more than one key, declared by repeating the directive: type User @key(fields: "id") @key(fields: "email") { id: ID! email: String! displayName: String! } Multiple keys are useful when different subgraphs know an entity by different identifiers. An identity subgraph might know users by internal id, while a marketing subgraph that imports data from an email platform knows them only by email. If the identity subgraph declares both keys, the router can enter it with either, and can therefore bridge the two. The router will fetch the user from marketing with whatever it has, ask identity to resolve the other key, and continue. Composition verifies that such bridges exist where needed. Compound and nested keys Keys can contain several fields, and even nested selections: type Shipment @key(fields: "orderId carrier { code } trackingNumber") { orderId: ID! carrier: Carrier! trackingNumber: String! } Compound keys are legitimate when no single field identifies an instance, but they carry costs. Every subgraph that references the entity must be able to supply every key field, and every representation sent over the wire contains all of them. In practice, a stable surrogate identifier is almost always a better key than a compound natural key, and teams that can introduce one usually should. Non-resolvable references Sometimes a subgraph needs to refer to an entity without contributing any fields to it. The reviews subgraph, for instance, might want a Review.author field that returns a User, but the reviews team has no data about users beyond their identifier. It can declare a stub: type User @key(fields: "id", resolvable: false) { id: ID! } type Review { id: ID! author: User! } The resolvable: false argument tells composition that the reviews subgraph cannot be used to look up users by key; it only returns references to them. The reviews service resolves author by returning an object containing only the id. The router then fetches whatever user fields the client asked for from the subgraph that does own users. This is how one domain links to another without taking on any responsibility for the other's data. Who resolves a field: the directive family In a simple graph, every field of an entity is resolved by exactly one subgraph, and composition can determine which one from the schemas alone. Real graphs are messier. Several subgraphs might all have a product's name available. One subgraph might need another's data to compute a field. A subgraph might be able to return some of another subgraph's fields cheaply in particular contexts. Federation handles these cases with a small set of directives, summarized in Table 2 and explained in the sections that follow. Table 2. Core Federation 2 directives that control field resolution. Directive Applied to What it declares Typical use @key Type Entity and its identifying fields Any shared business noun @shareable Field or type Several subgraphs may resolve this field Denormalized names, value types @external Field Field is owned elsewhere, referenced here Used with @requires or @provides @requires Field Resolving this needs external fields first Shipping cost needs product weight @provides Field This path returns some external fields too Avoid a hop for common fields @override Field Take ownership of a field from another subgraph Migrating fields between subgraphs @inaccessible Field or type Present in supergraph, hidden from clients Internal keys, staged rollouts Shareable fields Federation 2 assumes by default that each field is resolved by one subgraph. If two subgraphs both define a non-key field with the same name on the same type, composition fails unless both mark it @shareable. The rule exists to prevent accidental duplication: if two teams independently add Product.price, with different meanings, composition should stop them. When duplication is intentional, @shareable makes it legal. A common case is a denormalized field. The orders subgraph might store each product's name at the time of purchase and be able to return it without calling products. Marking Product.name as shareable in both subgraphs lets the router fetch the name from whichever subgraph it is already talking to, saving a hop. The obligation that comes with @shareable is semantic consistency. The router may resolve a shareable field from any subgraph that defines it, and it chooses based on the query plan, not on any notion of which source is more authoritative. If the orders subgraph's stored name differs from the products subgraph's current name, clients will see different values for the same field depending on the shape of their query. Teams should use @shareable only for fields whose values are guaranteed to agree, or where disagreement is acceptable and understood. Types that have no key and are defined identically in several subgraphs, such as a Money type with amount and currency, are called value types. In Federation 2, value types must be marked shareable, at the type level or on each field, if they are defined in more than one subgraph. Composition also checks that their definitions are compatible. Requires and external The @requires directive solves a common problem: a subgraph needs data owned by another subgraph to compute one of its own fields. The shipping team's subgraph computes an estimated shipping cost for a product, which depends on the product's weight and dimensions, owned by the catalog team. type Product @key(fields: "id") { id: ID! weightGrams: Int @external shippingEstimate(postcode: String!): Money @requires(fields: "weightGrams") } The @external marker on weightGrams tells composition that the shipping subgraph does not resolve this field itself; it only references it. The @requires directive on shippingEstimate tells the router that, before asking shipping for the estimate, it must fetch weightGrams from whichever subgraph owns it and include it in the representation. The shipping subgraph's reference resolver then receives { __typename: "Product", id: "42", weightGrams: 1800 } and can compute the estimate without calling catalog itself. This is a powerful pattern because it moves a service-to-service dependency into the query plan. The shipping service has no client for the catalog service and no knowledge of its endpoint; it simply declares what it needs. But @requires always adds a sequential step to the plan, since the required fields must be fetched first, and chains of requirements can lengthen plans significantly. It is the right tool when the dependency is genuine and the required data is small. It is the wrong tool for making one service an implicit consumer of large amounts of another service's data. Provides The @provides directive is the inverse optimization. It says that when a particular field is resolved by this subgraph, the returned entities will include certain fields that are normally owned elsewhere. If the reviews subgraph stores each author's display name alongside the review, it can declare that Review.author provides displayName: type Review { author: User! @provides(fields: "displayName") } type User @key(fields: "id") { id: ID! displayName: String! @external } When a client asks for reviews and their authors' display names, the router can get everything from reviews in one fetch instead of making a second call to the users subgraph. The same caveats as @shareable apply: the provided values must agree with the owner's, and the optimization is only worth its complexity on hot paths where the saved hop matters. Override The @override directive lets one subgraph take over resolution of a field from another. It exists primarily for migrations: moving Product.inStock from a legacy monolith subgraph to a new inventory subgraph, for example. The new subgraph declares the field with @override(from: "monolith"), and once that composes, the router sends all requests for the field to the new subgraph. Since Federation 2.7, an optional label argument enables progressive override, in which only a percentage of requests go to the new subgraph. Chapters 5 and 8 cover this in the context of safe migration. Interfaces and entities Interfaces complicate federation because an interface in one subgraph may be implemented by types owned by several other subgraphs. Federation 2.3 introduced two features for this. First, interfaces can themselves carry @key, making them entity interfaces, provided every implementing type in that subgraph also has the key. Second, the @interfaceObject directive lets a subgraph contribute fields to every implementation of an entity interface without knowing what the implementations are. Consider a media catalog with an entity interface Media, implemented by Book and Film, both owned by the catalog subgraph. A recommendations subgraph wants to add a similar field to every kind of media. Without @interfaceObject, it would have to define every implementing type and update its schema whenever catalog added a new one. With @interfaceObject, the recommendations subgraph declares Media as an object type marked @interfaceObject with the key, adds its field, and composition takes care of attaching the field to all implementations. When catalog adds Podcast, recommendations needs no change. This is federation's design principle applied to polymorphism: the team adding a field should not need to know details owned by another team. Contextual data Some fields depend on data from an ancestor in the query rather than from the entity itself. A product's price might depend on the currency of the user's account several levels up the query tree. Federation 2.8 added the @context and @fromContext directives for this situation. A type higher in the graph marks itself as providing a named context, and a field deeper down declares an argument whose value is taken from that context rather than supplied by the client. The router threads the value through the plan. These directives are less widely used than the core set and should be reached for only when the alternatives, passing the value explicitly as an argument or including it in a key, are clearly worse. Choosing keys well Because keys bind subgraphs together, choosing them is a lasting decision. A few rules hold up across many graphs. Prefer opaque, stable identifiers. A key that changes, such as a username a customer can edit, forces every subgraph that stores references to handle the change. A key with business meaning tempts teams to parse it. A globally unique, immutable identifier minted by the owning domain avoids both problems. Make sure every subgraph that needs to reference an entity actually has its key. If the reviews service stores reviews against a product SKU but products are keyed by internal id, either reviews must store the id, or products must offer a second key on SKU. Composition will reveal the mismatch, but the fix is a data change, which is far easier to make before the subgraph is built than after. Keep keys small. Every representation crosses the network and every reference resolver must parse it. A key of one short string is cheap. A compound key with a nested object, repeated across thousands of entities in a large list, is not. Decide early which subgraph is the origin for each entity. Although many subgraphs can contribute fields to Product, one of them is usually the system of record, the one that creates products and decides whether a given identifier exists. Other subgraphs should treat that origin as authoritative for existence. A reference resolver in inventory that receives an unknown product identifier should return null for its fields, not invent a product. Clarifying the origin for each entity heads off a class of subtle bugs in which subgraphs disagree about whether something exists. What entities make possible The mechanics in this chapter can seem fiddly in isolation, but their combined effect is profound. A team that owns a domain can extend any entity in the graph with fields drawn from its own data, without asking the entity's origin team for permission and without the origin team writing any code. The reviews team adds Product.reviews. The recommendations team adds Product.similar. The pricing team adds Product.price. Each ships on its own schedule. The client sees a single Product type that grows richer as more teams contribute, and the composition step ensures that every contribution fits. That capability is also federation's main hazard. If every team can extend every entity, the graph can grow in directions nobody planned, and a single client query can end up touching a dozen subgraphs. Whether that happens depends on how boundaries are drawn and how ownership is governed, which is the subject of the next chapter. Chapter 4: Designing Ownership The technical machinery of federation is indifferent to how a schema is divided. Composition will happily merge a supergraph in which every subgraph defines a few fields of every type, or one in which a single subgraph owns nearly everything. The router will plan queries across either. The difference between a federated graph that accelerates an organization and one that slows it down lies almost entirely in decisions about ownership: which subgraphs exist, which teams own them, which types and fields each one defines, and what conventions bind them into a coherent whole. This chapter is about those decisions. It is less about directives and more about domain modeling, team structure, and schema design discipline. Its central argument is that subgraph boundaries should follow business capabilities and team ownership, and that a federated graph needs a small amount of central governance to stay coherent even though its implementation is distributed. Boundaries follow capabilities, not tables The most common mistake in designing a federated graph is to split it along technical lines. A team looks at its existing services, notices there is a users database, an orders database, and a products database, and creates one subgraph per database. Or it looks at its monolithic GraphQL server and splits it by file, with each existing resolver module becoming a subgraph. These splits feel natural but tend to produce subgraphs that are anemic, merely exposing data with little behavior, and highly coupled, since nearly every client query needs data from several of them. A product listing page might touch products for names, pricing for prices, inventory for availability, media for images, and promotions for badges. If those are five separate subgraphs owned by five teams, a single page load becomes a five-way fan-out, and changing the page's data requirements involves five teams. The better starting point is the business capability: a coherent area of the business with its own rules, vocabulary, and decisions. Catalog management, which decides what products exist and how they are described, is a capability. Order fulfillment, which decides how orders become shipments, is a capability. Customer identity, which decides who a user is and how they authenticate, is a capability. The domain-driven design community calls these bounded contexts, and the term is useful: each is a context within which words have specific meanings and rules are consistent. A subgraph should usually correspond to one bounded context and be owned by the team that owns that context. Its schema expresses that context's view of the world. The fulfillment subgraph defines Shipment, Warehouse, and Carrier, and extends Order with shipments and Product with inStock. It does not define product descriptions, because those belong to catalog, and it does not define payment methods, because those belong to payments. This principle gives a test for any proposed field: which team would be paged if this field returned wrong data? That team's subgraph should define the field. If the answer is unclear, or if the honest answer is "two teams would argue about it," the boundary needs more thought. How many subgraphs There is no correct number. Organizations operate supergraphs with anywhere from a handful to hundreds of subgraphs. What matters is that the count reflects team structure rather than technical decomposition. A useful rule of thumb is that a subgraph should be owned by exactly one team, and a team may own several subgraphs, but a subgraph should not be shared between teams. Shared ownership recreates the coordination costs federation is meant to remove. Too few subgraphs, and the old bottleneck returns inside the largest one. Too many, and query plans become long, operational overhead multiplies, and cross-cutting changes such as renaming a widely used type involve a crowd. Most organizations do well to start with one subgraph per major domain team and split further only when a subgraph's ownership becomes contested or its deploy cadence becomes a constraint. Microservices are not subgraphs A subgraph need not correspond to a single microservice. A domain team that owns six backend services may expose them all through one subgraph, which acts as a domain-level facade. Conversely, a single legacy service might be split across two subgraphs during a migration. The subgraph is a unit of schema ownership. The microservice is a unit of deployment and runtime isolation. They often align, but there is no requirement that they do, and forcing one subgraph per microservice tends to produce exactly the anemic, highly coupled subgraphs described above. The entity origin and its contributors Most interesting entities in a graph will be extended by several subgraphs. The previous chapter introduced the idea of an origin subgraph, the one that is the system of record for an entity's existence. Making the origin explicit for each major entity is one of the most valuable governance decisions a federated organization can take. The origin owns the entity's identity, its key, its core descriptive fields, and the root query fields used to find it. Catalog owns Product, Query.product, and Query.searchProducts. Identity owns User and Query.me. Orders owns Order and Query.order. Contributors extend the entity with fields from their own domain. Reviews adds Product.reviews and User.reviews. Loyalty adds User.loyaltyTier. Fulfillment adds Order.shipments. This pattern has two consequences. First, root fields for finding an entity should almost always live in its origin. If three subgraphs each offer Query.product, clients have no clear entry point, and composition will require the field to be shareable with consistent semantics, which is hard to guarantee. Second, contributor subgraphs should not depend on each other unnecessarily. If loyalty needs data from orders to compute a tier, that dependency is real and can be expressed with @requires, but it should be a conscious design choice recorded somewhere, not an accident discovered when a query plan becomes three levels deep. A worked example: who owns the price? Abstract principles about ownership are easiest to test against a contested case. Price is a good one, because in a retail business several teams have a legitimate claim to it. The catalog team stores a list price for every product, entered by merchandisers when the product is created. The pricing team runs an engine that adjusts prices by region, currency, and customer segment. The promotions team applies discounts from campaigns and coupon codes. The checkout team computes the final amount a customer pays, including tax and shipping. When a product page shows a price, which of these is it, and which subgraph should define Product.price? The naive answer is to let whichever team gets there first define the field and have the others contribute adjustments elsewhere. That produces a graph in which Product.price means the catalog list price, Product.regionalPrice comes from pricing, Product.discountedPrice comes from promotions, and clients must understand the internal pipeline to know which one to display. Every client team will get it wrong at least once. The better approach starts from the question proposed earlier: which team would be paged if the price shown to customers were wrong? In most retail organizations, that is the pricing team, because it owns the rules that turn a list price into what a customer sees. The pricing subgraph should therefore own Product.price, returning a Money value for the requesting customer's context. It can declare @requires on the catalog's list price if its engine needs it as input, making the dependency explicit in the query plan rather than hidden in a service call. Promotions are a separate concept from price, and they deserve their own representation. The promotions subgraph can contribute Product.activePromotions, a list of promotion objects each describing a discount and its conditions. Whether the displayed price already includes promotional discounts is a business decision that must be made once, documented, and reflected in field descriptions, so that clients do not apply a discount twice. The checkout total is not a property of a product at all. It belongs to a Cart or Order entity owned by the checkout team, computed from line items, and should never appear on Product. Placing it there would tempt clients to display it out of context. The catalog's list price, finally, may still be useful to some clients, such as internal merchandising tools. The catalog subgraph can expose it as Product.listPrice and, if external clients should not see it, mark it @inaccessible or tag it for an internal contract only. The outcome is a schema in which each field has a single, clear meaning and a single owner, and in which the cross-team dependencies that genuinely exist, pricing depending on catalog, are expressed through federation rather than buried in service code. Reaching it required a conversation among four teams, and that conversation was the valuable part. Federation did not resolve the question; it forced it to be asked and gave the answer a precise place to live. Designing the schema clients will see Distributed ownership creates a risk that the client-facing schema becomes a patchwork, with each team's conventions visible in the parts it owns. One subgraph uses createdAt and another creationDate. One paginates with cursor connections and another with offset and limit. One returns errors in a union type and another throws GraphQL errors. Clients, who see one schema, experience this inconsistency as the API being poorly designed, and they are right. Federation does not prevent this. Composition checks that schemas are compatible, not that they are consistent. Preventing it requires schema design guidelines, agreed across teams and enforced as far as possible by automated linting in the delivery pipeline. The guidelines that matter most are few. Naming conventions for fields, types, enum values, and arguments should be explicit. A single pagination pattern should apply to all lists that can grow without bound; the cursor-based connection pattern described in the Relay specification is the most common choice. A single error-handling convention should cover mutations, whether that is a result union with typed error variants or a payload type with a userErrors list. Scalars for dates, times, money, and identifiers should be shared value types or custom scalars with documented formats. And there should be conventions for deprecation: when a field is marked @deprecated, what reason text it must include, and how long it must remain before removal. Most schema registries can run lint rules on every proposed change. Apollo GraphOS, Hive, and Cosmo all offer configurable linting. Codifying the guidelines as lint rules converts them from documentation that teams forget into checks that teams cannot skip. Design for clients, not for services A deeper principle sits beneath the conventions. A subgraph's schema is part of a client-facing API, and it should be designed around what clients need, not around how the backing service stores data. A subgraph that mirrors its database tables, exposing product_status_code as an integer because that is the column type, imposes the service's internals on every client. A subgraph that exposes ProductStatus as an enum with values ACTIVE, DISCONTINUED, and PREORDER serves clients. This principle matters more in federation than in a single server, because in a single server a central team usually performs this translation, while in federation every domain team must do it for its own subgraph. Organizations adopting federation should expect to invest in schema design skills across teams, through review of significant schema changes, shared guidelines, and examples of good practice. The role of a central graph team Federation distributes implementation and ownership, but most successful federated organizations keep a small central team responsible for the graph as a whole. The name varies: graph platform team, API platform team, graph champions. Its responsibilities are distinct from those of the central API team in a monolithic setup, and the difference is critical. The central graph team does not write resolvers, does not own domain types, and does not approve routine changes. It owns the router and its configuration, the schema registry and its checks, the delivery pipeline tooling that domain teams use to publish subgraphs, and the schema design guidelines and lint rules. It also acts as a resource when domain teams face hard design questions, and it arbitrates when two teams disagree about ownership of a type or field. The test of whether a central graph team is configured well is whether routine domain changes flow through without involving it. If the orders team can add a field to Order, pass automated checks, and ship to production without speaking to the graph team, the system is working. If every change requires the graph team's sign-off, the old bottleneck has been rebuilt with a new name. Schema review for significant changes Some changes do warrant human review beyond automated checks: introducing a new entity, adding a new root field, defining a new shared value type, or changing a convention. These changes affect the overall shape of the graph and are hard to undo once clients depend on them. A lightweight process works best: a proposal describing the change, reviewed asynchronously by a small group drawn from the graph team and interested domain teams, with a short time limit. Apollo's GraphOS includes a schema proposals feature on its paid plans that supports exactly this workflow, and other teams implement it with pull requests to a shared design repository. The format matters less than the principle: review is reserved for decisions that shape the graph, not applied to every field. Anti-patterns to recognize early Several patterns recur in federated graphs that have gone wrong, and recognizing them early saves considerable pain. The god entity. One type, often User or Product, accumulates fields from every subgraph until it has hundreds of fields contributed by dozens of teams. This is partly natural, since central entities really are touched by many domains, but it becomes a problem when contributors add fields that are not genuinely about that entity. User.recentOrders belongs on User. User.featureFlagsForCheckoutExperimentV3 probably does not, and might belong on a dedicated type reachable from User or from a root field. The remedy is periodic review of the most extended entities and a willingness to introduce intermediate types that group related fields. The chatty boundary. Two subgraphs that call each other constantly, through @requires in one direction and entity references in the other, signal that the boundary between them is in the wrong place. If every query that touches orders also needs fulfillment data and vice versa, perhaps they belong in one subgraph, or perhaps the data that crosses the boundary should be owned differently. The pass-through subgraph. A subgraph whose resolvers do nothing but call a single REST endpoint and return the result, with no domain logic, is often a sign that the team building the subgraph does not own the underlying domain. This recreates the ownership gap from the monolithic API layer, now distributed across many small services. Sometimes it is a legitimate transitional state; the legacy service's owners will eventually take over the subgraph. Increasingly, it is a case for declarative integration features such as Apollo Connectors, introduced in Federation 2.10, which let a subgraph be defined as a mapping onto REST endpoints without writing a service at all. Cosmo's gRPC-based integration and Grafbase's extensions address similar needs. Either way, a pass-through subgraph should be recognized for what it is. The shared everything. Teams, uncertain about ownership, mark many fields @shareable and define them in several subgraphs "just in case." This defeats composition's protection against accidental duplication and creates consistency problems. Shareable fields should be the exception, each justified by a specific performance need or genuinely duplicated data. Ownership is a living decision No organization gets its subgraph boundaries right the first time. Teams reorganize, products evolve, and the graph's shape reveals coupling that was invisible in the service architecture. A federated graph must therefore support moving fields between subgraphs, splitting subgraphs, and merging them, without breaking clients. The schema registry, composition checks, and the @override directive together make that possible, and the next chapter covers how to use them. What makes it organizationally possible is treating ownership as something that is decided explicitly, recorded, reviewed periodically, and changed deliberately when it stops fitting. A graph whose ownership is implicit will drift. A graph whose ownership is explicit can be steered. Chapter 5: Composition as Contract In a monolithic GraphQL server, the schema is protected by the fact that one codebase implements it. A change that breaks a type will usually break a test or a build, and the change cannot be deployed until it is fixed. In a federated graph, that protection disappears. Each subgraph has its own codebase, its own tests, and its own deploy pipeline. Nothing in the orders team's build knows that the reviews team depends on Order.id being a non-null ID. If the protection is to be restored, it must come from somewhere else. That somewhere else is the schema registry and the checks it runs. This chapter argues that the registry is not an optional convenience but the contract enforcement mechanism that makes independent deployment safe, and it describes how to wire it into each team's delivery pipeline so that conflicts are caught before they reach production and changes roll out without coordination meetings. What a schema registry does A schema registry stores the current published schema of every subgraph in a supergraph, along with its routing URL. Whenever a subgraph's schema is published, the registry recomposes the supergraph. If composition succeeds, the new supergraph schema becomes available to routers. If it fails, the previous supergraph stays in place and the publishing team receives the errors. Beyond storage and composition, a good registry offers three further capabilities. It runs checks against proposed schema changes before they are published, answering the question "would this change compose, and would it break any client?" It keeps history, so that teams can see when each change was made and by whom, and can roll back. And it manages variants, separate supergraphs for different environments such as development, staging, and production, each with its own published subgraphs. Several registries are available. Apollo GraphOS is the most widely used, tightly integrated with the Apollo router and Rover CLI. WunderGraph Cosmo includes a registry, called the control plane, alongside its router, and can be self-hosted or used as a managed service. The Guild's Hive is a registry that supports Apollo Federation and can serve composed schemas to Apollo's router, Hive Gateway, Hive Router, or others. Grafbase and ChilliCream offer registries for their own gateways. Some organizations build their own, typically by running a composition library in CI against schemas stored in a repository; this works for small graphs but becomes expensive to maintain as needs grow. The registry also mediates how routers receive new supergraphs. Apollo's router polls an endpoint called Uplink for the latest composed supergraph for its configured graph and variant. Cosmo's router fetches a configuration from a content delivery network fronting the control plane. Hive publishes composed artifacts to a CDN that routers poll. In every case, routers do not compose anything themselves; they receive a pre-validated supergraph and swap it in without restarting, a process usually called a hot reload. The three kinds of checks A schema check takes a proposed subgraph schema and evaluates it against the current state of the graph. Three kinds of check matter, and a mature pipeline runs all three. Composition checks The first check asks whether the supergraph would still compose if this subgraph schema replaced the current one. It catches every class of incompatibility that composition detects: conflicting type definitions, missing shareable markers, keys that no other subgraph can satisfy, fields that become unreachable, and so on. Composition checks are necessary but not sufficient. A change can compose perfectly and still break clients. Removing a field from a subgraph composes cleanly if no other subgraph depends on it, but any client that queries the field will start receiving validation errors. Operation checks The second check asks whether the change would break any operation that clients actually send. To answer it, the registry needs to know what operations clients send. Routers report usage data, typically aggregated counts of each operation and which fields it touched, to the registry. When a proposed change removes a field, changes its type incompatibly, makes a nullable argument required, or removes an enum value, the registry looks up recent traffic and reports which operations would fail, from which clients, and how often they ran. This is one of the most valuable capabilities a federated platform offers, because it replaces guesswork with evidence. A team that wants to remove a deprecated field no longer needs to email every client team and wait for replies. It runs a check, sees that the field has not been queried in thirty days, or that one Android version still queries it at a few hundred requests per day, and decides accordingly. Most registries let teams configure the time window and a threshold below which usage is considered negligible. Operation checks are only as good as the usage data behind them. Routers must be configured to report usage, clients should identify themselves using client name and version headers so that failures can be attributed, and the check window must be long enough to cover infrequent but important operations such as monthly batch jobs or annual reports. A check that looks at seven days of traffic will miss an operation that runs on the first of each month. Lint checks The third check asks whether the change follows the graph's design guidelines. Lint rules can enforce naming conventions, require descriptions on types and fields, forbid certain patterns such as returning bare lists from fields that might grow large, require deprecation reasons, and so on. Lint checks are what make the schema design guidelines from the previous chapter enforceable rather than aspirational. Lint rules should be introduced with care. A rule added to an existing graph will flag hundreds of pre-existing violations. Most registries support evaluating only the changed parts of a schema, which lets new rules apply to new work without demanding an immediate cleanup of everything. Wiring checks into the delivery pipeline Checks are useful only if they run at the right moment: on every proposed change, in the pipeline of the team proposing it, before the change is merged. The standard pattern has three stages. On every pull request that touches a subgraph's schema, the pipeline extracts the schema and runs a check against the production variant. With Apollo's tooling, that step looks like this: rover subgraph check retail-graph@production \ --name inventory \ --schema ./schema.graphql Cosmo's wgc subgraph check and Hive's hive schema:check commands play the same role in their ecosystems. The check results appear in the pull request, and a failing composition or operation check blocks the merge. A team that genuinely needs to make a breaking change can usually override the operation check with an explicit approval recorded in the registry, which creates an audit trail. When the pull request merges, the pipeline deploys the new subgraph code. Only after the deploy succeeds and the new code is serving traffic does the pipeline publish the new schema to the registry: rover subgraph publish retail-graph@production \ --name inventory \ --schema ./schema.graphql \ --routing-url http://inventory.internal/graphql Publishing triggers recomposition, and the router picks up the new supergraph within seconds. Order matters The order of deploy and publish matters, and it depends on the kind of change. For additive changes, such as a new field, deploy first, then publish. If the schema were published first, the router might start sending queries for the new field to subgraph instances that do not yet implement it, and those queries would fail. Deploying first means the subgraph can serve the field before the router knows it exists. For removals, the order reverses. Publish the schema without the field first, so that the router stops routing requests for it, then deploy code that no longer implements it. If the code were deployed first, there would be a window during which the router still believed the field existed and sent requests the subgraph could not answer. Most changes are additive, so most pipelines are written as deploy-then-publish. Removals are rarer and usually preceded by a deprecation period, during which the field remains implemented while clients migrate away. By the time a field is removed, operation checks should confirm that no clients use it, and the order of operations becomes a matter of tidiness rather than risk. Variants and environments Each environment should have its own variant of the supergraph. Development, staging, and production variants each receive publishes from the corresponding deployment of each subgraph. Checks for a pull request should run against production, since that is where client traffic and breaking changes matter, even if the first deploy target is staging. Running checks only against staging, which receives little real traffic, gives a false sense of safety. Moving fields without downtime Chapter 4 argued that ownership changes over time. The registry and the @override directive together make it possible to move a field from one subgraph to another without breaking clients and without a coordinated deploy. Suppose Product.inStock currently lives in a legacy monolith subgraph, and the fulfillment team has built a new inventory subgraph that should own it. The migration has four steps. First, the inventory team implements inStock in their subgraph, marked with @override(from: "monolith"), and deploys the code. Because the schema has not yet been published, the router keeps sending inStock requests to the monolith. Second, the inventory team publishes the schema. Composition sees two subgraphs defining inStock, one overriding the other, and records that the router should send requests to inventory. From this moment, all traffic for the field shifts. Third, after confirming that the new implementation is correct, the monolith's owners remove inStock from the monolith subgraph and publish. The override directive is now redundant. Fourth, the inventory team removes the @override directive in a later change. The second step is the risky one, since it shifts all traffic at once. Progressive override, available since Federation 2.7, reduces that risk by adding a label: type Product @key(fields: "id") { id: ID! inStock: Boolean! @override(from: "monolith", label: "percent(5)") } With this label, the router sends five percent of requests for inStock to inventory and the rest to the monolith. The team monitors error rates and latency, then publishes the schema again with a higher percentage, and so on until it reaches 100. If something goes wrong, they publish the schema with a lower percentage or remove the override, and traffic shifts back within seconds. The rollout and rollback are schema publishes, not code deploys, so they are fast and reversible. Apollo also supports custom labels evaluated by router plugins or coprocessors, enabling override based on client, region, or feature flag rather than random percentage. This capability has a practical effect beyond any single migration. It means boundary decisions are not permanent. A team that discovers it has put a field in the wrong subgraph can move it in an afternoon with a measured rollout. That reversibility makes it easier to start with a reasonable boundary design and correct it over time, rather than agonizing over getting every boundary right up front. Managing breaking changes Federation does not change the fundamental rule of API evolution: clients must not be broken without warning. What it changes is who is responsible for enforcing that rule and how. In a monolithic setup, the central API team guards against breaking changes. In a federated setup, each domain team is responsible for its own subgraph's compatibility, and the registry's operation checks give them the information to exercise that responsibility. The process for removing or changing a field is straightforward. The team marks the field @deprecated with a reason that tells clients what to use instead. The deprecation appears in schema documentation and in most client tooling, and lint rules can require a specific format for the reason. The team monitors usage of the deprecated field through the registry. Client teams migrate to the replacement on their own schedules. When usage drops to zero, or to a level the owning team judges acceptable, the team removes the field. The operation check confirms which clients, if any, would be affected. Mobile clients complicate this process because old app versions remain in use long after new ones ship. A field used by an app version from two years ago may still receive meaningful traffic from users who never update. Operation checks reveal this, and teams must decide whether to keep supporting the field indefinitely, return a safe default, or accept that very old versions will lose functionality. These are product decisions, but the registry makes them informed ones. Contracts as a governance tool Some registries support schema contracts, filtered variants of a supergraph that expose only a subset of types and fields to particular audiences. In Apollo's implementation, fields and types are tagged with @tag directives in subgraph schemas, and a contract variant includes or excludes elements based on tags. A public API might expose only fields tagged public, while an internal API exposes everything. Contracts let a single supergraph serve several audiences with different schemas, each derived automatically from the same source. They are a governance tool because they make audience an explicit property of each field, decided by the owning team at the time the field is added, rather than something a central team has to curate later. Chapter 7 returns to contracts as part of the security picture. The contract is the culture The tools in this chapter are mechanical, but their effect is cultural. When composition and operation checks run on every change in every team's pipeline, teams stop thinking of the shared schema as something owned by somebody else and start thinking of it as a contract they participate in. They see immediately when their change would affect another team. They see which clients depend on their fields. They can move fast because they can see what they would break. Organizations that skip this machinery, running composition only in a central environment or publishing schemas without checks, almost always end up recreating the central gatekeeper they were trying to eliminate. Somebody has to protect the graph, and if machines are not doing it, people will. The registry is how federation keeps that protection while removing the gate. Chapter 6: The Router in Production The router is the only component in a federated architecture that every client request passes through. When it is healthy and well configured, it is nearly invisible: queries arrive, plans execute, responses return, and the added latency is measured in single-digit milliseconds. When it is misconfigured or when subgraphs behave badly beneath it, it becomes the place where problems become visible to every client at once. This chapter treats the router as critical infrastructure and covers the practical concerns of running it: understanding query plan cost, preventing cross-subgraph inefficiency, caching, limiting expensive queries, handling real-time data, and seeing where time goes. The shape of a query plan determines its latency A query plan is a tree of fetches arranged in sequences and parallel groups. Its latency is dominated not by the number of fetches but by the length of its longest sequential chain, sometimes called its critical path. A plan with ten parallel fetches at one level is roughly as fast as its slowest single fetch. A plan with four sequential steps costs the sum of the slowest fetch at each step. Consider a query for a user's recent orders, the products in each order, the reviews of each product, and the authors of those reviews. If users, orders, products, reviews, and authors live in different subgraphs, the plan has at least four sequential steps: fetch the user and order references, fetch order details, fetch product details, fetch reviews, and fetch author details, each depending on keys returned by the previous step. If each subgraph responds in twenty milliseconds at the 95th percentile, the critical path is at least eighty milliseconds before any router overhead, and tail latencies compound, because the probability that at least one step hits its slow tail grows with the number of steps. This is why boundary design matters so much for performance. Every entity hop that crosses a subgraph boundary in a common query adds a step to the critical path. A graph in which the most frequent client queries can be satisfied with one or two sequential steps will perform well. A graph in which they routinely need five will not, no matter how fast the router is. Teams should make query plans visible during development. Apollo's router can return the query plan alongside a response when a debug header is set, GraphOS Studio and Cosmo Studio both render plans for any operation, and Hive provides similar tooling. Reviewing the plans for the most important client operations, especially after significant schema changes, is the simplest way to catch a boundary decision that has made a hot path slow. Requires lengthens plans The @requires directive, described in Chapter 3, always adds a sequential step: the required fields must be fetched before the requiring field can be resolved. A field that requires data from a subgraph that itself requires data from another creates a chain. Each such chain should be a deliberate choice, made with the plan in view. When a requirement appears on a hot path, it is often worth asking whether the requiring subgraph could store a copy of the required data, kept up to date by events, rather than fetching it on every request. The distributed N+1 problem GraphQL servers are notorious for the N+1 query problem: a list of N items, each with a field that triggers a separate database query, produces N+1 queries in total. Federation has an equivalent at the network level, and the router's design addresses half of it. The router's half is batching. When a query plan reaches a step where it needs fields from another subgraph for a list of entities, it sends a single _entities call containing all the representations, not one call per entity. A list of fifty products becomes one call to inventory with fifty representations. This is automatic and requires no configuration. The subgraph's half is handling that batch efficiently. As noted in Chapter 3, a reference resolver that issues one database query per representation turns the router's single batched call back into fifty queries. The subgraph must collect the keys from all representations in the call and load them together, typically with a DataLoader or equivalent batching utility, and an IN clause or batch API call against its data store. A subgraph that gets this wrong may not notice in development, where lists are short, but will show latency that grows linearly with list size in production. A related trap appears with nested lists. If a query fetches fifty orders, each with ten line items, each referencing a product, the router will send one _entities call to products with up to five hundred representations. That is still one network call, but the products subgraph must handle five hundred lookups efficiently, and the payload may be large. The router deduplicates identical representations within a fetch, so if many line items reference the same product, it is looked up once. But a query that genuinely references thousands of distinct entities will be expensive regardless, which is one reason to impose limits on query cost. Deduplication and connection reuse Routers offer several mechanisms to reduce redundant work. Query deduplication collapses identical in-flight subgraph requests: if a hundred clients simultaneously request the same popular product page, the router can send one request to each subgraph and share the result among all waiting clients. Apollo's router enables this with the deduplicate_query setting under traffic shaping, and Cosmo offers a similar single-flight mechanism. Deduplication is safe only for queries, not mutations, and only when the response does not depend on per-user headers; routers include relevant headers in the deduplication key to preserve correctness. Connection reuse matters too. Routers maintain pools of HTTP connections to each subgraph, and using HTTP/2 between router and subgraphs, where supported, allows many concurrent requests to share a connection. Setting appropriate timeouts per subgraph prevents a slow subgraph from holding router resources indefinitely; a subgraph that should answer in fifty milliseconds should not be given thirty seconds. Caching at several layers Federated graphs can cache at several distinct layers, and confusing them is a frequent source of misunderstanding. The query plan cache stores plans keyed by operation and supergraph version. Every router has one, in memory. Because planning complex operations can take tens of milliseconds, and because the cache is cleared when a new supergraph arrives, the first requests after a schema publish can see elevated latency. Routers mitigate this by warming the cache with recently seen operations before swapping in the new supergraph. Apollo's router also supports a distributed plan cache in Redis, a feature that requires a GraphOS enterprise plan, so that a fleet of router instances shares plans rather than each computing its own. Automatic persisted queries reduce request size rather than computation. The client sends a hash of the operation instead of the full text; the router looks up the text in a cache and executes it. If the hash is unknown, the client retries with the full text. This saves bandwidth for large operations on mobile networks but does not restrict which operations can run. Chapter 7 discusses the stricter variant, persisted query safelisting, as a security control. Response and entity caching store the results of subgraph fetches, so that a repeated request for the same entity can be served without calling the subgraph. This is the layer with the largest potential payoff and the most complexity. Cache keys must account for the entity, the fields requested, and any headers or claims that affect the result. Time-to-live values must reflect how fresh each type of data needs to be, often driven by Cache-Control hints that subgraphs attach to their responses. And invalidation, when the underlying data changes, must be handled. Federation 2.12, released in November 2025, introduced a @cacheTag directive that lets subgraphs tag cached entities so they can be invalidated by tag when the underlying data changes. Apollo's response caching is a licensed router feature; other routers offer their own approaches, and many organizations cache at the subgraph level or in a content delivery network for public, non-personalized queries instead. The right caching strategy depends heavily on the data. A product catalog that changes a few times a day is an ideal caching candidate. A shopping cart, which is per-user and changes constantly, is not. Teams should start with the simplest effective layer, often HTTP caching of public queries and subgraph-level caching of expensive computations, and add router-level entity caching only where measurements show it is needed. Limiting expensive queries GraphQL's flexibility allows clients to write queries that are extraordinarily expensive to execute: deeply nested queries, queries that request large lists of lists, or queries that fan out to many subgraphs. In a federated graph, the cost of a bad query is borne by many teams' services at once. Every production router should impose limits. The simplest limits are structural: maximum query depth, maximum number of fields or aliases, maximum number of root fields. Cosmo and Hive support configurable limits of this kind, and Apollo's router offers them as request limits, one of its licensed features. They are blunt but effective against accidental or malicious abuse. Cost-based limits are more precise. Each field is assigned a cost, and the router estimates a query's total cost before execution, multiplying the costs of fields inside lists by the expected list size. Queries over a budget are rejected. Federation 2.9 introduced two directives for declaring costs in subgraph schemas, @cost for the weight of a field or type and @listSize for the expected or maximum size of a list field, based on the IBM-authored GraphQL Cost Directive specification. Apollo's router uses these in a feature it calls demand control, which can estimate cost before execution and measure actual cost afterward: type Query { searchProducts(term: String!, first: Int = 20): [Product!]! @listSize(slicingArguments: ["first"], assumedSize: 100) } type Product @key(fields: "id") { id: ID! recommendations: [Product!]! @cost(weight: 5) @listSize(assumedSize: 10) } Here searchProducts declares that its list size is governed by the first argument, and recommendations declares a higher-than-default weight and an assumed size of ten. A query that requests recommendations of recommendations of a hundred search results will be priced accordingly and rejected if it exceeds the budget. Cost-based limits work best when the domain teams who own subgraphs assign costs to their own fields, since they know which fields are expensive. This is another case where distributed ownership extends to operational properties, not just data. For first-party clients, the strongest limit is to accept only known operations, a practice covered in the next chapter. Cost limits then matter mostly for third-party or public API consumers. Real-time data: subscriptions and defer GraphQL subscriptions let clients receive updates when data changes. In a federated graph, a subscription field is defined in one subgraph, but the updates it produces may include fields resolved by others. When the inventory subgraph emits a stock change event for a product, and the client's subscription asks for the product's name as well, the router must fetch the name from the products subgraph for each event. Apollo added subscription support to federation in version 2.4, and its router supports subscriptions over WebSocket connections to subgraphs and also through an HTTP callback protocol, in which the subgraph posts events to the router rather than holding a socket open. Subscription support in the Apollo router is a licensed feature. Cosmo takes a different approach with what it calls Cosmo Streams, formerly event-driven federated subscriptions, in which the router connects directly to a message broker such as Kafka, NATS, or Redis and resolves subscription events without the subgraph holding long-lived connections at all. Hive Gateway also supports federated subscriptions. Subscriptions are operationally demanding: each one holds a connection open, and every event may trigger fetches to several subgraphs. Teams should use them for genuinely real-time needs, such as live order tracking or collaborative editing, and prefer polling or push notifications for less time-sensitive updates. The @defer directive offers a related capability for queries. A client marks part of a query as deferred, and the router returns the rest immediately, streaming the deferred part when it is ready. This is especially useful in federation, where one slow subgraph can otherwise hold up an entire response. A product page might defer its reviews section, so that name, price, and stock render immediately while reviews arrive moments later. Apollo's router supports @defer for federated queries; client library support varies and should be checked. Seeing where time goes A slow federated query could be slow because of the router, any of several subgraphs, the network between them, or the query plan's shape. Without good observability, diagnosing which is guesswork. The essential tool is distributed tracing. Routers emit traces using OpenTelemetry, with spans for parsing, planning, and each subgraph fetch. If subgraphs also emit OpenTelemetry spans and propagate trace context from incoming requests, the full picture appears in a single trace: the router received the request, spent two milliseconds planning, fetched from products in fifteen milliseconds, then fetched from inventory and reviews in parallel, where reviews took two hundred milliseconds because of a slow database query. Every serious router supports exporting to standard backends such as Jaeger, Zipkin, Datadog, Honeycomb, or any OTLP-compatible collector. Metrics complement traces. The router should export request rates, error rates, and latency distributions, broken down by operation name and client, and by subgraph for fetches. Field-level usage metrics, reported to the registry, serve the operation checks described in Chapter 5 and also show which fields are hot and which are unused. Clients should name their operations and identify themselves. An anonymous query is nearly impossible to attribute when it causes problems. Requiring named operations and client identification headers, enforced at the router, makes every metric and trace more useful. Extending the router No router does everything an organization needs out of the box. Authentication schemes, custom header handling, request logging, and policy enforcement often require custom logic. Routers provide extension points for this. Apollo's router supports Rhai, an embedded scripting language for lightweight request and response manipulation, and external coprocessors, separate services the router calls at defined stages of request handling, which can be written in any language. Coprocessors are a licensed feature. Cosmo supports custom modules written in Go and compiled into the router. Hive Gateway, being JavaScript, supports plugins written in JavaScript or TypeScript, and Hive Router is adding its own extension model. Extension logic runs on every request, so it must be fast and reliable. A coprocessor that adds twenty milliseconds of latency adds it to every query in the graph. A coprocessor that fails takes down every query in the graph unless the router is configured to fail open. The router's extension points should be treated with the same care as the router itself: reviewed, tested, monitored, and owned by the central graph team. The router as a product The router is infrastructure, but it is also, in a sense, a product that the central graph team provides to every domain team and every client. Its reliability determines the graph's reliability. Its configuration encodes policies, such as timeouts, limits, and caching, that affect every team. Its observability determines how quickly any team can diagnose problems. Organizations that succeed with federation run their routers with the discipline they apply to their most critical services: multiple instances behind a load balancer, autoscaling on CPU and request rate, staged rollouts of new router versions and configuration, alerting on error rates and latency, and capacity planning ahead of known traffic peaks. The router's resource needs are usually modest compared with the subgraphs behind it, since it does no business logic and holds no data beyond its caches; vendors publish benchmarks, but the only numbers that matter are those measured with an organization's own operations and subgraphs. But modest resource needs are not the same as low importance. When the router fails, everything fails. Chapter 7: Security and Governance at the Edge A federated graph has a larger attack surface than a monolithic GraphQL server, and its security properties are less obvious. A single server has one entry point, one authentication layer, and one place where authorization decisions are made. A federated graph has a router facing clients and a fleet of subgraphs behind it, each written by a different team, each with its own idea of how to check permissions, and each exposing special fields that exist for the router's benefit and could be abused if reached by anyone else. This chapter describes how to secure that arrangement: where to authenticate, where to authorize, how to protect subgraphs from direct access, how to limit what clients can ask, and how to govern which parts of the graph different audiences can see. Authenticate once, at the edge Authentication, establishing who a request comes from, should happen at the router. The router is the single entry point, and verifying credentials there means subgraphs never deal with raw client credentials and every subgraph sees the same verified identity. The most common pattern uses JSON Web Tokens. The client obtains a token from an identity provider and sends it in an Authorization header. The router validates the token's signature against the provider's published keys, checks its expiry and audience, and extracts claims such as the user's identifier, roles, and scopes. Apollo's router supports JWT authentication natively as a licensed feature, fetching and caching the provider's JSON Web Key Set. Cosmo and Hive Gateway offer equivalent capabilities. Where built-in support is insufficient, a router extension can call an internal authentication service. Once the router has verified the identity, it must pass it to subgraphs. The usual approach is to forward a header, either the original token, which subgraphs can re-verify cheaply, or a set of derived headers containing the user identifier and claims. Routers have explicit header propagation configuration for this, and it is worth being deliberate about it. Forwarding every client header to every subgraph is convenient but can leak information and allows clients to inject headers that subgraphs might trust. A safer configuration forwards only named headers, and strips any client-supplied headers that use the names the router sets for derived claims. Authorize in the domain, with help from the schema Authorization, deciding whether an authenticated identity may access a particular piece of data, is harder to centralize, and in general it should not be fully centralized. The rules for who may see an order belong to the orders domain. The rules for who may see a payroll record belong to the payroll domain. A router that tried to encode all these rules would recreate the central bottleneck federation exists to remove, and would separate the rules from the teams that understand them. The baseline, then, is that each subgraph enforces authorization for its own data, in its own resolvers, using the identity the router forwards. The orders subgraph's resolver for Query.order checks that the requesting user owns the order or has a support role. Its reference resolver for Order performs the same check, a point the next section explains in detail. Federation adds a complementary layer. Federation 2.5 introduced the @authenticated and @requiresScopes directives, and version 2.6 added @policy. These let subgraph authors declare access requirements in the schema itself: type Query { me: User @authenticated } type User @key(fields: "id") { id: ID! email: String! @requiresScopes(scopes: [["profile:read"]]) taxId: String @policy(policies: [["pii_access"]]) } The router reads these directives from the supergraph and enforces them before calling any subgraph. If an unauthenticated request asks for me, or a request without the profile:read scope asks for email, the router removes those fields from the query plan and returns null with an error for them, while still resolving the rest of the query. The @policy directive delegates the decision to custom logic, typically a coprocessor or script, which evaluates named policies against the request. In Apollo's router these authorization directives are licensed features. Schema-declared authorization has real advantages. The access requirement is visible in the schema, reviewable in pull requests, and enforced uniformly. Unauthorized requests are stopped before they reach subgraphs, saving work. And composition carries the directives into the supergraph, so the rule travels with the field even if ownership changes. But it is a complement to domain authorization, not a substitute. The directives express coarse rules, such as "must be logged in" or "must have this scope." They cannot express "may see this order only if it belongs to them," which requires looking at the order's data. Domain checks remain necessary for row-level and relationship-based rules. The sound design uses both: schema directives for coarse, declarative rules and early rejection, resolver checks for fine-grained rules and defense in depth. The entities backdoor The most important security property of a federated graph, and the one most often overlooked, is that subgraphs must not be directly reachable by clients. The reason is the entities field. Every subgraph exposes it so that the router can look up entities by key. It accepts arbitrary representations: any type name, any key values. If a client can send a query directly to a subgraph, it can call entities with any product, user, or order identifier it likes and receive whatever fields that subgraph resolves for that entity. Suppose the orders subgraph checks authorization carefully in Query.order, verifying that the requester owns the order, but its reference resolver for Order simply loads the order by identifier, on the reasonable assumption that the router only calls it with keys that came from an authorized root field. A client who reaches the subgraph directly can bypass Query.order entirely and call _entities with an arbitrary order identifier. The root-field check is never executed. There are two defenses, and a serious deployment uses both. First, network isolation: subgraphs should accept traffic only from routers. This can be enforced with private networking, Kubernetes network policies, a service mesh with mutual TLS so that subgraphs verify the router's identity cryptographically, or at minimum a shared secret header that the router adds and subgraphs require. Second, authorization in reference resolvers: every reference resolver should perform the same authorization checks as the root fields that return that entity. The router may be the only intended caller, but defense in depth means not trusting that assumption. The same reasoning applies to the _service field, which returns the subgraph's full schema including any fields marked @inaccessible. It is harmless when only the router can reach the subgraph, and a disclosure risk when anyone can. Hiding what clients should not see Not every field in the supergraph should be visible to clients. Internal keys, fields used only to satisfy @requires, and fields still under development may need to exist in the supergraph for the router's benefit without appearing in the API schema clients see. The @inaccessible directive serves this purpose. A field or type marked inaccessible in any subgraph is included in the supergraph and can be used by the router in query plans, but is removed from the API schema. Clients cannot query it, and it does not appear in introspection. This is useful for keys that clients should not depend on, such as an internal numeric identifier used to join subgraphs when the public identifier is an opaque string, and for staging new fields: a team can deploy and compose a field marked inaccessible, test it through internal tooling, and remove the marker when ready to release it. Introspection itself is a governance question. In development, introspection lets tools and developers explore the schema. In production, many organizations disable it on the public router to avoid giving attackers a map of the API. Disabling introspection is not a substitute for real access control, since a determined attacker can often reconstruct a schema from error messages and client code, but it raises the cost of reconnaissance, and there is rarely a reason for anonymous production clients to need it. Developers can use the registry's schema explorer instead. Error messages need similar care. Subgraph errors can contain stack traces, internal service names, or database details. Routers can be configured to mask subgraph errors, replacing their messages with generic text for clients while logging full details internally. Apollo's router, for example, controls this with its include_subgraph_errors setting. Production routers should mask by default and allow detailed errors only for trusted internal clients or in non-production environments. Accepting only known operations For first-party clients, such as an organization's own web and mobile apps, the strongest control over what the graph will execute is to accept only operations that have been registered in advance. This practice is usually called persisted queries with safelisting, or trusted documents. The mechanism is simple. When a client is built, its tooling extracts every GraphQL operation the client contains and registers them with the registry or router, each identified by a hash or identifier. In production, the client sends only the identifier, and the router executes the corresponding registered operation. Any request containing an unregistered operation is rejected. This shrinks the attack surface dramatically. An attacker who obtains a user's token can call only the operations the official clients use, with only the variables those operations accept. They cannot craft a deeply nested query to exhaust resources, cannot probe for fields the clients do not use, and cannot exploit any authorization gap in a field the clients never query. Safelisting also improves performance, since registered operations can have their plans precomputed, and it simplifies operation checks, since the registry knows exactly which operations exist. Apollo's router supports safelisting through persisted query lists as a licensed feature, managed through GraphOS. Cosmo supports persisted operations, and Hive provides an app deployments feature for registering client operations and serving them from its CDN. The client-side tooling for extracting operations exists for the major GraphQL client libraries. Safelisting is not appropriate for every audience. A public API for third-party developers must accept arbitrary queries, and there the defenses described in Chapter 6, depth limits, cost analysis, and rate limiting, carry the load. Many organizations run two router configurations, or two graph variants: a safelisted one for first-party clients and a cost-limited one for third parties. Different schemas for different audiences The idea of serving different audiences connects to schema contracts, introduced in Chapter 5. A contract is a filtered variant of the supergraph, derived automatically by including or excluding fields and types according to @tag directives in subgraph schemas. A typical setup has three audiences. Internal tools and back-office applications see the full graph. First-party customer-facing clients see everything except internal administration fields. A public partner API sees only fields explicitly tagged for partners. Each audience gets its own router deployment or variant, serving its contract's API schema. A field not included in a contract does not exist for that audience: it cannot be queried, does not appear in introspection, and cannot be reached by any query plan. Contracts move audience decisions to the point where a field is created. When a domain team adds a field, its tags declare who may see it. A lint rule can require that every new field carry at least one audience tag, so that nothing is accidentally exposed to partners by default. This is federation's ownership model applied to exposure: the team that understands the data decides who sees it, and the platform enforces the decision mechanically. Rate limits, telemetry, and the data the router sees Two further concerns sit at the edge and are easy to neglect until they cause trouble. The first is rate limiting. Cost limits stop any single query from being too expensive, but they do not stop a client from sending a very large number of cheap queries. The router, or a gateway in front of it, should enforce request rate limits per client, per user, or per token, and ideally limits on total query cost per time window, so that a client cannot exhaust its budget by issuing many moderately expensive queries in quick succession. Routers differ in how much of this they provide natively; many organizations rely on an existing API gateway or load balancer in front of the router for coarse rate limits and use router-level cost accounting for finer control. Whatever the mechanism, limits should be tied to client identity, which is another reason to require clients to identify themselves. The second is telemetry. The router sees every query, every variable, and every response that passes through the graph. That makes its traces and logs enormously useful for debugging and also a significant store of sensitive data. A trace that records the variables of a updatePaymentMethod mutation, or a log that captures a response containing addresses and phone numbers, can turn the observability system into a place where personal data accumulates outside the controls applied to the subgraphs that own it. Routers can be configured to omit variable values and response bodies from telemetry, or to redact specific fields. The default for production should be to record operation names, timings, and error codes, but not variable values or response data, with detailed capture enabled only temporarily and deliberately for specific debugging sessions. Subgraph owners should be consulted about which of their fields are sensitive, and schema tags can mark them so that redaction rules can be generated rather than hand-maintained. Audit requirements follow from the same position. Because the router is the single entry point, it is a natural place to record who accessed which operation and when. For regulated data, that record may be required. But it should be designed as an audit log with its own retention and access controls, not as a side effect of debug logging. Governance without gatekeeping The security practices in this chapter share a common shape. Each moves a decision into the schema or the platform where it can be declared by the owning team and enforced automatically, rather than leaving it to a central reviewer. Authorization requirements are declared with directives. Visibility is declared with @inaccessible and tags. Allowed operations are declared by client builds. The central graph team configures the router and the registry to enforce these declarations, sets defaults that err on the side of safety, and audits periodically. A brief checklist summarizes what a production federated graph should have in place. Subgraphs reachable only from routers, verified by mutual TLS or an equivalent mechanism. Authentication performed at the router, with only explicit headers forwarded. Authorization checks in every reference resolver as well as every root field. Schema-level authorization directives for coarse rules. Subgraph error masking in production. Introspection disabled on public routers. Safelisting for first-party clients, cost limits for third parties. Contracts for distinct audiences, with lint rules requiring audience tags on new fields. Telemetry that does not record sensitive field values or variables. None of these controls is exotic. The difficulty is that in a federated graph, each depends on many teams doing their part consistently, and a single subgraph that skips authorization in its reference resolver or accepts traffic from outside the router undermines the whole. That is why these controls work best when they are built into the shared templates, libraries, and pipelines that domain teams use to create subgraphs, so that the secure path is also the default path. Chapter 8: Migrating from a Monolith Almost no organization builds a federated graph from scratch. They arrive at federation from somewhere else: a large monolithic GraphQL server that has become a bottleneck, a collection of backends-for-frontends that duplicate logic, a REST aggregation gateway, or a schema stitching setup that has grown unwieldy. The question is not whether federation would be a better architecture in the abstract but how to get there from here without stopping feature work, breaking clients, or betting the company on a single cutover. This chapter lays out a migration path that has been used successfully by many organizations. Its central idea is incremental extraction: put a router in front of what already exists without changing anything clients see, then move ownership one domain at a time, using the composition and override mechanisms from earlier chapters to make each move safe and reversible. The technical steps are straightforward. The harder parts, sequencing, team readiness, and knowing when to stop, get equal attention here. Principle: the client never notices The single most important constraint in a federation migration is that clients should not have to change. Their endpoint stays the same. Their queries stay the same. Their responses stay the same. If the migration requires every web and mobile team to update their code at the same time, it will stall, because those teams have their own roadmaps and an old mobile app version cannot be updated at all. Federation makes this constraint achievable because the router presents a single schema at a single endpoint, regardless of how many subgraphs sit behind it. If the supergraph's API schema is identical to the monolith's schema, clients cannot tell the difference. The migration happens entirely behind the router. That requires discipline about the schema. Every field the monolith exposes must continue to exist in the supergraph with the same name, type, nullability, and arguments, throughout the migration. Field moves must preserve semantics exactly. Any schema improvements, such as renaming awkward fields or fixing inconsistent pagination, should be done as separate, additive changes with deprecation periods, not bundled into the migration, because bundling them makes it impossible to tell whether a client break came from the move or the redesign. Phase one: the monolith becomes a subgraph The first step is to turn the existing GraphQL server into a subgraph. In most server libraries, this means adding a federation package, wrapping the schema construction with a function that adds the service and entities fields, and adding a @link declaration to the schema. For a monolith built with Apollo Server, it involves importing buildSubgraphSchema from the @apollo/subgraph package and passing the existing type definitions and resolvers through it. For other libraries the steps are analogous. At this point the monolith exposes the same schema plus the federation fields. Nothing has changed functionally. Next, identify the entities. For each type that future subgraphs will want to extend, typically the core business nouns, add a @key directive and implement a reference resolver. This is often the most laborious part of phase one, because a monolith's types may not have clean identifiers, or the logic to load one by identifier may be buried inside resolvers that load them in other ways. It is also valuable work in its own right, since it forces explicit decisions about entity identity that the monolith may have left implicit. Phase two: a router in front With the monolith operating as a subgraph, compose a supergraph containing only that one subgraph, and deploy a router serving it. The supergraph's API schema should be identical to the monolith's original schema, and a schema diff between them should be empty apart from federation fields that the router does not expose. Route a small percentage of production traffic through the router, using whatever traffic-splitting mechanism the organization already has, such as a load balancer or service mesh. Compare responses, error rates, and latency between the router path and the direct path. The router will add a small amount of latency for parsing, planning, and the extra network hop; measure it and make sure it fits the budget. Increase the router's share of traffic until it carries everything, then remove the direct path. This phase delivers no new capability on its own, and teams are sometimes tempted to skip it and go straight to extracting a domain. That is a mistake. Phase two establishes the router as production infrastructure, proves the deployment and observability setup, sets up the registry and the publish pipeline for the monolith subgraph, and gives the central graph team experience operating the router before the graph becomes complex. Problems found here, such as header propagation errors, timeout misconfiguration, or tracing gaps, are far easier to diagnose with one subgraph than with five. Phase three: extract domains one at a time Now the real migration begins. Each extraction follows the same pattern. A domain team builds a new subgraph that implements a set of fields currently resolved by the monolith. Initially, those fields are marked with @override(from: "monolith", label: "percent(0)") or are left without override, so composition records the monolith as the resolver. The team deploys the new subgraph and publishes its schema. The router now knows about the new subgraph but routes nothing to it for the overridden fields. The team then increases the override percentage in steps, perhaps 1, 5, 25, 50, and 100 percent, watching error rates, latency, and any correctness signals at each step. If something goes wrong, they lower the percentage and traffic returns to the monolith within seconds. Because the change is a schema publish rather than a code deploy, rollback is fast and does not depend on the deploy pipeline. When the new subgraph is serving all traffic for the fields, the monolith's owners remove those fields and their resolvers from the monolith and publish, and the new subgraph's team removes the override directive. The extraction can also move root fields, entire types, and mutations. Root fields and mutations can carry @override just like entity fields. Types that exist only in the domain being extracted move wholesale; types shared with other domains become entities, with the new subgraph contributing its fields and the monolith retaining the rest until they too are extracted. Data must move too, eventually A subgraph that resolves fields by reading the monolith's database is not truly independent. It shares a failure domain and a schema with the monolith, and changes to the database require coordination. In the early stages of a migration, this is often acceptable as a transitional state: the new subgraph reads from the shared database while the team builds its own data store. But it should be recognized as technical debt with a plan to pay it down, because a federated graph whose subgraphs all share one database has distributed only the code, not the ownership. The standard techniques for separating data apply: dual writes with reconciliation, change data capture streaming from the old database to the new store, or a period during which the old database remains the system of record and the new subgraph reads from a replica. These are the same techniques used in any service extraction, and federation neither helps nor hinders them. What federation does provide is a stable client-facing contract that stays fixed while the data moves underneath. Phase four: knowing when to stop Not everything needs to be extracted. Many organizations reach a point where the remaining monolith subgraph contains domains that change rarely, have no natural owner, or are scheduled for retirement. Extracting them would cost effort without improving any team's throughput. A monolith subgraph that remains in the supergraph indefinitely is a legitimate end state, provided it has a clear owner and is not a bottleneck for active domains. The goal of migration is to remove the bottleneck for the domains that change often, not to achieve architectural purity. Teams should regularly ask which domains in the monolith are slowing someone down and extract those, rather than working through a list mechanically. Table 3 summarizes the four phases and what each one establishes. Table 3. Phases of an incremental federation migration. Phase Main change Client impact What it proves 1. Monolith as subgraph Add federation library, keys, reference resolvers None Entities are identifiable 2. Router in front Single-subgraph supergraph behind router None, small latency Router operations and pipeline 3. Extract domains New subgraphs take fields via progressive override None Independent ownership works 4. Stop deliberately Leave stable remnants in monolith None Effort follows real bottlenecks Choosing the first domain The first extraction sets the tone for the whole migration, so the choice of domain matters. Three criteria help. The domain should have a clear owning team that wants to own its subgraph. A team that is enthusiastic about controlling its own part of the graph will push through the inevitable rough edges. A team assigned the work reluctantly will treat it as a chore, and its experience will spread. The domain should be relatively loosely coupled to the rest of the monolith. A domain that adds fields to one or two entities and has few dependencies on other domains is easier to extract and produces a simpler query plan than one whose fields are woven through every type. Reviews, recommendations, and notifications are often good candidates. Core domains such as users or orders, on which everything depends, are usually poor first choices despite their importance. The domain should matter enough that success is visible. Extracting a domain nobody cares about proves little. Extracting one that has been visibly blocked by the monolith's release cycle, and showing that its team can now ship daily, makes the case for continuing. Migrating from REST and other starting points Organizations without a GraphQL monolith face a different starting point. If the existing aggregation layer is a REST gateway or a set of backends-for-frontends, there is no existing schema to preserve and the clients must change to adopt GraphQL at all. The migration is therefore a client migration as well as a backend one. The usual approach is to build the supergraph alongside the existing layer, adding subgraphs domain by domain, and to move client features to GraphQL as the needed parts of the graph become available. Clients run both the old and new API during the transition. This is slower to show results than wrapping a GraphQL monolith, but it allows the schema to be designed well from the start rather than inheriting a monolith's quirks. Declarative integration features can accelerate this path. Apollo Connectors, available since Federation 2.10, let a subgraph schema declare how its fields map to existing REST endpoints using @source and @connect directives, with the router itself making the HTTP calls. A domain team can expose its existing REST API as part of the supergraph without writing or operating a new service, and later replace the connector with a native subgraph if needed. Cosmo offers a related path for gRPC services through its Connect feature, and Grafbase supports extensions that integrate non-GraphQL sources. These tools do not remove the need for good schema design, since a connector that mirrors a poorly designed REST API produces a poorly designed graph, but they lower the cost of getting started. Organizations coming from schema stitching can usually convert stitched services to subgraphs one at a time, since many stitching setups already have the notion of a type merged across services by key. The conversion involves replacing stitching configuration with federation directives in each service and replacing the stitching gateway with a federation router once all services are converted. From Federation 1 to Federation 2 Organizations that adopted federation early may still have subgraphs written for Federation 1, which used extend type and an @extends directive to mark types defined elsewhere and treated all fields as shareable by default. Federation 2 composition can include Federation 1 subgraphs, converting their schemas automatically during composition, so a supergraph can be moved to Federation 2 composition before every subgraph is updated. Each subgraph can then be updated individually by adding a @link declaration, removing the obsolete extend keywords where appropriate, and adding @shareable where fields are genuinely resolved by multiple subgraphs. Composition errors after the switch typically reveal fields that were accidentally shared under Federation 1's permissive defaults, which is useful information in itself. The organizational migration The technical migration is only half the work. Every extracted domain needs a team that is prepared to operate a production GraphQL service: to be on call for it, to understand federation directives and reference resolvers, to handle schema checks in their pipeline, and to design client-facing schema. Teams that have only ever built internal REST services may need training and support. The central graph team can make this much easier by providing a subgraph template for each major language used in the organization. A good template includes the federation library correctly configured, a DataLoader-based reference resolver pattern, authentication header handling, OpenTelemetry tracing with context propagation, health checks, the schema check and publish steps in the pipeline, and network policy that restricts access to routers. A team starting from such a template inherits the organization's best practices on day one, and many of the problems described in earlier chapters, such as unbatched reference resolvers and directly reachable subgraphs, simply do not arise. Migration also changes on-call responsibilities. Before federation, a broken product page is the GraphQL team's problem. After, it might be the router team's, the catalog team's, or the reviews team's, depending on which component failed. Distributed tracing makes the diagnosis possible, but the incident process must also change: alerts routed to the owning team based on which subgraph is failing, runbooks for router-level problems, and clear escalation paths. Organizations that migrate the architecture without migrating the incident process find that the old central team still gets paged for everything, which defeats a large part of the point. A realistic timeline Migrations take longer than planned. Phase one for a large monolith, identifying entities and implementing reference resolvers, can take weeks. Phase two, establishing the router in production, typically takes a few weeks including traffic shifting. Each domain extraction in phase three might take one to three months depending on the domain's size and the amount of data separation involved. A migration of a dozen domains can therefore run for a year or more. That is acceptable, because the incremental approach delivers value along the way. Each extracted domain is immediately able to ship independently. The migration can pause for a quarter when priorities shift and resume later without leaving anything broken. The worst outcome of an incremental migration is a partially federated graph that works correctly and is better than where it started. The worst outcome of a big-bang migration is a rewrite that never ships, and that outcome is common enough that it should be avoided deliberately. Chapter 9: Choosing the Stack For most of federation's history, choosing a stack was not much of a decision. Apollo invented the specification, built the gateway, ran the registry, and wrote most of the documentation. Organizations that wanted federation used Apollo, and the main question was which plan to buy. That is no longer true. By late 2026 there are several mature routers, several registries, and an emerging vendor-neutral specification, and the differences between them, in licensing, performance, operational model, and cost, are large enough to matter. This chapter compares the main options as they stand, explains the licensing questions that have shaped the market, describes the open specification effort and what it means for the future, and closes with the situations in which federation is the wrong choice altogether. Product details change quickly, and any organization making a decision should verify current terms and features directly with vendors. The structural points, however, are likely to hold. The components to choose A federated stack has four parts that can, in principle, be chosen independently. The subgraph libraries implement the federation subgraph protocol in each language the organization uses. These are largely interchangeable across vendors, because they implement the same service and entities protocol and the same directives. Apollo's @apollo/subgraph package for JavaScript remains under the MIT license. Most other subgraph libraries are maintained by their language communities under permissive licenses. The composition implementation turns subgraph schemas into a supergraph. Apollo's composition library, Cosmo's, and Hive's are separate implementations of the same rules, and they publish compatibility results against one another. The registry stores schemas, runs checks, keeps history, and distributes composed supergraphs to routers. The router executes queries at runtime. In practice, the registry and router are often bought or adopted together, because the registry's distribution mechanism, usage reporting format, and check features are tied to a particular router. But mixing is possible. Hive's registry can feed Apollo's router. Several routers can run from a supergraph file produced by any compliant composition. Understanding this separability helps avoid lock-in. The main options Apollo GraphOS Apollo remains the reference implementation and the most widely deployed stack. Its router, written in Rust, is in its second major version, with version 1 having reached end of support on March 31, 2026. Since early 2026 Apollo has followed a support policy with long-term support releases roughly every six months, each supported for nine months, or twelve for customers with premier support. The Federation specification itself reached version 2.15 in July 2026, a release in which composition was rewritten in Rust. Apollo's distinguishing strengths are maturity and breadth. It originated the specification, so new federation features generally appear in Apollo's tooling first: progressive override, authorization directives, demand control, Connectors for REST integration, and cache tagging all arrived in Apollo before or alongside competitors. GraphOS Studio's schema explorer, operation checks, and usage insights are well developed. The documentation is extensive. Its distinguishing constraints are licensing and pricing. Apollo's router and its Federation 2 composition libraries are licensed under the Elastic License v2. Many of the router's production-critical capabilities are licensed features that require a GraphOS plan, including subscriptions, JWT authentication, the authorization directives, persisted query safelisting, request limits, demand control, external coprocessors, distributed query plan caching, and Redis-backed response caching. In April 2025 Apollo introduced a free GraphOS plan, replacing a previous ninety-day enterprise trial, that includes the registry, the schema explorer, and basic insights, but caps self-hosted routers at sixty requests per minute and limits organizations to three members. That plan is suitable for prototyping and evaluation, not production. Organizations should expect to budget for a paid plan if they run Apollo's router in production with the features most federated graphs need. WunderGraph Cosmo WunderGraph launched Cosmo in September 2023 as a complete federation platform: router, control plane, studio, and command-line tool, all under the Apache License 2.0. The router is written in Go and supports Apollo Federation version 1 and version 2 subgraphs, so organizations can point it at existing subgraphs without changing them. Cosmo's strengths are its permissive license, which allows self-hosting the entire platform including the control plane with no feature gates in the open-source code, and a set of distinctive features. Cosmo Streams, formerly event-driven federated subscriptions, lets the router consume events directly from Kafka, NATS, or Redis to drive subscriptions, which removes the need for subgraphs to hold long-lived connections. Cosmo Connect provides a path for integrating gRPC services. Custom router modules are written in Go and compiled into the router binary. WunderGraph also offers Cosmo as a managed cloud service with commercial support for organizations that do not want to operate the control plane themselves. Its constraints are those of a younger ecosystem: a smaller community than Apollo's, fewer third-party integrations, and the need, when self-hosting, to operate the control plane's dependencies, which include databases and an analytics store. The Guild's Hive The Guild, a collective of open-source GraphQL maintainers responsible for widely used libraries such as GraphQL Code Generator, GraphQL Mesh, and graphql-tools, maintains Hive, a schema registry and federation platform under the MIT license. Hive's registry supports Apollo Federation, runs composition and operation checks, collects usage data, and distributes composed supergraphs through a CDN. It can be used as a hosted service or self-hosted. Hive offers two runtimes. Hive Gateway is written in JavaScript, runs on Node.js, Bun, Deno, and edge runtimes such as Cloudflare Workers, and integrates naturally with JavaScript tooling and plugins. In November 2025 The Guild announced a Rust query planner for Hive Gateway to improve performance. Hive Router, announced in September 2025, is a separate router written in Rust, also MIT licensed, designed for teams that want maximum throughput from a systems-language runtime. The Guild reports that Hive Router passes all test cases in its federation compatibility audit, an open test suite that The Guild maintains and uses to compare gateways from several vendors. Hive's strengths are its fully permissive licensing, its registry's ability to work with routers from other vendors, and its maintainers' long involvement in the GraphQL ecosystem. Its constraints, as with Cosmo, are a smaller commercial footprint than Apollo's and, for Hive Router, a newer codebase with less production history. Other options Grafbase offers a gateway written in Rust that supports Apollo Federation and partially implements the GraphQL Foundation's composite schemas specification, along with an extension system for integrating non-GraphQL data sources such as databases and REST or gRPC services. ChilliCream, the company behind the Hot Chocolate GraphQL server for .NET, offers Fusion, a gateway built on ASP.NET Core that implements the composite schemas specification and, as of its version 16.5 released in July 2026, also supports Apollo Federation subgraphs in the same graph. Fusion is a natural choice for organizations whose services are predominantly .NET. Several API management platforms, such as Tyk, have added federation support as well. Table 4 summarizes the principal options as of late 2026. Licensing and runtime language are from each vendor's public documentation; organizations should confirm the details current at the time of their decision. Table 4. Principal federation platforms as of late 2026. Platform Router language License Registry Notable distinction Apollo GraphOS Rust Elastic License v2 GraphOS (hosted) Reference implementation, broadest feature set WunderGraph Cosmo Go Apache 2.0 Cosmo control plane Whole platform self-hostable, broker-driven subscriptions The Guild Hive JavaScript and Rust MIT Hive (hosted or self-hosted) Registry works with other routers ChilliCream Fusion C# (.NET) See vendor Nitro Composite schemas spec plus Apollo Federation Grafbase Rust See vendor Grafbase platform Extensions for non-GraphQL sources Why licensing became a deciding factor Licensing is not usually the first concern in an architecture decision, but in federation it has become one, for reasons worth understanding. When Apollo released Federation 2 in 2022, it moved the federation composition libraries and its gateway to the Elastic License v2. Its router, released the same year, was under that license from the start. Earlier federation packages had been MIT licensed, and Apollo maintained them through the end of 2022. The Elastic License v2 is not an open-source license under the Open Source Initiative's definition. It permits free use, modification, and redistribution, including for internal commercial use, but forbids providing the software to others as a hosted or managed service, forbids circumventing license-key functionality, and forbids removing licensing notices. For most companies running their own supergraph for their own products, the Elastic License itself imposes no meaningful restriction. They are not offering the router as a service to third parties. The practical constraint comes instead from the commercial model that sits alongside it: the open-source router core runs without a license, but the features that most production federated graphs need, such as authentication, authorization directives, safelisting, subscriptions, and coprocessors, are gated behind a GraphOS license. An organization adopting Apollo's stack is therefore, in effect, adopting a commercial product with an open-core component, and should evaluate it as such. Competitors responded by offering permissively licensed alternatives. Cosmo's Apache 2.0 license and Hive's MIT license allow self-hosting the full feature set, offering the software as a service, and forking it if necessary. For organizations with strong preferences for open source, regulatory requirements that make dependence on a vendor's cloud for license checks difficult, or a need to run in air-gapped environments, this is significant. Apollo does offer offline licenses for enterprise customers in disconnected environments, but on an as-needed basis. Licensing should be weighed alongside other factors, not in isolation. A permissive license does not by itself make a platform cheaper to operate; self-hosting a control plane has its own costs in infrastructure and staff time. A commercial license does not by itself make a platform risky; it brings vendor support and a funded roadmap. The point is to know what is being bought. The open specification The most important long-term development in federation is not any one product but the effort within the GraphQL Foundation to standardize it. The Composite Schemas Working Group was announced in May 2024 as a subcommittee of the GraphQL Working Group, with participation from Apollo, ChilliCream, The Guild, and other companies and individual contributors. Its goal is a vendor-neutral specification for composing a single GraphQL schema from many source schemas, drawing on the strongest ideas from Apollo Federation, Fusion, schema stitching, and other approaches. The draft specification uses its own vocabulary. Individual services expose source schemas, which compose into a composite schema. Instead of a special _entities field, a source schema declares ordinary query fields as lookups using a @lookup directive, which tells the gateway it can fetch an entity by key through that field. Directives such as @key, @shareable, @provides, @external, @override, and @inaccessible carry over with similar meanings, while @require handles field-level dependencies on data from other source schemas and @is maps lookup arguments to entity fields. The use of ordinary query fields for entity lookup is one of the most significant departures from Apollo Federation, because it means a source schema is a perfectly normal GraphQL schema that can be queried directly, with no special protocol fields. As of late 2026 the specification is still pre-release and actively evolving, with the working group meeting regularly. ChilliCream's Fusion implements it fully, Grafbase partially, and other vendors have expressed support. Some participants have begun referring to it simply as the GraphQL Federation specification. The practical implications for an organization starting today are modest but real. Apollo Federation remains the dialect with by far the broadest support, and choosing it is not a bet against the standard; Apollo participates in the working group, and gateways such as Fusion already serve Apollo Federation and composite schema subgraphs side by side. But teams should design schemas in ways that will translate easily: clean entity keys, minimal reliance on vendor-specific directives, and lookup-style root fields such as productById that would map naturally onto @lookup. When the specification stabilizes, migration should then be a matter of tooling rather than redesign. Making the decision Several questions sort most organizations toward one option or another. What is the expected traffic and feature profile? A graph that needs subscriptions, fine-grained authorization directives, safelisting, and coprocessor extensions from day one will need Apollo's paid tiers or a permissively licensed alternative that includes those capabilities. What are the constraints on hosting? Organizations that must run entirely within their own infrastructure, or in disconnected environments, should weigh the ability to self-host the registry heavily. What languages and runtimes does the organization use? A predominantly .NET shop may find Fusion's integration compelling. A JavaScript-heavy organization may value Hive Gateway's plugin model. An organization that prefers to extend its router in Go may prefer Cosmo. How much does the organization value ecosystem breadth and vendor support? Apollo's documentation, community, and support are the most extensive. That counts for a lot when teams are learning federation for the first time. What is the exit plan? Every option uses the same subgraph protocol, so subgraphs are portable. Registries, checks, and router extensions are less portable. Organizations should prefer configurations that keep custom logic in subgraphs or in standards-based extension points where possible, and should test that their supergraph can run on at least one alternative router, even if they never plan to switch. When not to federate Federation solves a specific problem: the coordination cost of a single client-facing API shared by many teams. Where that problem does not exist, federation adds complexity without corresponding benefit. A single team owning both the API and all its backends does not need federation. A well-structured monolithic GraphQL server, with clean modules per domain, will be simpler to build, faster at runtime, and easier to debug. An organization with two or three backend teams and a GraphQL server that is not yet a bottleneck probably does not need federation either. A modular monolith, in which each team owns a module of the schema and its resolvers within one codebase, with code ownership rules enforcing boundaries, can provide much of the ownership benefit with none of the distributed systems cost. Federation can come later, and the migration path described in Chapter 8 starts precisely from such a server. An organization whose underlying services are deeply entangled, sharing databases and calling each other in cycles, should be cautious. Federation will expose the entanglement in its query plans and composition errors, which may be useful as a diagnostic, but it will not fix the underlying design. And an organization whose clients are primarily other services rather than user-facing applications may find that GraphQL itself is not the right choice for those interactions, and that service-to-service protocols such as gRPC, combined with an event backbone, serve better. Federation is for unifying a client-facing API, and its benefits depend on having clients that value a single flexible schema. For organizations that do face the problem federation addresses, many teams, many clients, and a central API layer that has become a bottleneck, the choice of stack matters less than the practices described in earlier chapters. A well-designed graph with clear ownership, continuous composition checks, a well-run router, and sound security will succeed on any of the platforms in this chapter. A poorly designed one will struggle on all of them. Conclusion The usual way to describe federation is as a technique for splitting a GraphQL schema across services. The chapters of this booklet suggest a different description, and the difference has consequences. Federation turns the client-facing API from an artifact that one team builds into a protocol that many teams speak. The supergraph schema is not written by anyone. It is the output of a process, composition, applied continuously to contributions from every domain, and its quality is determined by the rules of that process and the discipline of its participants rather than by the skill of a central author. Once the API is understood this way, several things follow that are not obvious at the outset. Invest in the contract before the runtime Organizations adopting federation tend to begin with the router. It is the visible component, it has benchmarks, and choosing one feels like progress. But the evidence of the preceding chapters points the other way. The router is the part of the system with the least organizational leverage. Every serious router can plan and execute federated queries competently. What separates graphs that work from graphs that stall is the machinery around composition: a registry that every subgraph publishes to, composition and operation checks in every team's pipeline, lint rules that encode design conventions, and usage data good enough to make removal decisions on evidence. An organization starting out should therefore set up the registry and the check pipeline before it extracts its first domain, and should judge its readiness for federation by whether a domain team can propose a change and learn within minutes whether it would break composition or any client. If that loop does not exist, adding subgraphs will add coordination costs rather than remove them, and someone will end up manually guarding the graph, which is exactly the role federation was meant to retire. The central team changes job, it does not disappear A common expectation is that federation eliminates the central API team. It does not. It changes that team's job from writing the API to running the platform on which others write it. The difference is the difference between a gate and a road. A gate approves each change. A road makes good changes easy and bad ones hard, through templates, defaults, checks, and tooling. This has staffing implications that organizations often miss. The skills needed to run a federation platform, router operations, registry configuration, pipeline tooling, schema design coaching, and incident coordination across subgraphs, are different from those needed to write resolvers. A central team that is simply relabeled, without changing what it does, will continue to act as a gate, and domain teams will continue to wait on it. The test proposed in Chapter 4 is the right one to apply repeatedly: can a routine domain change reach production without anyone on the central team noticing? If not, the platform is not finished. Boundaries are a performance decision Federation is usually sold on organizational grounds and debated on performance grounds, and the two arguments are normally kept separate. The chapters on entities and on the router show that they are the same argument. The latency of a federated query is set by the length of its critical path, and the length of its critical path is set by how many subgraph boundaries the query crosses in sequence. Boundaries drawn along genuine business capabilities, where the common client queries mostly stay within one or two domains, produce short plans and fast queries. Boundaries drawn along database tables or service inventories produce long plans and slow queries. The practical consequence is that query plans for the most important client operations deserve the same scrutiny as database query plans in a performance-sensitive application. They should be reviewed when boundaries are designed, rechecked after significant schema changes, and treated as evidence when deciding whether a boundary is in the wrong place. A graph whose hot paths routinely cross four boundaries is telling its owners something about their domain model, and the fix is usually a boundary change, made safely with progressive override, rather than a faster router. Security is a property of the weakest subgraph In a single server, security is designed once. In a federated graph, it is the product of every team's choices, and it is only as strong as the weakest subgraph. A single reference resolver that skips an authorization check, or a single subgraph that accepts traffic from outside the router, can expose data that every other team has protected carefully. Federation does not create this risk from nothing, since the same mistakes are possible in a monolith, but it distributes the opportunity to make them across many more people. The response cannot be central review of every resolver, which would rebuild the gate. It has to be secure defaults built into the platform: subgraph templates that enforce network isolation and authorization in reference resolvers from the first commit, schema directives that make access requirements visible and enforced at the edge, safelisting for first-party clients, and audience tags required by lint. Organizations that treat security as each team's individual responsibility, without providing these defaults, should expect it to be uneven. What remains unsettled Several questions remain genuinely open, and anyone building a federated graph today should plan for them rather than assume answers. The first is standardization. The GraphQL Foundation's composite schemas specification promises a vendor-neutral foundation, and its use of ordinary lookup fields instead of a special entities protocol is a real improvement. But it is not yet final, Apollo Federation remains the dialect most tools speak, and the relationship between the two over the coming years will shape every vendor's roadmap. The safe course is to design schemas that would translate cleanly, with simple keys and lookup-style root fields, and to avoid deep dependence on features that exist in only one dialect. The second is the commercial structure of the market. The split between Apollo's open-core model under the Elastic License and the permissively licensed alternatives from WunderGraph and The Guild has produced healthy competition, including Apollo's free plan in 2025 and rapid feature development everywhere. It has also produced uncertainty. Organizations should keep their subgraphs portable, keep custom logic out of proprietary router extension points where they can, and verify periodically that their supergraph runs on more than one router. The third is caching. Federation makes it easy to assemble a response from many sources and hard to know when that response has become stale. Entity-level caching with tag-based invalidation is a promising direction, but it requires subgraphs to emit invalidations reliably and routers to apply them consistently, and the practice is young. Organizations with strict freshness requirements should be conservative here. The fourth is the changing nature of clients. Federated graphs were designed for human-facing applications with known, relatively stable queries. Increasingly, graphs are also queried by automated agents that compose operations on the fly, which strains assumptions behind safelisting and makes cost limits and field-level authorization more important. How federated graphs should be exposed to such consumers, and how cost and access should be governed for them, is not yet a solved problem. What to do differently For an organization considering federation, the most useful change of perspective is to stop asking which router to use and start asking whether its teams are ready to own their parts of a shared contract. That means asking whether domain boundaries are clear enough that each type has an obvious owner, whether teams have the skills to design client-facing schemas, whether there is a group willing to run the platform as a road rather than a gate, and whether the incident process can route problems to the subgraph that caused them. Where the answers are yes, federation will remove a real bottleneck and let a growing organization keep a single, coherent API. Where they are no, the first work is organizational, and the technology can wait. For an organization already running a federated graph, the most useful change is to look at it through the lens of ownership rather than infrastructure. Which entities have no clear origin? Which subgraphs are chatty with each other? Which fields are shareable without justification? Which reference resolvers lack authorization checks? Which hot query plans cross too many boundaries? Each of these is a symptom of an ownership decision that was never made explicitly. Making it explicit, and then using the migration tools federation provides to act on it, is how a federated graph keeps pace with the organization it serves. The single endpoint that clients see is federation's promise. The distributed ownership behind it is federation's price, and also its point. Organizations that pay that price deliberately, with contracts enforced by machines, boundaries drawn along capabilities, and platforms built for the teams that use them, get an API that grows with them. Organizations that pay it by accident get a distributed monolith. The difference is almost never the router. Glossary Composition. The build-time process that merges subgraph schemas into a supergraph schema and verifies that every field can be resolved through some path across the subgraphs. Entity. A type identified by a key that more than one subgraph can contribute fields to, such as a product, user, or order. Key. The field or fields, declared with @key, that uniquely identify an entity and allow the router to look it up in any subgraph that defines it. Operation check. A registry check that compares a proposed schema change against recorded client traffic and reports which operations it would break. Progressive override. Gradually shifting resolution of a field from one subgraph to another by percentage, using @override with a label argument. Query plan. The router's program for executing a client operation: which subgraphs to call, in what order and grouping, and how to merge their results. Reference resolver. The subgraph code that takes an entity representation containing key fields and returns the full entity; in many JavaScript libraries it is named __resolveReference. Representation. A small object containing an entity's type name and key fields, sent by the router to a subgraph's _entities field. Router. The runtime component that accepts client operations against the supergraph, plans them, calls subgraphs, and assembles responses. Schema registry. The service that stores published subgraph schemas, composes the supergraph, runs checks, keeps history, and distributes supergraphs to routers. Subgraph. A GraphQL service owned by one team that defines and resolves part of the supergraph and implements the federation subgraph protocol. Supergraph. The single composed graph presented to clients; also the annotated schema document from which routers derive both query plans and the client-facing API schema. Further Reading Apollo Federation documentation and changelog. apollographql.com/docs/graphos/schema-design/federated-schemas. The reference for Federation 2 directives, composition rules, and the version history, including the features introduced in each release from 2.0 through 2.15. Apollo GraphOS Router documentation. apollographql.com/docs/graphos/routing. Covers router configuration, query planning, traffic shaping, telemetry, and the list of licensed features. The licensing FAQ at apollographql.com/trust/licensing explains the Elastic License v2 terms. WunderGraph Cosmo documentation. cosmo-docs.wundergraph.com. Documentation for the Apache 2.0 licensed Cosmo router, control plane, and CLI, including self-hosting guides and Cosmo Streams. The Guild, Hive documentation. the-guild.dev/graphql/hive/docs. Documentation for the Hive schema registry, Hive Gateway, and Hive Router, along with The Guild's open federation gateway compatibility audit. GraphQL Composite Schemas Specification. github.com/graphql/composite-schemas-spec. The working draft of the vendor-neutral specification being developed by the GraphQL Foundation's Composite Schemas Working Group. The announcement post on graphql.org from May 2024 explains the group's goals. ChilliCream Fusion documentation. chillicream.com/docs/fusion. A gateway implementation of the composite schemas specification with Apollo Federation interoperability, useful for seeing how the draft standard works in practice. Netflix Technology Blog, "How Netflix Scales its API with GraphQL Federation." netflixtechblog.com, November 2020. A two-part account of Netflix studio engineering's move from a single GraphQL API to a federated architecture, describing the organizational bottleneck that motivated it. Marc-André Giroux, Production Ready GraphQL (2020). A practitioner's book on GraphQL schema design, performance, and security. It predates Federation 2 but its guidance on schema design and API evolution applies directly to subgraph authors. Eric Evans, Domain-Driven Design (Addison-Wesley, 2003). The origin of the bounded context concept used in this booklet to argue for drawing subgraph boundaries along business capabilities. Matthew Skelton and Manuel Pais, Team Topologies (IT Revolution, 2019). A framework for organizing software teams, including stream-aligned and platform teams, that maps closely onto the relationship between domain subgraph teams and a central graph platform team. Sam Newman, "Backends For Frontends" (samnewman.io, 2015). The essay that popularized the BFF pattern, useful background for understanding the problem federation addresses and why per-client aggregation layers alone did not solve it. Melvin E. Conway, "How Do Committees Invent?" Datamation, April 1968. The original statement of Conway's law, the observation that system designs mirror the communication structures of the organizations that produce them. Hashtags: #GraphQLFederation #GraphQL #FederatedGraphQL #ApolloFederation #ApolloFederation2 #SupergraphArchitecture #GraphQLSubgraphs #MicroservicesArchitecture #APIScalability #DistributedAPIs #APIArchitecture #GraphQLRouter #GraphQLGateway #SchemaComposition #SchemaRegistry #GraphQLSchemaDesign #SchemaGovernance #DomainDrivenDesign #BoundedContexts #EntityResolution #GraphQLResolvers #QueryPlanning #DistributedQueryExecution #GraphQLPerformance #DataLoader #APISecurity #GraphQLObservability #ContinuousIntegration #IndependentDeployment #APIManagement