Download the Book (PDF): Introduction In January 2022 Moxie Marlinspike, the cryptographer who created the Signal messaging protocol, minted a non-fungible token as an experiment. Anyone who looked at it on OpenSea saw one picture. Anyone who looked at it on Rarible saw a different one. Anyone who bought it and opened it in a crypto wallet saw a large emoji of a pile of excrement. Nothing about the token had changed between those three views. The token was the same entry in the same contract on the same blockchain. What changed was the answer returned by an ordinary web server that Marlinspike controlled, which looked at who was asking and served a different file to each. A few days later OpenSea removed the token from its marketplace, and, more revealingly, it then vanished from the wallet on his own phone, because the wallet was not reading the blockchain at all; it was asking OpenSea's servers what he owned. The stunt was a joke with a serious point, and Marlinspike made the point plainly: the token contained a URL, and there was no cryptographic commitment to what that URL would return. Whoever controls the server, whoever buys the domain name when it lapses, or whoever breaks into the machine can change the image, the title and the description of the "unique digital asset" whenever they like. The blockchain faithfully records who owns the token. It has no opinion about what the token is a token of. That gap is the subject of this book. Its argument is simple to state and surprisingly demanding to act on. A non-fungible token secures exactly as much as its chain of references commits to cryptographically, and no more. The ledger entry recording ownership is genuinely robust: it is replicated across thousands of machines and cannot be altered without the owner's key. Everything that makes the token worth owning, though, lives at the end of a chain of pointers, from contract to metadata document to image or video file. Each link in that chain is either a hash, which binds it to exact bytes, or a location, which binds it only to whoever controls that place. A hash gives integrity: it guarantees that the bytes you fetch are the bytes the creator committed to. It does not give availability: nobody is obliged to keep those bytes anywhere. Durable digital ownership therefore requires two separate things done deliberately. Every link from ledger to file must be content-addressed, and someone, identifiable and funded, must keep the content alive. Why the architecture matters The NFT market has been through a speculative boom and a long bust, and much public discussion of the subject has been about prices, celebrities and fraud. This book is about none of those. It is about the machinery, because the machinery determines what a buyer has actually bought, and the answer is frequently less than the buyer assumes. When someone buys an NFT, they acquire control of an entry in a smart contract: a small program on a blockchain, usually Ethereum or a network compatible with it, which keeps a table mapping token identifiers to owner addresses. That control is real and enforceable in the narrow sense that nobody can move the token without the owner's private key or an approval the owner granted. The token also exposes a function that returns a string, typically a URL, pointing to a small JSON document. That document names the asset, describes it and contains another URL pointing to the image or media file. The media file is what the collector looks at, displays and talks about. In many collections, especially those launched quickly in 2021, the JSON document and the image both sat on a company's web server or a commercial cloud bucket. If the company stopped paying its hosting bill, the token would continue to exist and continue to be owned, but it would point at nothing. There is a better way to build this, and it is well understood. The InterPlanetary File System, or IPFS, names data by a cryptographic hash of its contents rather than by the server that holds it. An address of the form ipfs:// followed by a content identifier cannot silently point to different bytes, because changing a single bit of the file changes its identifier. Used properly, IPFS closes the integrity gap Marlinspike exploited. It does not, however, close the availability gap, and a great deal of confusion in the NFT world comes from treating the two as one. Content on IPFS persists only as long as at least one node chooses to keep it. "It's on IPFS" is a statement about naming, not about storage. What this book covers The book moves from the ledger outward to the file, and then back again to the decisions a builder or buyer has to make. Chapter 1 establishes what a token is at the level of data structures: a mapping in contract storage, an event log and a set of rules about who can change them. It separates the three things people mean by "owning an NFT" (control of the ledger entry, access to the content, and legal rights in the work) and shows that only the first is enforced by the chain. Chapter 2 examines ERC-721, the standard that defined the non-fungible token on Ethereum in 2018: its required functions, its approval model, its "safe" transfers and the receiver check behind them, its interface-detection mechanism and the optional metadata extension that introduced tokenURI. Each design decision is read for what it protects and what it leaves open. Chapter 3 turns to ERC-1155, the multi-token standard that came out of game development. It holds many token types in one contract, moves them in batches and treats fungibility as a matter of supply rather than kind. Its metadata scheme, with a single URI template and a substituted identifier, has direct consequences for how assets can be stored. Chapter 4 opens the metadata layer itself: the JSON document, the conventions marketplaces built on top of the bare standard, and the chain of indirection from contract to file. It lays out the hosting options as a series of trust assumptions. Chapter 5 is about failure. It catalogues the ways the reference chain breaks: lapsed servers and domains, mutable base URIs controlled by an owner key, swap-after-reveal tricks, gateways that disappear, and indexers that become the de facto source of truth. Chapter 6 explains content addressing and IPFS from first principles: hashing, content identifiers and their encodings, Merkle DAGs, how files are chunked and linked, how peers find and exchange blocks, and what an HTTP gateway does and does not guarantee. Chapter 7 confronts persistence. It covers pinning, the economics of pinning services, Filecoin's storage deals, Arweave's one-time endowment model, and fully on-chain storage, along with the honest costs of each. Chapter 8 assembles all of this into a working architecture: how to write a contract whose metadata cannot be swapped, how to publish and verify content, how to arrange redundant persistence, and how a collector can audit any token before buying it. The conclusion draws out what follows from the argument for creators, platforms and owners. A note on scope The book concentrates on Ethereum and the many networks that run the Ethereum Virtual Machine, because that is where the ERC-721 and ERC-1155 standards live and where most NFT architecture has been worked out. Other chains, and Bitcoin's Ordinals inscriptions, appear where they illuminate a design choice, particularly about storing data directly on a ledger. Code is described rather than reproduced at length; where function names and event signatures matter, they are given exactly, because precision is the point. Prices and market history appear only where they help explain why the architecture took the shape it did. Readers do not need to be programmers. They do need patience with a small number of technical ideas, chief among them the cryptographic hash: a function that turns any input into a short fixed-length fingerprint such that changing the input in any way produces an unrelated fingerprint, and such that nobody can feasibly find two inputs with the same one. Almost everything in this book follows from what hashes can and cannot do. A hash can prove that a file is the file. It cannot make anyone keep it. Chapter 1: What a Token Actually Is Strip away the marketplaces, the profile pictures and the language of collecting, and a non-fungible token is a row in a table. The table lives inside a smart contract, the contract lives at an address on a blockchain, and the row says that a particular number belongs to a particular account. Everything else is built on top of that row or around it. To understand what NFT architecture can and cannot guarantee, it is necessary to begin with the row: where it is stored, who can change it, what record is left when it changes, and what it does not contain. The ledger entry Ethereum is, at the level that matters here, a shared computer whose state is agreed upon by every participating node. That state includes accounts, each holding a balance of ether, and contracts, each holding code and a private area of persistent storage. Contract storage is organised as a very large key-value store in which both keys and values are 32-byte words. High-level languages such as Solidity hide this behind familiar constructs. When a developer declares a mapping from token identifier to owner address, the compiler arranges for each entry to occupy a storage slot whose location is derived by hashing the token identifier together with the mapping's position in the contract. A token, then, is a number, usually a 256-bit unsigned integer, and owning it means that the storage slot associated with that number contains your address. The contract also keeps a second mapping from each address to a count of the tokens it holds, so that a balance can be read without scanning every token, and further mappings for approvals, which are discussed in the next chapter. That is all the core data there is. A collection of ten thousand tokens is a contract with ten thousand populated slots in its owner mapping. Changing a slot requires a transaction: a message, signed with an account's private key, that calls a function on the contract. The contract's code checks whether the caller is entitled to make the change, for example whether the caller is the current owner of the token, and either updates storage or reverts, undoing everything. Once a transaction is included in a block, and that block is buried under enough subsequent blocks, the change is effectively permanent. Every full node has executed the same code on the same inputs and arrived at the same storage. To forge a transfer, an attacker would have to produce a valid signature for the owner's key, which is computationally infeasible, or rewrite the chain's history, which on Ethereum since its move to proof of stake in September 2022 would require controlling and sacrificing a very large amount of staked ether. The same standards and contracts also run on many other networks that execute Ethereum's virtual machine, including rollups such as Arbitrum, Optimism and Base, and independent chains such as Polygon's. The token code can be identical; the security of the ledger entry is not. A rollup inherits much of its security from Ethereum but typically relies on a single sequencer to order transactions and on upgradeable bridge contracts whose administrators could, in principle, alter its rules. An independent chain relies on its own validators. A token is exactly as durable as the ledger it lives on, and "on the blockchain" names a family of systems with very different guarantees. This book uses Ethereum as its reference point, and the arguments about content apply on every one of these networks unchanged. This is the part of the system that justifies the excitement. The ownership record is public, verifiable by anyone running a node, independent of any single company and resistant to tampering. It can be read and written by any other program on the same chain, which is why a token minted by one project can be listed on a marketplace built by another, lent against in a lending protocol built by a third and displayed in a wallet built by a fourth, without any of them asking permission. Storage is expensive, and that cost shapes every architectural decision that follows. On Ethereum, writing a 32-byte word into a previously empty storage slot costs 20,000 units of gas, with a surcharge on top when the slot is first touched in a transaction. Storing one kilobyte of data as raw storage therefore costs on the order of 640,000 gas, a meaningful fraction of a whole block's capacity. Whatever the price of gas at a given moment, the conclusion has always been the same: the chain is a place to keep small, valuable facts, such as who owns token 4,071, and not a place to keep a high-resolution image. The image had to go somewhere else, and the token had to point to it. The rest of this book is, in a sense, about the consequences of that pointer. Events and the history of a token Contract storage records the present. It says who owns token 4,071 now, not who owned it last year. History lives in a different structure: the event log. When a contract executes, it can emit events, which are structured records attached to the transaction receipt. Events are not readable by other contracts and do not occupy contract storage, so they are much cheaper than storage writes, but they are part of the chain's permanent record and can be retrieved by anyone with access to a node. The NFT standards require contracts to emit specific events whenever ownership changes. Under ERC-721, every transfer, including the creation of a token and its destruction, emits a Transfer event naming the previous owner, the new owner and the token identifier. Creation is signalled by a transfer from the zero address, destruction by a transfer to it. These events are how the outside world sees NFTs. A wallet that wants to show a user their tokens does not, in practice, call ownerOf on every token in every contract that exists; there are far too many. Instead, indexing services read the event logs of the entire chain, reconstruct who owns what and serve the result through an ordinary web API. Marketplaces do the same. The convenience is enormous and the trade-off is easy to miss: the user is now trusting the indexer's database, not the chain. Marlinspike's token disappeared from his wallet not because anything happened on Ethereum but because the wallet asked OpenSea what he owned and OpenSea had decided to stop telling it. This distinction, between the authoritative state held by the chain and the convenient view assembled by an intermediary, recurs at every layer of NFT architecture. At the ownership layer it is mostly harmless, because anyone can check the indexer's answer against the chain. At the content layer, as later chapters show, it can be decisive. Contracts are programs, not certificates It is tempting to think of an NFT contract as a registry with fixed rules, like a land title office with a published procedure. It is better to think of it as a program written by the project's developers, which may or may not do what the standard describes, and which may or may not contain functions beyond the standard. The ERC standards define an interface: a set of function signatures and events, together with prose describing how they should behave. A contract that implements the interface can be used by any application that understands the standard. Nothing enforces the prose. A contract can claim to be an ERC-721 token while containing a function that lets the deployer move any token at will, or pause all transfers, or burn tokens from other people's wallets. Such functions are sometimes legitimate, for example a pause mechanism for responding to a discovered bug, and sometimes they are the mechanism of a fraud. Either way, they are part of what the buyer owns. A token's guarantees are the contract's code, read in full, not the standard's description. The same is true of the pointer to content. The standard requires only that a function exist which returns a URI for a given token. How that URI is computed is up to the contract. It might be a constant string stored at deployment that can never change. It might be a base string concatenated with the token number, where the base string is held in storage and can be replaced by anyone holding an administrative key. It might be generated entirely by code inside the contract, with no external reference at all. All three contracts are equally standard-compliant. They offer completely different guarantees about whether the thing you bought today will be the thing you own next year. Upgradeable contracts add a further layer. Many projects deploy their tokens behind a proxy: a small contract that holds the storage and forwards every call to a separate implementation contract, whose address can be changed by an administrator. This pattern lets developers fix bugs after launch. It also means that the code governing a token can be replaced wholesale. An upgradeable NFT contract is, as a matter of architecture, a promise by its administrators rather than a guarantee by its code. Whether that promise is acceptable depends on who holds the upgrade key, whether it is protected by a multi-signature wallet or a time delay, and whether it has been renounced. None of this is visible on a marketplace listing. Three meanings of ownership When someone says they own an NFT, they may mean any of three things, and the chain enforces only one of them. The first is control of the ledger entry. This is the narrow, technical sense: the owner mapping contains your address, and only you, or someone you have approved, can change it. This meaning is enforced by consensus and cryptography. It is as strong as the contract's code and the chain's security. The second is access to and integrity of the content: the ability to retrieve, display and rely on the image, video, 3D model or document that the token represents, and confidence that it is the same content the creator issued. This meaning depends on the chain of references from contract to file, on whether each link commits to exact bytes, and on whether anyone is keeping those bytes available. The chain enforces none of it directly. At best, the chain stores a hash that lets you verify content you obtained some other way. The third is legal rights in the work: copyright, licences to display or commercialise the image, rights to physical goods or event access. These are matters of contract law and intellectual property, governed by terms the issuer publishes, often on a website, sometimes nowhere at all. Owning a token is not, by default, owning the copyright in the image it references, any more than owning a print is owning the copyright in the photograph. Some projects grant broad commercial licences to holders; some grant almost nothing; some changed their terms after launch. In 2022 the venture firm Andreessen Horowitz published a set of standard licences for NFT projects, under the name "Can't Be Evil", precisely because the legal layer had been left so inconsistent. Licence terms can themselves be stored on IPFS and referenced from the contract, which binds the token to a specific text, but the enforceability of that text is still a question for courts, not code. Most disappointment with NFTs arises from confusing the first meaning with the second or third. The ledger entry is strong, so buyers assume the content and the rights are equally strong. They are not, unless someone made them so. This book is concerned mainly with the second meaning, the integrity and survival of the content, because that is where architecture can make the largest difference and where the most avoidable failures have occurred. The legal layer reappears only where architecture touches it. Before the standard: CryptoPunks The history of the first widely known NFT project illustrates every point above, including the solution this book will eventually recommend. CryptoPunks was launched in June 2017 by Matt Hall and John Watkinson, working as Larva Labs. It comprised ten thousand small pixel-art characters generated by an algorithm, which anyone with an Ethereum wallet could claim for free at launch. It predated the ERC-721 standard by several months, so its contract was a custom design, closer to the fungible token conventions of the time, with its own functions for transferring, offering and bidding on punks. What the CryptoPunks contract did not contain was the images. Storing ten thousand images on Ethereum in 2017 would have been prohibitively expensive. Instead, Larva Labs composed all ten thousand characters into a single image file and embedded a SHA-256 hash of that file in the contract. The image itself was hosted by Larva Labs and widely copied. The design meant that anyone could check a candidate image against the hash recorded on the chain and know whether it was the genuine composite. The contract did not keep the image alive, but it fixed, permanently and verifiably, what the image was. That is integrity without availability: precisely the property this book attributes to content addressing. The weakness was that availability rested on copies, and the attributes of each punk, its type and accessories, were not recorded on the chain at all. In August 2021, after the market for punks had grown enormously, Larva Labs deployed an additional contract that stored the punk images and attributes directly on Ethereum, in compact encoded form, along with functions that render each punk as an image from that on-chain data. From that point the punks no longer depended on any server. The cost of doing so in 2021 was considerable, and it was paid long after launch, by which time the collection's value made it obviously worthwhile. The arc of CryptoPunks, from a hash in the contract to the full data on the chain, maps the spectrum of approaches that later chapters examine. At one end, a token points by location to a file on someone's server, and integrity and availability both depend on that someone. In the middle, a token points by hash, so integrity is guaranteed and availability depends on whoever keeps copies. At the far end, the data lives in the same replicated state as the ownership record, and both are equally durable. Each step along the spectrum costs more and guarantees more. The job of the standards CryptoPunks also showed why standards were needed. Because its contract was custom, every wallet and marketplace that wanted to support punks had to write code specifically for it. When CryptoKitties launched in November 2017, with its breeding cats and a contract designed by a team at Axiom Zen that included Dieter Shirley, the industry had two incompatible non-fungible designs and the prospect of hundreds more. CryptoKitties also demonstrated the pressure that popular NFT applications could put on Ethereum: within days of launch its transactions were a substantial share of network traffic, and the network slowed under the load. The response was ERC-721, drafted by William Entriken, Dieter Shirley, Jacob Evans and Nastassia Sachs, which defined a common interface so that any wallet or marketplace could support any compliant collection without special code. The standard's job was interoperability of the ledger layer. It deliberately said little about content. Its metadata extension specified a function returning a URI and a minimal JSON schema with three fields, and left almost everything about where that JSON should live, how it should be addressed and who should keep it to the implementer. That silence was a reasonable choice for a standard concerned with ownership and transfer. It also left the most important question about the token's value, whether the thing would still be there and still be the same thing, to the discretion of each project. The next two chapters examine the two standards that govern NFT ownership in detail. The chapters after that follow the pointer out of the contract and into the world the standards declined to specify. Chapter 2: ERC-721 and the Grammar of Ownership ERC-721 reached final status as an Ethereum standard in 2018 and has governed the great majority of non-fungible tokens since. It is short, as standards go, and almost every line in it encodes a judgement about risk. Reading it closely shows what the designers thought needed protecting, what they thought could be left to applications, and where the gaps that later caused trouble were already visible. This chapter walks through the interface in the order a builder would meet it: identity, ownership, transfer, delegation, discovery and, last and least specified, metadata. Identity and ownership Every ERC-721 token is identified by a pair: the address of the contract and a 256-bit unsigned integer, the token identifier. The standard says the identifier must not change for the life of the contract and that the pair is globally unique. It deliberately does not say how identifiers are chosen. Many collections number tokens sequentially from zero or one. Others derive identifiers from hashes of some property of the asset, which allows identifiers to be computed off-chain before minting. The standard's authors noted explicitly that callers must not assume identifiers follow any pattern, and treat them as opaque. Two read functions expose ownership. balanceOf(address owner) returns how many tokens an address holds in this contract. ownerOf(uint256 tokenId) returns the address that owns a given token. The standard requires ownerOf to throw an error for a token that does not exist rather than returning the zero address, and requires balanceOf to reject queries about the zero address. These small rules matter: a token whose owner reads as zero would be indistinguishable from a destroyed one, and code that treated "no owner" as a valid answer could be tricked. Notice what these two functions do not provide. There is no way, using only the required interface, to ask a contract which tokens an address owns, or to list every token that exists. The authors considered that enumeration would be expensive to maintain in storage for large collections, so they moved it into an optional extension, discussed below. The practical consequence was that the ecosystem came to rely on off-chain indexers, reading Transfer events, to answer the most basic question a user asks: what do I own? The trust implications of that reliance were noted in the previous chapter. Transfer and the receiver check Three functions move tokens. transferFrom(from, to, tokenId) moves a token from one address to another if the caller is entitled to do so. safeTransferFrom(from, to, tokenId) does the same with an additional safety check, and a second version of safeTransferFrom takes an extra bytes argument that is passed along to the recipient. The safety check exists because of a failure that had already cost users money with fungible tokens. On Ethereum, an address can belong either to a person, through a private key, or to a contract. A contract that was not written to handle tokens has no way to move them out once they arrive. Send a token to such a contract and it is locked forever, owned in the ledger by code that will never transfer it. With fungible tokens, a portion of a balance is lost. With a non-fungible token, the entire asset is. The safeTransferFrom functions address this by checking, after updating ownership, whether the recipient is a contract. If it is, the token contract calls a function on the recipient named onERC721Received, passing the operator, the previous owner, the token identifier and the extra data. The recipient must return a specific four-byte value, the first four bytes of the hash of that function's own signature, which works out to 0x150b7a02. If the recipient returns anything else, or does not implement the function at all, the whole transfer reverts. A contract can therefore receive tokens through a safe transfer only if it has been deliberately written to accept them, signalled by returning that magic value. This is a clever piece of defensive design, and it carries its own hazards. Calling into the recipient hands control of execution to arbitrary code in the middle of the token contract's operation. If the token contract has not finished updating its own state, or if it performs further actions after the call, the recipient can call back into the token contract and exploit the inconsistency, a pattern known as reentrancy. Several NFT minting contracts have been exploited this way: a contract that minted a token with a safe transfer, invoked the receiver hook, and only then incremented its count of minted tokens or checked a per-wallet limit allowed an attacker's receiver to mint again from inside the hook. The receiver check prevents one kind of loss and opens a door to another, which is why careful implementations update all state before calling external code and guard minting functions against reentry. The plain transferFrom remains in the standard because some callers know the recipient is safe and do not want to pay the gas for the check, and because some recipient contracts predate the standard. Wallet interfaces and marketplaces generally use the safe variants. Approvals and operators An NFT that could be moved only by its owner would be difficult to sell. A marketplace needs to transfer the token to a buyer at the moment payment arrives, without the seller being online to sign the transfer. ERC-721 handles this with two levels of delegation. approve(address approved, uint256 tokenId) lets an owner nominate one address that may transfer a specific token. There can be only one approved address per token at a time, and the approval is cleared automatically whenever the token is transferred. getApproved(tokenId) reveals the current approved address. setApprovalForAll(address operator, bool approved) is broader. It authorises an operator to transfer every token the owner holds in this contract, now and in the future, until the approval is revoked. isApprovedForAll(owner, operator) reports whether such authorisation exists. Marketplaces typically ask a seller to grant operator approval once per collection, after which listings can be created by signing off-chain messages, without further on-chain transactions. Each change emits an event: Approval for single-token approvals and ApprovalForAll for operator grants and revocations. These events let wallets and security tools show a user what permissions they have outstanding. The operator model is efficient and has been the single largest source of NFT theft. An operator approval is a standing instruction: any address the owner has approved for all can drain every token in that collection from the owner's wallet at any time. Phishing schemes have exploited this relentlessly, presenting users with a transaction that looks like a mint or a login but is in fact setApprovalForAll to the attacker's address. Once signed, the attacker does not need to act immediately. The approval persists until revoked, and many users never review their approvals. Architecturally, this is a choice to optimise for convenience at the cost of a large and long-lived attack surface. Later proposals and wallet features, including clearer transaction previews and approval-revocation tools, have mitigated it at the edges. The standard itself is unchanged, because changing a deployed standard would break the applications that depend on it. Discovering what a contract supports Applications need to know whether an arbitrary address is an ERC-721 contract, and which optional extensions it implements. ERC-721 requires compliant contracts to implement ERC-165, a separate standard for interface detection. ERC-165 defines a single function, supportsInterface(bytes4 interfaceId), which returns true if the contract implements the interface with that identifier. An interface identifier is computed by taking the four-byte selector of each function in the interface and combining them with exclusive-or. Under this scheme the core ERC-721 interface has identifier 0x80ac58cd, the metadata extension 0x5b5e139f and the enumeration extension 0x780e9d63. A marketplace encountering an unfamiliar contract can call supportsInterface with these values and adapt accordingly. The mechanism is honest only to the extent that the contract is: a contract can return true for an interface it implements badly, or not at all. It is a declaration, not a proof, and it is another instance of the general rule that the standard describes behaviour it cannot enforce. A sale, step by step It helps to trace a single marketplace sale through these functions, because the sequence shows where each protection operates and where the trust lies. A collector holding token 4,071 decides to list it. The marketplace's interface first checks, by calling isApprovedForAll, whether its exchange contract is already an operator for the collector in this collection. If not, the collector signs and sends a transaction calling setApprovalForAll with the exchange contract's address and true. The token contract records the grant and emits ApprovalForAll. This is the only on-chain step the seller takes. The listing itself, stating the token, the price and an expiry, is a message the collector signs off-chain and the marketplace stores in its database. A buyer accepts the listing. The buyer's transaction calls the exchange contract, passing the seller's signed message and the payment. The exchange contract checks the signature, confirms the listing has not expired or been cancelled, transfers payment to the seller (minus any fees and royalties it chooses to pay), and then calls safeTransferFrom on the token contract, moving token 4,071 from the seller to the buyer. The token contract checks that the caller, the exchange, is an approved operator for the seller, updates the owner mapping and the two balances, clears any single-token approval on 4,071, emits Transfer, and, if the buyer is a contract, calls onERC721Received and checks the magic value. If any step fails, the entire transaction reverts, and neither payment nor token moves. Two features of this sequence matter. First, the atomicity is real: the token contract and the exchange contract execute together, so the buyer cannot pay without receiving the token or receive it without paying. That is a genuine improvement on any arrangement that relies on an escrow agent. Second, nothing in the sequence touched the token's content. The buyer's decision to pay was based on the image and attributes the marketplace displayed, which the marketplace obtained from its own cache of the token's metadata. The transaction verified ownership and payment with full cryptographic rigour and verified the thing being bought not at all. The enumeration extension The optional enumeration extension adds three functions. totalSupply() returns the number of tokens in existence. tokenByIndex(index) returns the identifier of the token at a position in a list of all tokens. tokenOfOwnerByIndex(owner, index) returns the identifier of the token at a position in the list of an owner's tokens. Together they allow a client to list every token, or every token an address holds, by calling the contract directly. Supporting these functions requires the contract to maintain additional arrays and index mappings in storage and to update them on every transfer, which makes transfers noticeably more expensive. During the period of intense minting in 2021 and 2022, many projects dropped the extension to save gas, and some went further. Azuki's ERC-721A implementation, released in early 2022, restructured ownership storage so that minting many tokens in one transaction costs little more than minting one, by recording ownership only for the first token in a batch and inferring the owners of the rest. Such optimisations are legitimate and widely used, and they push the job of answering "what exists and who owns it" further toward off-chain indexers. The metadata extension The metadata extension is the part of ERC-721 that matters most for this book and the part the standard says least about. It adds three functions. name() returns a descriptive name for the collection. symbol() returns a short abbreviated symbol. tokenURI(uint256 tokenId) returns a "distinct Uniform Resource Identifier" for a given token, which the standard says may point to a JSON file conforming to a schema it provides. That schema, labelled "ERC721 Metadata JSON Schema", has three properties: name, identifying the asset; description, describing it; and image, a URI pointing to an image resource. The standard suggests the image be between 320 and 1080 pixels wide with an aspect ratio between 1.91:1 and 4:5. That is all. There is no field for attributes or traits, no field for animation or audio, no field for a content hash, no rule about the URI scheme and no requirement that the URI or the document it points to be immutable. The URI scheme was left open deliberately. A URI may begin with https://, pointing to a web server; with ipfs://, pointing to content-addressed data; with ar://, pointing to Arweave; or with data:, containing the JSON inline in the string returned by the function itself. The standard permits all of these and privileges none. A token whose metadata lives on a company's web server is exactly as compliant as one whose metadata is generated on the chain. The decision to leave metadata unspecified had reasonable motives. In early 2018 the community had little experience with decentralised storage, gas costs ruled out on-chain media for most projects, and a standard that mandated a particular storage system would have tied the ownership layer to that system's fate. A minimal standard that let practice evolve was defensible. The cost was that the most important property for a buyer, whether the thing the token points to is fixed, was left to each implementation and became invisible at the point of sale. Other omissions followed from the same restraint. The standard defines no event for a change in metadata, so when a project changed what its tokens pointed to, marketplaces had no reliable signal and kept showing cached versions. A later proposal, EIP-4906, added MetadataUpdate and BatchMetadataUpdate events so that contracts could announce changes and indexers could refresh. The proposal is useful, but notice what it does: it makes mutation easier to coordinate. It does not make immutability easier to verify. Extensions that grew around the standard ERC-721's minimalism invited extensions, and several are now widespread enough to be considered part of the working architecture. EIP-2981, proposed in 2020 and now widely implemented, defines a royaltyInfo(tokenId, salePrice) function that returns a recipient address and an amount owed to the creator on resale. It is purely informational: it tells a marketplace what the creator would like to be paid, and it cannot make the marketplace pay. When competition between marketplaces in 2022 and 2023 pushed several to make creator royalties optional, the limits of a signalling standard became obvious. OpenSea briefly attempted to enforce royalties through an on-chain registry that let collections block transfers via marketplaces that did not honour them, then abandoned the approach in 2023. The episode is outside this book's main subject, but it reinforces its theme: a standard can describe an intention, but only code that actually executes on the chain enforces anything. EIP-4907 adds a time-limited "user" role separate from the owner, so that a token can be rented without being transferred. ERC-6551, proposed in 2023, gives every NFT its own smart-contract account, which can hold other tokens and act on the chain, controlled by whoever owns the NFT. Both treat the token as an identity or key rather than as a pointer to a picture, and both inherit the same separation between strong ledger state and whatever off-chain content the token refers to. What ERC-721 settles and what it leaves open Read as a whole, ERC-721 settles the ledger layer carefully. Ownership is unambiguous, transfers emit events, accidental loss to incompatible contracts is guarded against, delegation is explicit and capabilities are discoverable. Where the standard is weak, the weakness lies in delegation that is too broad and in receiver hooks that open reentrancy paths, both well understood and both manageable by careful implementation. What the standard leaves open is everything between tokenURI and the asset. It does not say whether the URI may change, who may change it, whether the document it names is bound to exact bytes, where that document should be kept or who should keep it. A collector comparing two ERC-721 tokens on a marketplace sees the same interface and the same apparent guarantee of ownership. The difference between a token whose content can be swapped by a single key and one whose content is fixed forever is not part of the standard at all. It is in the contract's code and in the URI it returns, and the only way to know is to read both. The next chapter examines ERC-1155, which approaches the ledger layer differently and whose metadata design makes the storage question sharper still. Chapter 3: ERC-1155 and the Many-Token Contract ERC-721 treats each contract as a single collection of unique items. That model fits a set of ten thousand portraits well and fits a video game badly. A game might have a sword of which there are exactly one, a healing potion of which there are a million, a currency used to buy both, and thousands of other item types besides. Under ERC-721 the unique sword is natural, the million potions would be a million separately tracked tokens, and the currency would need a different standard altogether, ERC-20, in a different contract. Moving a player's inventory from one place to another would require a separate transaction, or at least a separate call, for each item. ERC-1155, the multi-token standard, was written to solve that problem. Its authors, Witek Radomski, Andrew Cooke, Philippe Castonguay, James Therien, Eric Binet and Ronan Sandford, were led by Radomski of the blockchain gaming company Enjin and included developers from other game-focused teams, and the proposal was created in June 2018. It reached final status and has become the second pillar of NFT architecture, used by games, by editioned art, by ticketing and membership systems, and by many platforms that let creators mint without deploying their own contracts. One contract, many token types The central change in ERC-1155 is that a token identifier no longer names a single item. It names a type, and each address holds a quantity of each type. Where ERC-721 keeps a mapping from token identifier to owner, ERC-1155 keeps a mapping from token identifier and owner to balance. The core read function reflects this: balanceOf(address account, uint256 id) returns how many units of type id the account holds. There is no ownerOf. The concept does not exist at the level of the standard, because in general a token type may be held by many accounts in many quantities. A type with a total supply of one, held by a single account, behaves like an ERC-721 token, and a type with a supply of a million behaves like a fungible currency, but the contract does not distinguish them. Fungibility in ERC-1155 is a matter of supply, decided by the contract's minting logic, not a matter of kind fixed by the standard. This has a real consequence for what a buyer can verify. With an ERC-721 token, the question "is this unique?" is answered by the standard itself: each identifier has exactly one owner. With an ERC-1155 type, uniqueness depends on whether the contract will ever mint more units of that type. A creator can sell a type as a one-of-one and later mint a second unit, unless the contract's code prevents it. Some contracts record a maximum supply per type and refuse to exceed it; some expose no supply information at all. The standard is silent. A buyer who cares about scarcity has to read the minting function. Batch operations The second change is batching. ERC-1155 has safeTransferFrom(from, to, id, amount, data) for moving a quantity of one type and safeBatchTransferFrom(from, to, ids, amounts, data) for moving quantities of many types in a single call. There is a matching balanceOfBatch(accounts, ids) that reads many balances at once. Batching matters for cost. Every Ethereum transaction carries a fixed base cost, and every external call between contracts carries overhead. Moving fifty items in one batch amortises those costs across all fifty. For a game that needs to grant a player a bundle of rewards, or for a platform that airdrops editions to many collectors, the saving is substantial. Notice that ERC-1155 has no unsafe transfer at all. Every transfer, single or batch, performs a receiver check when the recipient is a contract. The recipient must implement onERC1155Received for single transfers, returning 0xf23a6e61, and onERC1155BatchReceived for batches, returning 0xbc197c81. The designers had watched tokens get locked in contracts that could not handle them and chose to make the protective check mandatory. The reentrancy concern described in the previous chapter applies here too, and with more force, since every transfer to a contract invokes external code. Events and the reconstruction of state ERC-1155 defines TransferSingle and TransferBatch events. Each names the operator who initiated the transfer, the sender, the recipient, and the identifiers and amounts moved. Minting is a transfer from the zero address and burning a transfer to it, as with ERC-721. The standard places an unusually strong requirement on these events. It requires that the balances of every account can be reconstructed from the event log alone, which means a contract must emit a transfer event for every change in balance, including minting, with no silent adjustments. This is a direct acknowledgement of the role indexers play. Because the standard offers no enumeration function, nothing like ERC-721's optional tokenOfOwnerByIndex, the only practical way to answer "what does this account hold?" is to replay the events. The standard made that replay reliable by obligation. There is also an ApprovalForAll event and the corresponding setApprovalForAll and isApprovedForAll functions. ERC-1155 has no per-token approval. An owner either authorises an operator for every type and quantity they hold in the contract, or authorises nothing. This is simpler than ERC-721's two-level model and concentrates risk further. A single mistaken approval exposes everything the owner holds in that contract, which for a game or a shared minting platform may be a great many items from many creators. Interface detection uses ERC-165 as before. The core ERC-1155 interface has identifier 0xd9b67a26, and the optional metadata URI extension 0x0e89341c. Metadata by template ERC-1155's approach to metadata differs from ERC-721's in a way that matters for storage. The optional metadata extension defines a function uri(uint256 id) that returns a URI for a token type. Crucially, the standard allows, and expects, that the same string be returned for every type, containing the placeholder {id}. Clients are required to replace that placeholder with the actual token identifier, formatted as lowercase hexadecimal, without a 0x prefix, padded with leading zeros to exactly sixty-four characters. So a contract might return https://example.com/api/{id}.json for every type, and a client wanting the metadata for type 314,592 would compute its hexadecimal form, 4cce0, pad it to sixty-four characters, and request a file whose name is fifty-nine zeros followed by those five digits. The standard's own example uses exactly that identifier. The reason for the template is economy. A contract with thousands of types need store only one string, and need not compute a different URI per type on chain. The cost is that it ties metadata strongly to a directory model: one base location containing one file per type, named by a rigid convention. That works naturally with a web server, which can generate responses dynamically for any identifier. It also works with a content-addressed directory, in which all metadata files are bundled together and the directory as a whole receives a single hash. It works less naturally when each type's metadata is stored and addressed independently, because there is then no single base under which every file can be found. The standard also defines a URI event, emitted with the new string and the type identifier when a type's URI changes. Like ERC-721's later metadata-update events, this exists to let indexers notice changes. Its presence in the core standard reflects the gaming origins of ERC-1155, where items might reasonably evolve, but it also normalises mutation. An ERC-1155 contract that emits URI events is announcing that the metadata of its types is not fixed. The metadata JSON schema in ERC-1155 extends ERC-721's with a decimals field, indicating how a type's balance should be displayed if it is divisible, and a properties object for arbitrary additional data. It also includes a localisation scheme in which the metadata can point to translated versions using a {locale} placeholder. None of this changes the basic structure. A URI, possibly templated, points to a JSON document, which contains a further URI for the image. Semi-fungibility and changing state Because ERC-1155 treats fungibility as a matter of supply, a single contract can model items that move between fungible and non-fungible states, and this has proved useful well beyond games. Consider an event ticket. Before the event, all general-admission tickets are interchangeable, so they are naturally one type with a supply of, say, two thousand. After the event, a ticket that was used might become a commemorative collectible, perhaps unique to the seat or to the attendee. An ERC-1155 contract can implement this by burning one unit of the general-admission type and minting a unit of a new collectible type in the same transaction, emitting a TransferSingle event for each. The ticket holder's balance of the first type goes down by one and of the second goes up by one. Nothing about this requires a second contract or a second standard. The same pattern covers crafting in games, where several common items are burned to mint a rarer one, and redemption schemes, where a token representing a claim on a physical object is burned when the object is shipped and replaced with a token recording that it was delivered. Each of these is a state transition expressed as burns and mints of different types within one contract, which is efficient and transparent, since every step is in the event log. The flexibility has an architectural price that bears directly on this book's argument. When item types are created dynamically, as players craft or as events are scheduled, the metadata for new types cannot all be prepared and content-addressed in advance. The natural implementation is a URI template pointing to a server that generates metadata for any identifier on request, and that is how many games and ticketing systems work. The template then names a location, not content, and every type, including those minted long ago, depends on the server. A more careful design separates the two cases: types whose metadata is fixed point to content-addressed files, published as each type is created, and only genuinely live data is served dynamically. ERC-1155's single template makes this harder, since one string must serve all types, which is why many contracts override uri to return a different value for different ranges of identifiers or store a per-type URI in addition to the template. Both are standard-compliant. Both give up some of the simplicity that made the template attractive. A batch transfer traced A concrete trace shows how the pieces fit. Suppose a game contract governs three types: type 1, a fungible gold coin; type 2, a potion with a supply in the hundreds of thousands; and type 77, a unique sword. A player wants to move 500 gold, 3 potions and the sword to a second account, which happens to be a smart-contract wallet. The player's wallet sends one transaction calling safeBatchTransferFrom with the player's address as sender, the smart wallet as recipient, the identifier list one, two and seventy-seven, the amount list 500, 3 and 1, and empty data. The token contract checks that the caller is the sender or an approved operator, then checks that the sender's balance of each type is sufficient. It subtracts each amount from the sender's balances and adds it to the recipient's. It emits a single TransferBatch event recording the operator, the two addresses and both lists. Because the recipient is a contract, it then calls onERC1155BatchReceived on the smart wallet, which must return 0xbc197c81. If it does, the transaction completes; if not, every balance change reverts. A client wanting to show the sword afterwards calls uri(77) and receives, say, the template string with the {id} placeholder. It replaces the placeholder with seventy-seven written as sixty-four hexadecimal characters, which ends in 4d after sixty-two zeros, and fetches that file. Whether that file is fixed, and whether it will be there next year, depends on what the template points to. The batch transfer, for all its efficiency, is silent on the matter, exactly as an ERC-721 transfer is. Choosing between the standards The choice between ERC-721 and ERC-1155 is partly a matter of fit and partly a matter of the guarantees each makes visible. Table 1 sets out the differences that matter most for architecture. Table 1. Key differences between ERC-721 and ERC-1155 (source: EIP-721 and EIP-1155 texts). Property ERC-721 ERC-1155 Unit identified One unique token A token type with balances Ownership query ownerOf(id) balanceOf(account, id) only Batch transfer Not in standard safeBatchTransferFrom Unsafe transfer transferFrom allowed None; receiver check always Approvals Per token and per operator Per operator only Metadata function tokenURI(id), per token uri(id), {id} template Change event None in core standard URI event For a collection of unique works of art, each sold individually, ERC-721 says more on its face. The existence of ownerOf means uniqueness is enforced by the standard rather than by minting logic, and the per-token approval lets owners delegate narrowly. For editions, games, tickets and anything with many identical copies, ERC-1155 is far more efficient, and its batch operations and mandatory receiver checks are genuine improvements. A common pattern on creator platforms is a shared ERC-1155 contract in which many artists mint types. This is cheap for the artist, who need not deploy a contract, but it concentrates control. The platform's contract governs every type, often holds administrative keys that can change URIs, and may be upgradeable. An artist who mints a one-of-one on such a platform has produced a token whose scarcity depends on the platform's code and whose content depends on the platform's administration of URIs. Some platforms handle this responsibly, storing metadata on IPFS and limiting administrative powers; the point is that the standard does not make that responsibility visible, and the buyer has to check. Where the two standards meet For all their differences, the two standards share the structure that this book is concerned with. Both secure ownership on the chain with the full strength of consensus. Both expose a function that returns a string. Both say nothing binding about whether that string can change, whether it commits to exact bytes, or where the bytes live. Both let a contract return a web address on a server that the issuer controls, and both are equally compliant when it does. There is one practical difference worth carrying forward. ERC-721 lets each token compute its own URI, so a contract can return a different content identifier for every token if it chooses, and tokens can be added to a collection later, each with its own independently published metadata. ERC-1155's template encourages a single directory published at once, which is excellent for a fixed set of types and awkward for a catalogue that grows. A growing ERC-1155 collection that wants content-addressed metadata must either publish a new directory, and therefore a new root identifier, each time types are added, which means changing the template and emitting URI events, or override uri to return a separate identifier per type. The first approach reintroduces a mutable reference at the top of the chain; the second departs from the template that clients expect to receive. Neither is wrong, but each moves the burden of proving immutability onto the contract's code, which is where a careful buyer must look for it. The standards also share an assumption that metadata is a separate document rather than part of the token. That separation was driven by cost and has been useful, letting rich descriptions, attributes and media sit alongside a compact ledger. It also means that every NFT, under either standard, is a two-part object: a strong core and a soft periphery joined by a URI. The next chapter examines that periphery in detail: what the metadata document contains in practice, how marketplaces interpret it, and the options for where it can be kept. Chapter 4: The Metadata Layer Ask a collector what their NFT is and they will describe a picture: a pixelated punk, a cartoon ape, a generative abstraction, a photograph. Ask the contract and it will return a string. Between the two lies the metadata layer, a small JSON document and the chain of addresses that leads to it and away from it. This layer is where the token becomes an asset in anyone's eyes, and it is where most of the architectural risk sits. This chapter describes what the document contains, how its conventions arose, how the chain of references is assembled, and what the alternatives are for where each piece lives. The document A typical NFT metadata document is a few hundred bytes of JSON. The ERC-721 standard specified three fields, name, description and image. In practice almost every collection uses more, because marketplaces, above all OpenSea in the period when it dominated NFT trading, defined additional fields and displayed them prominently, and projects followed whatever made their tokens look good where they were sold. The fields that became conventional are summarised in Table 2. Most are informal: they are documented by marketplaces rather than by any Ethereum standard, and interpretation varies at the edges. Table 2. Common NFT metadata fields and their origin (sources: EIP-721, EIP-1155, OpenSea metadata documentation). Field Purpose Defined by name Title of the individual token EIP-721 description Human-readable description EIP-721 image URI of the main image EIP-721 attributes List of traits with values Marketplace convention animation_url URI of video, audio, 3D or HTML Marketplace convention external_url Link to the project's own page Marketplace convention properties Arbitrary extra data EIP-1155 The attributes field deserves attention because it carries much of a collection's economic structure. In a generative collection, each token is assembled from layers (background, body, clothing, eyes, accessory) and each layer's choice is recorded as an attribute, an object with a trait_type and a value, such as a trait type of "Eyes" with a value of "Laser". Rarity rankings, which strongly influenced prices in large collections, are computed from these attributes across the whole collection. If the attributes can change, the rarity of a token can change, and with it the price buyers paid on the basis of it. The animation_url field matters for a different reason. It can point to an HTML page, which marketplaces render in a sandboxed frame. That allows interactive and generative works that run in the viewer's browser, but it also means the "asset" may be a program that fetches further resources at run time. An HTML artwork that loads a library from a public content-delivery network, or data from an API, has dependencies that do not appear in the metadata at all. Its integrity depends on those external resources as well. The metadata document is also small enough to be placed directly in the contract's response. A data: URI, of the form data:application/json;base64, followed by the encoded document, lets tokenURI return the metadata itself rather than a pointer to it. Contracts that generate their metadata on chain use exactly this technique. It removes one link from the chain entirely, since there is no external document to fetch, and moves the question of integrity to whatever the inline document's image field points to. If that field is also a data: URI containing an encoded SVG image, the whole token is self-contained; if it is an https:// URL, the inline metadata has merely moved the weak link one step further out. The chain of references Assembling the pieces, a typical ERC-721 token is resolved like this. A client calls tokenURI(4071) on the contract and receives a string. In many contracts, that string is formed by joining a stored base URI with the token number, perhaps with a .json suffix. The client then fetches the JSON document at that URI. Inside it, the client finds the image field, another URI, and fetches the image. If there is an animation_url, it fetches that too, and if that is an HTML page, the browser fetches whatever the page requests. Each arrow in that sequence is a reference, and each reference is one of two kinds. A location reference names a place: a domain name and a path on a web server, or a bucket and key in a cloud storage service. Whatever is at that place when you ask is what you get. A content reference names specific bytes, usually by a cryptographic hash: an IPFS content identifier, an Arweave transaction identifier, or a hash recorded alongside a location. Whatever you retrieve either matches the reference or is rejected. The integrity of the whole chain is the integrity of its weakest link. If the contract returns a content reference to a JSON document, but that document's image field is an https:// URL on the project's server, then the metadata is fixed and the image is not. The project can replace the picture at will, and the hashed JSON will continue to point faithfully at whatever the server now returns. Conversely, if the JSON and the image are both on IPFS but the base URI in the contract can be changed by an administrator, then the administrator can point every token at a different, equally content-addressed, set of files. Each link has to be fixed, and the first link, the contract's own logic for producing the URI, has to be fixed too. This is the reason the book's controlling claim is phrased in terms of a chain. A token is secure in its content only if every step from the storage slot to the final pixel is bound by a hash and the code computing the first step cannot be changed. Where the pieces can live Builders have several options for where to put metadata and media, and each embodies a different trust assumption. Table 3 summarises them, ordered roughly from weakest to strongest guarantee of integrity. Table 3. Hosting options for NFT metadata and media, by what each guarantees. Option Reference type Integrity Availability depends on Project web server or API Location (https://) None; host can change it Host's continued operation Cloud object storage Location (https://) None; account holder can change it Account and payment IPFS via a gateway URL Location wrapping a hash Only if client verifies Gateway operator and pinners IPFS native URI Content (ipfs://) Verifiable by hash Whoever pins the content Arweave Content (ar:// or ID) Verifiable by hash Arweave network and endowment On-chain data Contract storage or code Same as ledger The blockchain itself The first two options are what the industry calls centralised hosting. They are cheap, fast and familiar. They also give the operator unlimited power to change what a token shows, and they make the asset's survival depend on the operator remaining in business, remembering to renew its domain, paying its cloud bill and not being compromised. The reasons projects chose them were mostly practical. A dynamic API can generate metadata on demand, update it when game state changes, hide unrevealed tokens until a sale ends and serve images through a fast content-delivery network. All of those conveniences are also capabilities to mutate. The third row is subtle and widely misunderstood. Many contracts store URIs of the form https://ipfs.io/ipfs/ followed by a content identifier, or the equivalent on another gateway. The content identifier inside the URL is a hash, so in principle the content is fixed. But the URL as a whole is a location reference to a particular gateway. If that gateway shuts down, the URL stops working even if the content is available elsewhere on IPFS. And a client that fetches the URL over HTTP and does not itself check the returned bytes against the identifier is trusting the gateway operator to return the right content. In practice most clients have trusted gateways implicitly. The Cloudflare public IPFS gateway, which many projects had hard-coded into their metadata, was retired in August 2024, with traffic redirected for a transition period to gateways run by the IPFS Foundation's operators. Projects that had written the native ipfs:// form were unaffected; projects that had written Cloudflare's hostname depended on that redirect. The fourth and fifth rows give genuine integrity, because the reference is the hash and any compliant client can verify it. They differ in their persistence models, which are the subject of Chapter 7. The sixth row places the data in the same replicated state as the ownership record. It is the strongest option and the most expensive, and it constrains what kind of asset is practical. The case for centralised hosting, and its limits It would be unfair to treat centralised hosting as mere negligence. Its defenders make a serious argument, and it deserves a serious answer. The argument runs as follows. Most NFTs are issued by identifiable organisations, many of them established companies with legal obligations, reputations and customer-service departments. A buyer of such a token is trusting the issuer anyway, for the licence, for any associated perks and often for the token's market value, which depends on the issuer's continued activity. Adding content addressing does not remove that trust; it only removes one small part of it. Meanwhile, centralised hosting delivers fast loading, the ability to fix mistakes, compatibility with every browser and the flexibility to add features. Customers want those things. If the issuer misbehaves, the buyer has legal recourse, as with any other digital product. On this view, insisting on IPFS is ideological purity at the expense of user experience. Parts of this are right. Trust in the issuer is not eliminated by content addressing, and for tokens whose value lies mainly in a service the issuer provides, such as access to events or software features, the content of the metadata may matter much less than the issuer's ongoing performance. For those tokens, a hosted API may be a perfectly reasonable design. The argument fails, however, for tokens sold as digital objects whose value lies in the object itself: art, collectibles, records, anything advertised as permanent or unique. For those tokens, centralised hosting does not merely retain some trust in the issuer; it makes the issuer's continued existence and continued good behaviour a condition of the asset existing at all. Legal recourse against a company that has closed is worth little, and recourse against an attacker who hijacked an expired domain is worth nothing. The user-experience benefits, meanwhile, are almost all available with content addressing too: content-addressed files can be served through fast gateways and content-delivery networks, mistakes can be corrected before metadata is frozen, and dynamic features can be built from on-chain state. What content addressing removes is the ability to change the object silently after sale. For a token sold as a fixed object, that is not a feature the buyer should want the issuer to keep. The honest position, then, is that hosting should match the claim. A token that is presented as a service credential can live on a server. A token presented as a permanent object must not. Dynamic metadata and its legitimate uses It would be too simple to say that metadata should always be immutable. Some tokens are designed to change. A game character may level up. A ticket may record that it has been used. A generative artwork may respond to the time of day or to on-chain events. A membership pass may display a holder's tenure. Tokens like these are often called dynamic NFTs, and their metadata must change by design. The architectural question is not whether metadata changes but who can change it and under what rule. There is a large difference between metadata that changes because a server operator decided to change it and metadata that changes because a contract's code computes it from on-chain state according to logic anyone can read. In the first case, the buyer is trusting the operator's future behaviour. In the second, the buyer is trusting code they can inspect, which will behave the same way regardless of who runs the server. The cleanest dynamic designs keep the variable state on chain and either compute the metadata in the contract or compute it off chain deterministically from that state, with the rendering logic and any media assets content-addressed. A level-up then changes a number in contract storage, which everyone can see, and the image is drawn from fixed, hashed components according to a fixed, published rule. The token changes, but nobody can change it arbitrarily. Designs that keep the state in a company's database and expose it through an API offer no such assurance, whatever they call themselves. Reveals and provenance hashes One common practice deliberately starts with mutable metadata and then fixes it: the delayed reveal. During a generative collection's sale, every token points to the same placeholder image, so that buyers cannot pick the rare ones by inspecting metadata before minting. After the sale ends, the project changes the base URI to point at the real metadata, and each token acquires its traits. The reveal is reasonable in purpose and dangerous in mechanism, because the same function that sets the real base URI can set any base URI, and a project that knows which buyers hold which tokens could, in principle, arrange the assignment of rare traits to favoured wallets. The practice that emerged to constrain this is the provenance hash. Before the sale, the project computes a hash of every final image in a defined order, concatenates those hashes, hashes the result and publishes that single value, often in the contract itself. After the sale, a random starting offset, ideally derived from a value the project could not control, such as a future block's data or a verifiable random function, determines how tokens map to that ordered list. The provenance hash commits the project to the final set of images and their order before anyone knows which token will receive which. After the reveal, anyone can recompute the hash from the published images and check it matches. Bored Ape Yacht Club, launched in 2021, published such a provenance record, and the technique became common practice. Its guarantee is limited to what it commits to: if the commitment covers only image files and not the metadata, traits could in principle still be adjusted; if the offset is chosen by the project, the randomness is only as fair as the project's honesty. Used well, it shows what this book argues for in miniature: a hash, published before the fact, turning a promise into something checkable. Freezing Once metadata is final, the contract should make it impossible to change. The simplest way is never to provide a function that changes the URI at all, setting it once in the constructor. Where a reveal or a corrected deployment requires the ability to set it later, a contract can include a one-way switch, a function that sets a flag after which the URI setter permanently reverts. OpenSea popularised a convention of emitting a PermanentURI event to announce that a token's URI is frozen, and its interface displays frozen metadata accordingly. An event is a statement, not a constraint. A contract can emit PermanentURI and still contain a setter that works. The only reliable test of whether metadata is frozen is to read the code: confirm that every path that could change the URI is disabled, that no proxy can swap the implementation, and that the URI returned is a content reference whose content's own internal references are also content references. That is a demanding audit to ask of every collector, which is one reason the failures described in the next chapter happened as often as they did. Chapter 5: How the Pointers Fail A pointer fails in one of two ways. Either it stops resolving, and the token points at nothing, or it resolves to something other than what it originally named, and the token points at the wrong thing. The first is a failure of availability; the second, of integrity. Both have happened to NFTs repeatedly, and they have happened through a small number of recurring mechanisms. This chapter catalogues those mechanisms, because the defences described in the following chapters make sense only against a clear picture of what they defend against. Link rot, and why NFTs are not exempt The ordinary web loses its content steadily. A study by the Pew Research Center published in 2024 found that 38 percent of web pages that had existed in 2013 were no longer accessible a decade later, and that broken links were common even on government and news sites. Pages disappear because organisations reorganise their websites, change domain names, stop paying for hosting, are acquired or close. Nothing about a URL obliges anyone to keep it working. An NFT whose metadata or image is referenced by an https:// URL is subject to exactly the same decay. The fact that the reference is stored on a blockchain does not change the nature of the reference. The blockchain will preserve the URL perfectly, for as long as the chain exists; it will not preserve what the URL points to for even a day. The permanence of the ledger has, if anything, made the decay of the periphery more conspicuous. A broken link on a news site is an annoyance. A broken link in an asset that someone paid for, and which is advertised as permanent, is a broken promise. The most common path to an empty token is the most mundane: the issuing company stops operating. Startups fail, and NFT startups failed in large numbers after the market contracted in 2022. When a company that hosted its own metadata shuts down, its servers go dark, its cloud account is closed and its domain eventually lapses. The tokens persist, their ownership records remain accurate, and every one of them now resolves to an error. No hack, fraud or bad faith is needed. Ordinary business failure is sufficient. Corporate issuers are not immune merely because they are large. A brand that launches an NFT collection as a marketing initiative may wind the initiative down when priorities change, and the servers supporting the collection are then a cost with no budget line. Unless the collection was built so that its content does not depend on the issuer, the issuer's change of strategy becomes the owner's loss. Domains as attack surface A lapsed domain is worse than a dead one, because someone else can register it. Once a domain expires and is released, whoever registers it controls every URL beneath it. If an NFT collection's metadata lived at that domain, the new registrant can serve any metadata they like for every token in the collection. They can replace images, rewrite descriptions and alter attributes. They can serve images that harm the owners' reputations or redirect viewers to phishing pages. The contract, faithfully, continues to point at the domain. The same exposure exists without expiry. A domain can be hijacked by compromising the registrar account, by social engineering the registrar's support staff, or by manipulating the domain name system. A web server can be compromised directly. A cloud storage bucket can be misconfigured so that others can write to it. In every case, whoever controls the location controls the asset, and the token's on-chain ownership record offers no protection, because it never committed to the content in the first place. This is precisely the vulnerability Moxie Marlinspike highlighted: no hash commitment for the data at the URL. The deliberate swap Not every change is an accident or an attack by an outsider. The project itself may change what tokens point to, and the mechanisms described in the previous chapter make that easy. Most generative collections deployed in 2021 and 2022 store a base URI in contract storage with a setter function restricted to the contract owner. The setter exists to support the reveal, and in many contracts it remains active afterward. Until the owner disables it, the owner can, with a single transaction, point every token in the collection at a different set of metadata. The new set can be on IPFS, perfectly content-addressed, and still be different from what buyers bought. Content addressing protects a reference from silently changing its target. It does nothing if the reference itself is replaced. The mechanism has been used for benign purposes, such as migrating metadata from a centralised server to IPFS after launch, which is a real improvement for owners. It can also be used to alter the economics of a collection after the fact, for example by changing attributes so that tokens held by insiders become rarer. And it can be used to abandon a collection gracefully or abruptly. Because the swap function looks the same in all three cases, its mere existence means the buyer depends on the owner's continued good faith and on the owner's key not being stolen. A compromised owner key on a contract with a live base-URI setter hands an attacker the content of the entire collection. Upgradeable proxies widen the exposure. If the contract's implementation can be replaced, then the logic of tokenURI itself can be replaced, along with every other behaviour of the token. A frozen URI in the current implementation is no guarantee if the next implementation ignores it. Responding differently to different viewers Marlinspike's experiment points to a subtler integrity failure. A server can return different content to different requesters. It can inspect the requester's IP address, user-agent string, cookies or referring page and decide, request by request, what to serve. A marketplace's indexer might see one image, a collector's browser another and a journalist's laptop a third. Most people never think about this possibility because they assume, without examining the assumption, that a URL names a single thing. It does not. It names a request that a server will answer however it chooses. For ordinary web content this is a feature, enabling personalisation and localisation. For a token claiming to represent a unique and specific artwork, it means there may be no single answer to the question "what does this token show?" Content addressing removes the possibility entirely: a content identifier names bytes, and any server returning different bytes is returning something that fails verification. Gateways as hidden locations The previous chapter noted that a URL of the form https:// plus a gateway domain plus /ipfs/ plus a content identifier is a location reference wearing a content identifier. This failure mode deserves separate treatment because it has affected projects that believed they had done the right thing. When IPFS became the recommended way to store NFT metadata, many projects stored gateway URLs rather than native ipfs:// URIs, because browsers did not understand the native scheme and gateway URLs simply worked. That choice made the project dependent on the continued operation of a specific gateway. Public gateways are run by organisations that pay for bandwidth and storage and may decide to stop. Cloudflare's withdrawal of its public IPFS gateway in 2024 was handled carefully, with months of notice and a redirect period, but it illustrated the point: a project's tokens can depend on a third party that never agreed to support them. Gateways can also rate-limit, block content in response to legal demands or abuse reports, and, like any server, return wrong content if compromised or misconfigured. A client that fetches from a gateway without checking the hash has delegated its integrity guarantee to the gateway. Most wallets and marketplaces have historically done exactly that. The native ipfs:// form avoids hard-coding any one gateway. Clients translate it to a gateway of their choice, or fetch through a local IPFS node, or verify the content themselves. The data is named by what it is, not by who serves it, and the choice of server can change without changing the token. Caches that hide the damage A curious feature of NFT failures is that they are often invisible for a long time. Marketplaces and wallet services do not fetch every image from its origin each time a user views a token. They fetch it once, store a copy and serve that copy through their own content-delivery networks. When the origin disappears, the marketplace's cached copy continues to display, and neither the owner nor potential buyers notice anything wrong. Caching postpones the failure but does not prevent it. The cached copy belongs to the marketplace, which may purge it on a refresh, when the collection is delisted, when its cache policy changes or when the marketplace itself winds down. The owner usually discovers the problem only when trying to view the token somewhere else, or when the marketplace re-indexes and finds nothing at the origin. By then the original files may be unrecoverable. Caching also means that the image a buyer sees at the moment of purchase may not be the image the token currently points to. If the origin's content has changed since the marketplace last fetched it, the buyer is shown the old image and acquires a token pointing at the new one. This is the reverse of Marlinspike's demonstration and just as possible. The cache, like the indexer described in Chapter 1, is a convenient view assembled by an intermediary; it is not the asset. Dependencies outside the metadata Even a carefully content-addressed token can depend on resources its metadata does not list. The most common case is the HTML artwork referenced by animation_url. Such a work might load a JavaScript graphics library from a public content-delivery network, a font from a web font service, or data from an API. Each of those is a location reference embedded in the artwork's code. If the library's URL changes or the font service retires a version, the artwork breaks, even though every link the token itself declares is a hash. Format obsolescence is a slower form of the same problem. Digital preservation professionals know that a file is only as durable as the software able to interpret it. Adobe ended support for Flash Player at the end of 2020, and web browsers removed it, which made a large body of interactive web art unplayable without emulation. NFT media in common formats such as PNG, JPEG, MP4 and standard HTML are relatively safe, because those formats are well documented and widely implemented. Media in proprietary or niche formats, or interactive works that depend on particular browser behaviours, carry a risk that no storage system can address. The bytes may survive while the ability to experience them does not. Off-chain ownership claims A final category of failure involves tokens that are not quite on the chain at all. Some platforms offer "lazy minting", in which a creator signs a message authorising the creation of a token, and the token is actually written to the chain only when someone buys it. Until then, the "NFT" is a signed message in the platform's database. That is a reasonable way to save creators the cost of minting unsold work, but it means that an unsold lazily minted token has none of the properties of a ledger entry. If the platform loses the signature, the token never existed. Similarly, some services display NFTs that are in fact records in a custodial database, with on-chain settlement deferred or batched. These are not failures of the NFT standards; they are cases where the standards were never applied. They are mentioned here because the user experience can be identical to that of a real token, and the architectural guarantees are entirely different. Tracing a broken token The failure modes above are easiest to understand by following one token through its decline. The example is a composite, built from mechanisms that have each occurred, rather than a single documented case. A collection launches in 2021. Its contract stores a base URI of the form https://api. followed by the project's domain and /metadata/, with a setter controlled by the founders' single-key wallet. The API generates metadata on request from a database, and each document's image field is an https:// URL to a cloud storage bucket. Marketplaces index the collection, fetch every image once and cache them. In the first year, nothing appears wrong. The founders use the setter once, legitimately, to move from a placeholder to the revealed metadata. Holders see their tokens on every marketplace. In the second year, the project's revenue dries up. The founders stop paying for the API server, which goes offline. Marketplaces continue to display cached images and cached attributes, so the outage is unnoticed. A holder who tries to view a token in a new wallet application, which fetches metadata directly, sees a blank card and assumes the wallet is at fault. In the third year, the cloud account is closed for non-payment, deleting the images. A marketplace refreshes its metadata for the collection after a user requests it, finds nothing at the origin, and replaces the cached images with placeholders. Now everyone notices. The domain registration lapses some months later. A speculator registers it and points the API path at a server returning metadata for a different set of images, along with an external_url that leads to a phishing site styled to look like the original project. Marketplaces that refresh pick up the new metadata and display the substituted images under the original collection's name. At no point did anything happen on the blockchain. Every token's ownership record remained exact. The contract continued to return the same URI. The failure was entirely in the periphery, and it progressed through every category described in this chapter: availability loss hidden by caches, then exposed, then converted into an integrity failure by a domain takeover. The founders' setter, still live, was never used maliciously, but if their key had been stolen at any point, an attacker could have done the same damage directly. Each stage could have been prevented by a decision made at launch. Content-addressed metadata and images would have made the domain takeover irrelevant. A disabled setter would have closed the key-theft path. A persistence plan independent of the founders' finances would have kept the images available after the company failed. None of those decisions was expensive in 2021. All of them were expensive to retrofit afterward, and some were impossible, because once the original files were deleted, there was nothing left to content-address. A taxonomy of failure Gathered together, the failure modes fall into a short list. Availability fails when the host of a location reference stops serving it: business failure, lapsed hosting, gateway retirement, cache purge. Integrity fails when the content at a location reference changes: domain takeover, server compromise, deliberate swap at the origin, viewer-dependent responses. Integrity also fails when the reference itself is changed: a live base-URI setter, an upgradeable implementation, an administrator key that is stolen. Both can fail through dependencies outside the declared references: external libraries, fonts and data, and formats that lose software support. Content addressing, the subject of the next chapter, eliminates every integrity failure in the second group, provided the client checks the hash. It does nothing about the third group, which must be addressed in the contract's code, and nothing about availability, which must be addressed by deliberate persistence. The defences are therefore three, and they correspond to the three groups: hash every reference, freeze the code that produces the first reference, and make sure someone keeps the bytes. Chapter 6: Content Addressing and the InterPlanetary File System The web names things by where they are. A URL tells your browser to contact a particular server, identified by a domain name, and ask it for a particular path. The answer is whatever that server says. Content addressing turns this around and names things by what they are. The name of a file is derived from the file's bytes, so that anyone holding the bytes can compute the name and anyone holding the name can check the bytes. It is an old idea, used for decades in version-control systems such as Git and in peer-to-peer file sharing, and it is the idea at the heart of the InterPlanetary File System. IPFS was designed by Juan Benet and described in a 2014 paper, "IPFS - Content Addressed, Versioned, P2P File System". It was developed by Protocol Labs, the company Benet founded, and its main implementations are now maintained by Interplanetary Shipyard, a team that separated from Protocol Labs in 2024. The reference implementation in the Go language, long called go-ipfs, was renamed Kubo in 2022; a JavaScript implementation called Helia runs in browsers and Node.js. This chapter explains how IPFS names, structures, finds and transfers data, and then what that machinery guarantees. Hashes as names A cryptographic hash function takes any input and produces a fixed-length output, the digest. A good hash function has three properties that matter here. It is deterministic: the same input always produces the same digest. It is collision resistant: nobody can feasibly find two different inputs with the same digest. And small changes to the input produce unrelated digests, so there is no way to make a near-miss. SHA-256, the most widely used hash function in IPFS, produces a 256-bit digest; no collision has ever been found, and finding one is believed to be far beyond any foreseeable computing capacity. If a file is named by its SHA-256 digest, three useful consequences follow. First, the name is self-certifying: given the name and some bytes claiming to be the file, anyone can hash the bytes and check. No trust in the source is required. Second, the name is location-independent: the same bytes have the same name whether they are on a server in Frankfurt, a laptop in Lagos or a hard drive in a drawer, so anyone who has them can serve them. Third, the name is immutable: a different file has a different name, and there is no way to change what a name refers to. To "update" content-addressed data is to create new data with a new name. These are exactly the properties missing from the location references described in the previous chapter. A content-addressed metadata URI cannot be redirected by a new domain owner, cannot be served differently to different viewers without the difference being detectable, and cannot be quietly replaced at the origin. It is a commitment, in the cryptographer's sense, to specific bytes. Content identifiers IPFS does not use bare hashes as names. It wraps them in a self-describing format called a content identifier, or CID, built from a family of small specifications collectively called Multiformats. The design anticipates that hash functions, data formats and text encodings will change over time, and makes every CID carry enough information to be interpreted without guesswork. A CID has up to four components. A multihash combines the digest with a prefix identifying which hash function produced it and how long the digest is, so that a SHA-256 digest is unambiguously distinguishable from one produced by any other function. A multicodec identifies how the hashed bytes are to be interpreted: raw binary, a particular IPFS file-structure format, JSON, CBOR and so on. A version number distinguishes CID formats. And when the CID is written as text, a multibase prefix identifies which text encoding was used. There are two CID versions in use. Version 0, the original, is a SHA-256 multihash encoded in base58, always implying a particular file-structure codec; such CIDs are 46 characters long and always begin with the letters Qm. Many early NFT collections use them, and they are still recognised everywhere. Version 1 carries explicit version, codec and multibase information and by default is written in lower-case base32; for the most common kind of file data, these CIDs begin bafy. Version 1 is recommended for new work, partly because it is extensible and partly because base32 is case-insensitive, which lets CIDs appear safely in places that ignore case, such as domain-name labels. The same content can be expressed as either version; converting between them does not change the underlying hash. A subtlety matters for NFTs. The CID of a file depends not only on the file's bytes but on how IPFS structured them before hashing. Two people adding the same image to IPFS with different settings, discussed below, can obtain different CIDs. Both are valid names for data that, when reassembled, is the same image. This is rarely a problem, because a collection's CIDs are computed once by the creator and recorded on the chain, but it means that verifying a file against a recorded CID requires reproducing the same structure, or, more simply, fetching the structure by its CID rather than recomputing it from a file obtained elsewhere. Blocks, links and Merkle DAGs IPFS does not hash large files whole. It splits them into blocks, hashes each block, and then builds a structure of nodes that link to the blocks by their CIDs. A node that links to other nodes by their hashes, which in turn link to further nodes by their hashes, forms what is called a Merkle DAG, a directed acyclic graph in which every link is a hash. The CID of the top node, the root, therefore commits to the entire structure beneath it. Change any byte in any block, and that block's CID changes, which changes the node that links to it, and so on up to the root. The format used for ordinary files and directories is called UnixFS. By default, Kubo splits files into chunks of 256 kibibytes and arranges them under a balanced tree of link nodes. A directory is a node whose links are named entries pointing to the CIDs of files and subdirectories. This is how a whole NFT collection's metadata can be published under a single CID: the creator places ten thousand JSON files in a directory, adds the directory to IPFS, and obtains one root CID that commits to every file and every filename. A token's URI can then be written as ipfs:// followed by the root CID, a slash and the token's filename. That structure fits both standards. An ERC-721 contract can store the root CID as its base URI and append the token number. An ERC-1155 contract can store a URI template with the directory CID and the {id} placeholder, matching the sixty-four-character hexadecimal filenames the standard requires. In both cases one short string in contract storage fixes the content of the entire collection, provided that string cannot be changed. Chunking brings practical benefits too. Large media files can be fetched in parallel from several peers. Identical blocks in different files are stored once. A video can be streamed by fetching blocks in order. And a client can verify each block as it arrives rather than waiting for the whole file, rejecting bad data early. Finding and fetching A content address says what to fetch but not where. IPFS needs a way to discover which peers hold the blocks for a given CID and a way to transfer them. Discovery uses several mechanisms. The main public mechanism is a distributed hash table, or DHT, based on the Kademlia design. Nodes in the network share responsibility for storing provider records, small entries saying "peer such-and-such can provide CID such-and-such". A node that holds content announces itself as a provider; a node seeking content queries the DHT to find providers. Provider records expire and must be republished periodically, which is one of several reasons why content held by a node that goes offline becomes unfindable fairly quickly. Large providers and services also use faster indexing systems alongside the DHT, and nodes can be connected to each other directly when they know in advance where content lives. Transfer has traditionally used a protocol called Bitswap. A node sends its connected peers a list of the blocks it wants; peers that have them send them. As each block arrives, the receiving node hashes it and checks the result against the CID it requested. A peer that sends wrong data gains nothing, because the data is rejected, and the receiver can simply ask someone else. This is the property that makes content addressing robust in an untrusted network: integrity is checked at every hop by the party who needs it, using only the name. Gateways and the trust they reintroduce Most people, and most applications, do not run IPFS nodes. They use HTTP, and to bridge the gap the IPFS ecosystem relies on gateways: HTTP servers that run IPFS nodes behind the scenes, accept requests that include a CID and return the content. A request to a gateway at a path beginning /ipfs/ followed by a CID asks the gateway to fetch that content from the network and return it over ordinary HTTP. Public gateways such as ipfs.io and dweb.link are operated for the community; many companies run their own. Gateways are indispensable and they reintroduce exactly the trust that content addressing removes. When a gateway returns a file over HTTP, the browser has no built-in way to know that the file matches the CID in the URL. The gateway verified the blocks as it fetched them from the network, but the client is trusting the gateway's word that it did so, and that it is returning the same bytes. A malicious or compromised gateway could serve anything. For most uses this is an acceptable trust assumption, much like trusting a content-delivery network, but it is a trust assumption, and it is precisely the one the NFT architecture was supposed to eliminate. Two developments narrow the gap. The first is the subdomain gateway, which serves each CID from its own origin, such as a version-1 CID written as the first label of the hostname. This gives each piece of content its own security origin in the browser, preventing one HTML artwork from reading cookies or storage belonging to another, which matters when animation_url points to interactive content. The second is the trustless gateway model, in which the gateway returns the raw blocks, or a packaged bundle of blocks in the Content Addressable aRchive format, called a CAR file, rather than a reassembled file. The client then verifies every block against the CID itself. Libraries such as Helia's verified-fetch make this practical in browsers and applications. A marketplace or wallet using verified retrieval regains the full integrity guarantee without running a full node. Verifying a token by hand The guarantees above become concrete when one follows the steps a careful client takes to verify a single content-addressed token. Suppose a contract's tokenURI(42) returns ipfs:// followed by a version-1 CID beginning bafy, a slash, and 42.json. The client first parses the CID. The multibase prefix, the initial b, says the rest is base32. Decoding it yields the version number, a codec identifying the UnixFS file structure, and a multihash whose prefix identifies SHA-256 and whose remaining thirty-two bytes are the digest of the root node. The client now knows exactly what it is looking for and how to check it. The client requests the root node's block, from a local node, from peers or from a gateway in trustless mode. On receipt it hashes the block with SHA-256 and compares the result with the digest in the CID. If they differ, the block is discarded and requested elsewhere. If they match, the client decodes the block, which is a directory listing, and finds the entry named 42.json, which carries that file's own CID. It repeats the process: request the block, hash it, compare, decode. If the file is small it may be a single block; if larger, the block is a list of links to chunks, each of which is fetched and verified in turn. When the last chunk is verified and the bytes reassembled, the client has a JSON document that it knows, with cryptographic certainty, is the document the contract's URI committed to. The document's image field contains another ipfs:// URI, and the client verifies the image in the same way. At the end of the process, every byte displayed has been checked against a hash that traces back, link by link, to the string in contract storage. No server was trusted. A gateway may have relayed every block, but the gateway could not have substituted anything without detection. Compare the same process for a token whose tokenURI returns an https:// URL. The client sends a request and receives some bytes, and there is nothing to check them against. If the server returned the wrong document, the client has no way to know. The difference between the two processes is the difference between verification and belief. Mutable names on top of immutable data Content addressing produces immutable names, and sometimes people want names that can change, such as a pointer to the latest version of a website. IPFS provides a mechanism for this called IPNS, the InterPlanetary Name System, in which a name derived from a public key can be updated to point to different CIDs by whoever holds the corresponding private key. The Domain Name System can also be used, by publishing a CID in a DNS text record. These mechanisms are useful for websites and harmful for NFT metadata, and the reason is now familiar. An IPNS name is a mutable reference controlled by a key holder, exactly like a base-URI setter in a contract. A token whose URI is an IPNS name has content that can change whenever the key holder wishes. It has the vocabulary of IPFS and the trust model of a web server. A buyer checking whether a token is "on IPFS" needs to check that the URI uses /ipfs/ or ipfs:// with a CID, not /ipns/ with a key or domain name. What IPFS guarantees, and what it does not It is now possible to state precisely what content addressing provides. It provides integrity: any client that verifies blocks against the CID can be certain that the data it has is the data the CID names, regardless of where it came from. It provides location independence: any peer, gateway or archive holding the data can serve it, so the failure of one host need not mean the loss of the data. And it provides deduplication and verifiable structure: one root CID can commit to an entire collection. It does not provide persistence. IPFS is a protocol for naming and moving data, not a storage service. When a node adds a file, the file is stored on that node. Other nodes that fetch it keep a temporary copy in a cache that is periodically cleared by garbage collection. Nothing in the protocol obliges any node to keep any data. If every node that holds a CID's blocks deletes them or goes offline, the CID still exists as a name, and it still proves what the data was, but the data cannot be retrieved. The name is immortal; the content is not. This is the most important fact in the book, and the source of the most widespread misunderstanding about NFTs. A token whose metadata is "on IPFS" is protected against being changed. Whether it is protected against being lost depends entirely on who has committed to keep it, which is the subject of the next chapter. Chapter 7: Keeping the Bytes Alive Content addressing solves integrity. It turns every reference into a promise about exact bytes that anyone can check. What it cannot do is make anyone keep those bytes. Persistence has to be arranged, and arranging it is a question of incentives, money and institutions as much as of technology. This chapter examines the options in turn: pinning on IPFS, contracted storage on Filecoin, prepaid permanent storage on Arweave, and storing the data on the blockchain itself. Each answers the question "who keeps the bytes, and why?" differently. Pinning On an IPFS node, data is either pinned or cached. Pinned data is kept indefinitely, exempt from garbage collection. Cached data, which the node fetched or relayed in the ordinary course of operation, is deleted when the node needs space or when a periodic clean-up runs. To keep a collection available, someone must pin its root CID on at least one node that is reliably online and announcing itself as a provider of that content. Pinning a root CID pins every block beneath it, so one command can preserve an entire collection. The person or organisation doing the pinning carries the cost: disks, bandwidth, electricity and the labour of keeping a server running. A creator can run their own node, which gives full control and carries the same business-continuity risk as running a web server; when the creator stops, so does the pin. Many projects instead use commercial pinning services, which run large IPFS clusters and pin content on behalf of customers for a monthly fee. Pinata, founded in 2018, is the best known of these in the NFT world; others offer similar services, and the IPFS project defines a common Pinning Service API so that clients can talk to different providers in the same way. A pinning service is, in effect, a hosting company for content-addressed data. That makes it vastly better than a hosting company for location-addressed data, because the service cannot alter the content and because anyone else can pin the same CID without coordination. But the availability it provides lasts exactly as long as someone pays the bill and the service stays in business. A pin is a subscription, not a deed. It is also invisible from the chain: nothing in the contract or the metadata records who is pinning a CID or until when, so a buyer cannot tell from the token itself whether its content is held by five reliable parties or by none. The only test is to try retrieving it, from several sources, and to ask the issuer for the plan. The history of free pinning illustrates the point. In 2021 Protocol Labs launched NFT.Storage, a service offering free storage for NFT data, pinned on IPFS and backed by deals on the Filecoin network. It was enormously popular and absorbed a great deal of NFT content. Free storage at that scale was a subsidy, and subsidies end. The service later stopped accepting uploads to its original free tier and now offers long-term preservation in exchange for a one-time fee. Content uploaded under the old arrangement had to be reviewed by its owners for where it would continue to live. Nothing was lost through malice; the lesson is simply that the party paying for storage determines how long it lasts, and if nobody is paying, eventually nobody will be storing. The structural advantage of content addressing for pinning is that persistence can be redundant and uncoordinated. A creator, a pinning service, a collector, a marketplace and a museum can all pin the same CID independently. None of them needs permission from the others, and none of them can corrupt the content. A collection pinned by five unrelated parties survives the failure of any four. For the collector, this is the most important practical fact in the book: you can pin the content of what you own yourself, cheaply, and in doing so you make your asset less dependent on anyone else. Filecoin: storage by contract Filecoin, also developed by Protocol Labs and launched on its main network in October 2020, adds an economic layer to content-addressed storage. It is a blockchain whose participants, called storage providers, are paid to store data and must prove, continuously and cryptographically, that they are doing so. A client makes a storage deal with a provider, specifying the data, identified by a content identifier, the price and the duration. The provider seals the data in a process that produces a unique encoded replica, and then submits periodic proofs to the Filecoin chain. A Proof-of-Replication shows that the provider has created a distinct physical copy, and recurring Proofs-of-Spacetime show that the copy is still held over time. Providers put up collateral in Filecoin's token, which is forfeited if proofs are missed. The arrangement converts a pinning service's promise into a contract enforced by a blockchain. Filecoin deals are finite. They run for fixed terms and must be renewed to continue. Retrieval, getting data back out quickly, has historically been a weaker point than storage, and many Filecoin-backed services pair storage deals with IPFS pinning for fast access. Filecoin is therefore best understood as a verifiable, economically incentivised backstop: strong evidence that copies exist, for a known period, held by parties with money at stake. It still requires that someone keeps renewing deals and paying for them. Without renewal, the data is subject to the same eventual fate as an expired pin. Arweave: paying once Arweave, launched in 2018, takes a different approach. Rather than paying for storage over time, a user pays once, at upload, and that fee is intended to fund storage permanently. The mechanism rests on an endowment. Part of each upload fee is placed into a pool that is released gradually to miners over time. Miners are rewarded for producing blocks, and to produce a block a miner must prove access to randomly selected pieces of previously stored data, so miners who store more of the network's history have better chances of earning rewards. The design depends on an assumption about the future: that the cost of storing a given amount of data will keep falling, as it has for decades, so that a finite endowment can pay for storage indefinitely. Arweave's design documents set the fee conservatively, aiming to fund at least two centuries of replicated storage under pessimistic assumptions about how quickly costs decline. Data on Arweave is addressed by transaction identifiers, which are derived from hashes, and metadata URIs can use the ar:// scheme or an Arweave gateway URL. The same gateway caveats from Chapter 6 apply: a URL hard-coding a particular gateway's domain is a location reference, and a client that does not verify returned data is trusting the gateway. Arweave's model offers the most direct answer to the persistence problem short of putting data on the main chain. Its guarantee is economic and assumption-laden: it depends on storage costs continuing to fall, on the network continuing to attract miners and on the value of its token remaining sufficient to make mining worthwhile. Those are reasonable bets, but they are bets, and anyone evaluating the claim of "permanent" storage should understand what it rests on. Arweave has become widely used for NFT metadata, particularly on the Solana blockchain, where the Metaplex token-metadata standard encouraged it. On-chain storage The strongest form of persistence is to put the data in the same replicated state that holds the ownership record. If every full node of the blockchain must hold the data in order to validate the chain, then the data lasts exactly as long as the chain does, and no separate arrangement is needed. Tokens built this way are usually called on-chain NFTs. On Ethereum, contract storage is costly, as Chapter 1 showed: about 20,000 gas per 32-byte word. A cheaper technique stores data as the bytecode of a separate contract, since deploying code costs 200 gas per byte, and reading it back is efficient; this pattern is often called SSTORE2 after a widely used library. Each contract's code is limited to 24,576 bytes, so larger assets are split across several contracts. Even so, a single high-resolution photograph would be expensive, and a video impractical. The on-chain NFTs that work therefore exploit compression, generation and vector graphics. Larva Labs' Autoglyphs, released in 2019, generate their images entirely from code in the contract. Loot, released in August 2021 by Dom Hofmann, returns its metadata from tokenURI as a data: URI containing base64-encoded JSON, whose image is itself a data: URI containing an SVG, all assembled by the contract at call time. Art Blocks stores the generating script for each of its projects on Ethereum and gives each token a hash that seeds the script; the image is rendered by running the script, and the collection's software libraries have increasingly been registered on the chain as well. The CryptoPunks data contract described in Chapter 1 stores compact pixel data and renders it on request. A rough calculation shows why on-chain storage is reserved for compact assets. Take a collection of one thousand vector images, each a five-kilobyte SVG file, five megabytes in all. Stored as contract bytecode at 200 gas per byte, the code deposit alone comes to one billion gas, before counting the cost of sending the data in transactions, which adds a further sixteen gas per non-zero byte, and the fixed overhead of deploying a few hundred storage contracts. At an assumed gas price of 10 gwei, one billion gas is ten ether. The figure scales linearly with data: a collection of one thousand 500-kilobyte photographs would cost about a hundred times as much. Gas prices on Ethereum have varied by more than an order of magnitude in each direction over its history, so the result in currency can swing widely, but the ratio between compact generative art and photographic media is fixed by the arithmetic. Rollups and other networks that execute Ethereum's virtual machine offer much cheaper storage, with the different security assumptions described in Chapter 1. The arithmetic explains why the most successful on-chain projects store the recipe rather than the meal. A generative script of a few kilobytes and a 32-byte seed per token can reproduce images whose rendered files would be orders of magnitude larger. Storing the recipe on chain and the rendered images on IPFS or Arweave gives the integrity and permanence of the chain to the irreducible core of the work, while the convenient forms live where storage is cheap. Not all on-chain data is equally permanent. Contract storage and contract code are part of Ethereum's current state, which every full node must hold. Transaction input data and event logs, by contrast, are part of the chain's history, and Ethereum's client developers have been implementing a long-planned change, set out in a proposal known as EIP-4444 and usually called history expiry, that allows ordinary nodes to discard old history and rely on specialised archives to keep it. Data posted in the "blobs" introduced by the March 2024 Dencun upgrade is deleted by nodes after roughly eighteen days and was never meant for permanent storage. A design that hides an image in calldata or an event, reasoning that it is "on the chain", may find that it is on the chain only in the sense that someone, somewhere, keeps an archive. Bitcoin's Ordinals, introduced by Casey Rodarmor in January 2023, store files in the witness data of Bitcoin transactions, and associate them with individual satoshis through an ordering convention. The files are part of every unpruned full node's copy of the Bitcoin blockchain. The model is fully on-chain in that sense, but it relies on a convention and dedicated indexing software, outside Bitcoin's own rules, to define which inscription belongs to whom. Comparing the options Each of these approaches answers the persistence question with a different party and a different incentive. Table 4 summarises them. Table 4. Persistence options for NFT content, by who keeps the data and why. Option Who keeps the data Why they keep it Main residual risk Self-run IPFS node Creator or collector Own interest Node owner stops IPFS pinning service Commercial provider Ongoing payment Payment lapses or firm closes Filecoin storage deal Storage providers Paid deal, collateral at stake Deal not renewed Arweave Network miners Endowment-funded rewards Economic assumptions fail On-chain state Every full node Required to validate chain Cost limits asset size The table makes a pattern visible. As the guarantee strengthens, the obligation moves from a single named party to a protocol, and the cost moves from a recurring payment to an upfront one, or to the price of fitting the asset into the chain itself. No option is free of residual risk, but the risks are of very different kinds. A pinning service can fail tomorrow. Arweave's economics would need to break down over a long period. Ethereum's state would have to be abandoned by its entire community. Who should pay Every persistence option ultimately reduces to a question of funding, and the NFT ecosystem has been slow to answer it. In the boom years, storage was treated as an incidental cost of launch, often paid by a free service or a single month's subscription, while the sale itself raised sums many thousands of times larger. The mismatch is striking: an asset sold on the promise of permanence, with nothing set aside to make it permanent. There are several reasonable ways to close the gap. The simplest is for issuers to treat storage as part of the cost of goods: to allocate a defined portion of primary-sale proceeds to prepaid storage, whether an Arweave upload, multi-year Filecoin deals or a prepaid pinning contract, and to say publicly what was bought and for how long. A second is to dedicate part of any ongoing revenue, such as creator royalties where they are still paid, to renewal. A third is collective: holders of a collection can fund pinning jointly, which content addressing makes safe, since no single funder controls the content. A fourth is institutional: museums, libraries and archives that collect digital art can add pinning and storage deals to their ordinary preservation budgets, as some have begun to do with born-digital works generally. What these approaches share is that they make persistence somebody's explicit job, with money attached and a time horizon stated. That is the minimum a buyer should look for. An answer of "it's on IPFS" with no named party keeping it is not a persistence plan; it is a description of the naming scheme. Redundancy as the practical answer For most collections, the right answer is not one of these options but several in combination. The content is content-addressed, with the same CID usable on any IPFS node. It is pinned by the creator and by at least one commercial service, so that it is fast to retrieve. It is backed by Filecoin deals or uploaded to Arweave, so that a provider with economic incentives holds a copy independently of any single company. For small or generative assets, the essential data is placed on chain. And collectors are encouraged, and given the means, to pin what they own. Redundancy works because content addressing makes copies interchangeable and self-verifying. Ten copies of a location-addressed file on ten servers are ten separate claims, any one of which may differ from the others. Ten copies of a content-addressed file are the same file, and any one of them can be checked against the name recorded on the chain. That is why the integrity guarantee of Chapter 6 and the availability arrangements of this chapter must be read together. Integrity makes redundancy safe. Redundancy makes integrity worth something. The final chapter turns these principles into a design: how a contract, a publication process and a persistence plan can be put together so that the token's guarantees reach all the way from the storage slot to the file. Chapter 8: An Architecture That Holds The previous chapters took the NFT apart. This one puts it back together. The aim is a design in which every guarantee the token claims is actually delivered by something: by consensus, by a hash or by a party with a reason to keep doing its job. The chapter follows a single hypothetical collection from preparation to long-term care, and then turns the same logic around into an audit that a buyer can apply to any token. The collection is imaginary; the techniques are standard and in use. The working example Imagine an artist releasing a collection of one thousand generative images, each with a set of attributes, plus one longer animated work that will be issued in an edition of fifty. The artist wants the images to be fixed forever once revealed, wants the edition to be strictly limited, and wants collectors to be able to verify everything without trusting the artist, the platform or any marketplace. The artist is not wealthy enough to store a thousand full-resolution images on Ethereum, but can afford a modest one-time outlay for storage. The architecture has four layers: the media, the metadata, the contracts and the persistence plan. Each layer is designed so that the layer above it can commit to it by hash. Preparing the media The first decisions are about the files themselves, because the files are what everything else will point to. Formats should be chosen for longevity as well as appearance. PNG for still images and MP4 with a widely supported codec for video are safe choices, because they are openly documented and implemented in every browser and operating system. If the animated work is interactive and delivered as HTML, it should be self-contained: every script, font and data file it needs should be bundled into the same directory and referenced by relative path, so that the whole work is captured by one content identifier. A work that loads a library from a public content-delivery network has, as Chapter 5 showed, an undeclared dependency on a location reference. Bundling removes it. The artist should also decide what "the work" is at the level of bytes. If the generative images are produced by code, the code and its seeds may be a more fundamental and more compact representation than the rendered images. Publishing the code and seeds alongside the images, also content-addressed, gives future custodians a way to regenerate the work if every copy of a rendered file is somehow lost, and lets anyone confirm that the rendered images match what the code produces. The media files are placed in one directory, the animated work in another, and both are added to IPFS. Using version-1 CIDs and recording the exact tool and settings used makes the result reproducible: anyone holding the same files and the same settings will compute the same root CIDs. The output can be exported as CAR files, archives that contain every block of the structure along with its root CID, so that the precise IPFS representation can be uploaded to any pinning service or storage network without being recomputed, and can be archived offline. Building the metadata Next, the artist generates one metadata document per token. Each document's image field is an ipfs:// URI composed of the media directory's root CID and the image's filename. The animation_url for the edition is likewise an ipfs:// URI pointing into its own directory. No field contains an https:// URL to anything the artist controls. If the artist wants a link to a project website, it goes in external_url, which is understood as a convenience rather than part of the asset. Attributes are written into the metadata and, for the generative collection, should be derivable from the code and seed, so that they can be independently verified. A licence document, stating what rights a holder receives, can be content-addressed and referenced from each metadata file, binding every token to a fixed text rather than to a web page that might be edited later. The metadata files are placed in a directory, named by token identifier, and added to IPFS, giving a single root CID for all one thousand documents. For the edition, which will use ERC-1155, the metadata file's name must follow that standard's sixty-four-character lower-case hexadecimal convention so that clients substituting {id} into the template find it. At this point a chain of hashes exists: one CID for the metadata directory, which commits to every metadata document, each of which commits by CID to its image or animation. Change any pixel, any attribute or any word of any description, and the metadata root changes. Handling the reveal honestly The artist wants a delayed reveal, so buyers cannot pick rare tokens during the sale. This requires that, for a time, the contract point somewhere other than the final metadata. The honest way to do this is to commit to the final metadata before the sale and to make the assignment of final metadata to token numbers depend on randomness the artist cannot control. Before the sale, the artist publishes the metadata root CID, or a hash of it, in the contract's constructor as a provenance commitment. Because a CID is already a hash of the entire collection, this single value commits to every image and every attribute in a fixed order. During the sale, tokens point to a placeholder. When the sale ends, the contract obtains a random offset from a verifiable random function or from a future block value fixed in advance, and the mapping from token number to metadata file becomes the token number plus the offset, modulo the collection size. The contract then sets the base URI, but only if the CID being set matches the one committed in the constructor. The setter cannot be used to install anything else. This design narrows the reveal from a trusted act to a verified one. The artist cannot change which images exist, because they were committed in advance. The artist cannot choose which token gets which image, because the offset was not under the artist's control. And the base URI cannot become anything other than the committed root. Writing the contracts The contract layer must close the remaining gap identified in Chapter 5: the reference itself must not be changeable after the reveal. For the one thousand unique images, an ERC-721 contract is appropriate. Its tokenURI function returns ipfs://, the committed metadata root CID, a slash and the shifted token number. After the reveal, there is no function in the contract that can change the root. If the contract supports EIP-4906, it emits a single BatchMetadataUpdate event at reveal time so that marketplaces refresh their caches, and never again. It may also emit OpenSea's PermanentURI event for each token as a courtesy, but the guarantee comes from the absence of a setter, not from the event. The contract should not be deployed behind an upgradeable proxy. If the artist wants the safety of being able to fix a bug before launch, the right tool is testing and an independent audit, not an upgrade key that persists for the life of the collection. Administrative functions that remain, such as withdrawing sale proceeds, should not touch ownership or metadata. Where some ongoing administration is unavoidable, the key should be held in a multi-signature wallet with a published list of signers, and every power the key holds should be documented. For the edition of fifty, an ERC-1155 contract is appropriate, but the standard's silence about supply must be filled by code. The contract records a maximum supply for the type, fixed at deployment, and its minting function reverts if that maximum would be exceeded. After the fifty are minted, minting for that type is permanently closed. The uri function returns a template containing the metadata directory's CID and the {id} placeholder, and no function can change it. Because ERC-1155 defines a URI event for changes, the absence of any code path that would emit it is itself evidence of immutability that an auditor can check. The artist might go further for the generative collection and place the generating code and seeds on chain, using code-as-storage techniques, so that the collection can be regenerated from the blockchain alone even if every off-chain copy is lost. For one thousand images whose generator is a few kilobytes of code, that is affordable and turns the off-chain media into a convenience cache for something the chain itself preserves. The persistence plan With integrity sealed, availability must be provided deliberately, by more than one party with more than one kind of incentive. The artist pins the media and metadata roots on a self-operated IPFS node, for direct control, and with at least one commercial pinning service, for reliability. The same CAR files are used to obtain Filecoin storage deals, or uploaded to Arweave, so that a copy exists under an arrangement that does not depend on the artist continuing to pay a monthly bill. Where budget allows, both. The CAR files and the generating code are also kept offline, on media the artist controls, as a last resort. The artist publishes the root CIDs, the CAR files' hashes and instructions for pinning them, and encourages collectors to pin what they hold. Some collectors will. A museum or archive acquiring a token can pin the whole collection as a matter of course, since content addressing means its copy is exactly as authoritative as the artist's. The plan should be written down and published, with dates. If a pinning subscription is due for renewal or a Filecoin deal will expire, those facts are part of what a buyer is buying. A collection whose storage is funded until a certain year and then depends on community pinning is a different asset from one funded indefinitely, and both are different from one whose survival depends on the artist remembering to pay. Serving and verifying The last layer is the one users see. The artist's own website and any marketplace display images fetched through gateways for speed. The architecture does not change that, but it changes what the gateway is trusted for. Because every reference is a CID, any client can verify. A well-built viewer uses verified retrieval, fetching blocks or CAR files and checking them against the CID, and falls back from one gateway to another if one fails. A viewer that does not verify still benefits, because a wrong answer from a gateway can be detected by anyone who checks. The artist should resist, where possible, the convenience of returning gateway URLs from the contract. A gateway can be chosen by the viewer at display time. Written into the contract, it becomes a permanent dependency on a company that may not exist in a decade. The buyer's audit The same reasoning can be run in reverse to evaluate any NFT before buying it. The audit follows the chain from the ledger to the file and asks, at each link, whether it is bound by code, by a hash or only by someone's continued good behaviour. 1. Read the contract. Confirm the code is verified on a block explorer, so it can be read. Check whether it sits behind a proxy and, if so, who controls upgrades. List every administrative function and who can call it. 1. Check the URI logic. Find how tokenURI or uri builds its result. Look for any function that changes the base URI or individual token URIs, and whether it has been permanently disabled. 2. Inspect the returned URI. It should begin ipfs:// with a CID, ar://, or data:. An https:// URL to the project's domain means content can change. A gateway URL means a dependency on that gateway. An /ipns/ path means a key holder can change it. 3. Open the metadata. Check that image, animation_url and any other asset fields are themselves content-addressed. For HTML works, check whether the page loads anything from outside its own directory. 4. Check supply. For ERC-1155 types, find whether the contract enforces a maximum. For ERC-721, check whether minting is closed. 5. Check persistence. See whether the CIDs can be fetched through more than one gateway, whether the project publishes a storage plan, and whether you can pin the content yourself. None of these steps requires trusting the seller's description. All of them are answerable from public information, though they reward some technical fluency. Tools that automate parts of the audit exist and will improve, but their value is in executing these checks quickly, not in replacing the logic. Objections to immutability Making content impossible to change has costs, and three objections are raised often enough to deserve answers. The first is that mistakes become permanent. A typo in a description, a wrong attribute or a corrupted image file, once frozen, cannot be fixed. This is true, and the remedy is process rather than mutability: publishing all metadata and media to a test deployment, checking every file against its CID and every attribute against its generator, and inviting independent review before freezing. The reveal design described above already includes a window between commitment and reveal, during which errors found in the committed set can be addressed only by abandoning that commitment openly, which is itself a form of accountability. After freezing, an issuer who discovers an error can publish a correction alongside the token, for example in a separate content-addressed errata document, without altering the original. Archives and museums have always handled errors this way: the record stays as it was, and corrections are added. The second objection concerns illegal or harmful content. If an image turns out to infringe someone's copyright, or contains material that must be removed by law, an immutable architecture seems to make removal impossible. The reality is more nuanced. Content addressing fixes what a name refers to; it does not force anyone to store or serve the content. Pinning services, gateways and marketplaces can and do decline to host or display specific CIDs, and public gateways maintain deny-lists for that purpose. What an immutable architecture prevents is the issuer silently replacing the content for everyone, which is the property buyers need. Removal from public availability remains possible through the ordinary means by which hosts respond to legal demands. The token continues to point at the CID, but the content may become hard to retrieve. That is an uncomfortable outcome for an owner, and it is the right outcome when the content is unlawful. The third objection is that immutability limits creative possibility, closing off works designed to evolve. As Chapter 4 argued, the answer is to make change rule-governed rather than to forbid it. A work that evolves according to code on the chain, drawing on content-addressed components, is as mutable as its artist intended and no more. What the architecture excludes is not change but discretionary change by a party the buyer did not agree to trust. What the design achieves At the end of this process, the collection's guarantees can be stated plainly. Ownership is held by consensus on Ethereum. The link from contract to metadata is fixed by code that cannot change. The link from metadata to media is fixed by hashes. The media's integrity can be checked by anyone with any copy. The media's availability is provided by several independent parties under different incentives, at least one of which does not depend on the artist's continued payments, and the essential generative data is on chain. Every link that matters is either a hash or immutable code, and every copy that matters has someone responsible for it. The design costs somewhat more effort than putting files on a web server and storing a URL. It costs less than the reputational and financial damage of tokens that break, and it costs far less than the alternative of storing everything on chain. Most importantly, it makes the asset's guarantees match its claims. Conclusion The public argument about NFTs has mostly been conducted in the wrong terms. Enthusiasts claimed that blockchains made digital ownership permanent and verifiable. Sceptics replied that an NFT was "just a link to a JPEG". Each side was describing a real feature of the architecture and mistaking it for the whole. The ledger entry is as robust as its advocates said. The link is as fragile as its critics said. What decides whether a particular token is robust or fragile is the chain between them, and that chain is built by people, link by link, with choices that the standards leave open and the marketplaces rarely display. Several consequences follow from taking that seriously. For creators and issuers The first consequence is that the responsibility for an NFT's durability belongs to whoever issues it, and that responsibility can be discharged. Nothing about it is exotic. Content-addressing every asset, writing native ipfs:// or equivalent URIs, removing setters and upgrade keys after launch, committing to final metadata before a reveal and arranging redundant storage under more than one incentive are all well documented and within the reach of any competent team. Issuers who did not do these things in 2021 can be partly excused by the speed of that period and the immaturity of the tools. Issuers who do not do them now are choosing to sell something that can be changed or lost and describing it as something that cannot. That responsibility includes honesty about what remains mutable. Some tokens should change: game items, credentials, tickets, works designed to evolve. The architectural virtue is not immutability for its own sake but correspondence between what is promised and what is enforced. A dynamic token whose change follows readable rules from on-chain state is honest. A "permanent" artwork whose base URI can be reset by an administrator is not, however carefully the administrator behaves. It also includes planning for the issuer's own disappearance. Companies close, artists move on, and brands abandon initiatives. A collection designed so that it continues to function without its issuer, with its content pinned by others, archived in a storage network and verifiable by anyone, is a collection whose owners do not need to trust the issuer's future. That is the whole point of putting ownership on a blockchain in the first place. An NFT that still depends on the issuer's server has adopted the cost and complexity of decentralisation while retaining the dependency it was meant to remove. For platforms and marketplaces The second consequence is that platforms have been concealing the most important property of the assets they sell. A marketplace listing typically shows an image, a name, attributes, a price history and an owner. It rarely shows whether the image is content-addressed, whether the contract can change it, or who is storing it. Buyers cannot price what they cannot see, and the result has been a market in which durable and fragile tokens trade as if they were the same thing. Displaying these properties would be straightforward. The checks described in Chapter 8 can largely be automated: whether tokenURI returns a content reference, whether the metadata's asset fields do, whether the contract has a live URI setter or an upgradeable proxy, whether the content is retrievable from more than one source. A platform that surfaced these facts, as prominently as it surfaces rarity scores, would change issuers' incentives quickly. A platform that verified the content it displays, rather than trusting a gateway or its own cache, would stop being part of the problem described in Chapter 5. Platforms that index and cache NFTs are also, whether they intend it or not, among the largest holders of NFT content. A marketplace that pinned, rather than merely cached, the content-addressed assets it lists would add a meaningful layer of redundancy at modest cost. It would also turn a liability, the cached copy that silently diverges from the origin, into a contribution to the asset's survival. For owners The third consequence concerns owners, and it is the least appreciated. Content addressing changes who can preserve an asset. With location-addressed content, only the controller of the location can keep it available; everyone else holds unverifiable copies. With content-addressed content, any owner can pin the exact bytes their token commits to, and their copy is as authoritative as anyone's. The owner is no longer a passive holder of a claim against an issuer. The owner can be a custodian. This is a practical power, not a theoretical one. Pinning a collection's root CID on a personal node or a paid service costs little for typical NFT media. An owner who cares about an asset can make it materially more durable by doing so, and owners collectively, if enough of them pin, can keep a collection alive long after its issuer has gone. For high-value works, it would be reasonable to expect collectors and institutions to treat pinning and archiving as part of ownership, the way they treat insurance, conservation and climate control for physical art. The same logic sets a standard for due diligence. An owner who does not check whether a token's content is fixed and kept has bought a ledger entry and a hope. The audit in Chapter 8 takes less time than most people spend reading a listing's description. Owners who skip it are relying on precisely the trust that the technology was supposed to make unnecessary. What the technology does and does not do The deepest lesson of NFT architecture is about the limits of cryptography in preserving culture. A hash can bind a name to bytes with a strength no legal instrument can match. It can let strangers verify a file without trusting each other. It can let a thousand independent copies be one copy. What a hash cannot do is make anyone care. Persistence is ultimately a social and economic fact: some person or institution, with resources and a reason, keeps the bytes and keeps serving them. Protocols such as Filecoin and Arweave can structure that reason as payment and collateral; blockchains can make it a condition of participating in the network. None of them removes the need for someone, somewhere, to do the work. Seen this way, the NFT is less a revolution in ownership than a careful division of labour. The ledger handles who owns what, with a strength that no previous digital system offered. Content addressing handles what exactly is owned, with a precision that the web never had. Storage, whether by pinning, contract or endowment, handles whether it still exists, and that part remains as dependent on human commitment as libraries and archives have always been. Durable digital ownership comes from building all three parts and connecting them with hashes. Where any part is missing, the token still records an owner, but there may be nothing reliable left for it to point to. Glossary CAR file. Content Addressable aRchive: a file that packages a set of IPFS blocks together with their root CID, so the exact structure can be moved, uploaded or verified as a unit. CID (content identifier). The self-describing name IPFS gives to data, built from a hash of the data plus information about the hash function, data format, version and text encoding. Content reference. A reference that names specific bytes, usually by hash, so that any retrieved data either matches it or is rejected. ERC-165. An Ethereum standard by which a contract declares, through supportsInterface, which interfaces it implements. Gateway. An HTTP server that retrieves IPFS or Arweave content and returns it to ordinary web clients; unless the client verifies the result, it must trust the gateway. Indexer. An off-chain service that reads blockchain events to reconstruct ownership and metadata, and serves the results to wallets and marketplaces. IPNS. The InterPlanetary Name System: mutable names, controlled by a private key, that can be repointed to different CIDs. Location reference. A reference, such as an https:// URL, that names a place; whatever is at that place when asked is what is returned. Merkle DAG. A graph of data in which every link is the hash of the linked node, so the root hash commits to the entire structure. Operator approval. Permission, granted with setApprovalForAll, for another address to transfer all of an owner's tokens in a contract. Pinning. Instructing an IPFS node to keep data permanently rather than treating it as a cache subject to deletion. Provenance hash. A hash of a collection's final content published before a sale or reveal, so that the content cannot later be altered without detection. Notes 1. Moxie Marlinspike, "My first impressions of web3," moxie.org, 7 January 2022. Source for the context-dependent NFT, its removal by OpenSea, its disappearance from the wallet and the observation that NFTs lack a hash commitment to the data at the URL (Introduction, Chapters 1 and 5). 1. William Entriken, Dieter Shirley, Jacob Evans and Nastassia Sachs, "ERC-721: Non-Fungible Token Standard," Ethereum Improvement Proposals no. 721, created 24 January 2018, eips.ethereum.org/EIPS/eip-721. Source for function and event signatures, interface identifiers, the receiver check and the metadata JSON schema (Chapters 1, 2 and 4). 1. Witek Radomski, Andrew Cooke, Philippe Castonguay, James Therien, Eric Binet and Ronan Sandford, "ERC-1155: Multi Token Standard," Ethereum Improvement Proposals no. 1155, created 17 June 2018, eips.ethereum.org/EIPS/eip-1155. Source for batch functions, events, receiver magic values, interface identifiers and the {id} substitution rule (Chapters 3 and 4). 1. Christian Reitwiessner et al., "ERC-165: Standard Interface Detection," Ethereum Improvement Proposals no. 165 (Chapter 2). 1. Zach Burks, James Morgan, Blaine Malone and James Seibel, "ERC-2981: NFT Royalty Standard," Ethereum Improvement Proposals no. 2981 (Chapter 2). 1. "ERC-4906: EIP-721 Metadata Update Extension," Ethereum Improvement Proposals no. 4906. Source for the MetadataUpdate and BatchMetadataUpdate events (Chapters 2 and 8). 1. OpenSea developer documentation, "Metadata Standards," docs.opensea.io. Source for the conventional attributes, animation_url and external_url fields and the PermanentURI freezing convention (Chapters 4 and 8). 1. Pew Research Center, "When Online Content Disappears," May 2024. Source for the share of 2013 web pages no longer accessible a decade later (Chapter 5). 1. Cloudflare, "Cloudflare's public IPFS gateways and supporting Interplanetary Shipyard," The Cloudflare Blog, 2024. Source for the transition of Cloudflare's public gateway traffic and the August 2024 end of its gateway hostnames (Chapters 4 and 5). 1. Juan Benet, "IPFS - Content Addressed, Versioned, P2P File System," arXiv:1407.3561 (2014) (Chapter 6). 1. IPFS documentation, "Content addressing and CIDs," docs.ipfs.tech. Source for CID versions, encodings and the Qm and bafy prefixes (Chapter 6). 1. Protocol Labs, "Filecoin: A Decentralized Storage Network" (2017), filecoin.io. Source for Proof-of-Replication and Proof-of-Spacetime (Chapter 7). 1. Arweave yellow paper and protocol documentation, arweave.org. Source for the storage endowment model (Chapter 7). 1. Dipanjan Das, Priyanka Bose, Nicola Ruaro, Christopher Kruegel and Giovanni Vigna, "Understanding Security Issues in the NFT Ecosystem," Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security (CCS 2022). Background on marketplace and metadata-hosting weaknesses (Chapters 4 and 5). 1. "EIP-4444: Bound Historical Data in Execution Clients," Ethereum Improvement Proposals no. 4444; and "EIP-4844: Shard Blob Transactions," Ethereum Improvement Proposals no. 4844. Source for history expiry and the short retention of blob data (Chapter 7). Further Reading The EIP texts for ERC-721 and ERC-1155 (eips.ethereum.org). Short, readable and more informative than most commentary on them. The rationale sections explain why the authors made the choices they did, including the decision to leave metadata storage open. Moxie Marlinspike, "My first impressions of web3" (2022). The clearest short account of why a token that stores a URL does not commit to content, and of how much of the ecosystem depends on centralised APIs. Juan Benet, "IPFS - Content Addressed, Versioned, P2P File System" (2014). The original design paper. Parts have been superseded by later implementations, but its account of Merkle DAGs and self-certifying names remains the best introduction to the ideas. The IPFS documentation, especially "Content addressing and CIDs," "Persistence, permanence and pinning" and the pages on gateways (docs.ipfs.tech). Practical and current, with a useful candour about what IPFS does not guarantee. The Multiformats project (multiformats.io). The specifications for multihash, multicodec and multibase, for readers who want to decode a CID by hand. Protocol Labs, "Filecoin: A Decentralized Storage Network." The whitepaper describing storage deals and the proofs that underpin them. The Arweave yellow paper. The source for Arweave's endowment model and the economic assumptions behind its claim of permanence. Das, Bose, Ruaro, Kruegel and Vigna, "Understanding Security Issues in the NFT Ecosystem" (CCS 2022). A systematic academic study of marketplace and token weaknesses, including how and where metadata is hosted. OpenZeppelin Contracts documentation for ERC-721 and ERC-1155 (docs.openzeppelin.com). The most widely used reference implementations, with discussion of extensions such as URI storage, and a good place to see the standards turned into working code. Pew Research Center, "When Online Content Disappears" (2024). Not about NFTs, but the best recent measurement of link rot on the ordinary web, and a sobering baseline for any asset that depends on a URL. Hashtags: #NonFungibleTokens #NFTArchitecture #NFTDevelopment #BlockchainTechnology #EthereumBlockchain #SmartContracts #Solidity #ERC721 #ERC1155 #EthereumVirtualMachine #NFTMetadata #IPFS #InterPlanetaryFileSystem #DecentralizedStorage #ContentAddressing #ContentIdentifiers #CryptographicHashing #MerkleDAG #MetadataIntegrity #ImmutableMetadata #NFTMinting #NFTSmartContracts #OnChainNFTs #OffChainMetadata #IPFSPinning #Filecoin #Arweave #DigitalOwnership #NFTSecurity #Web3Architecture

Download the Book (PDF): Introduction On a Tuesday night in October, a sophomore sits on the floor of her chapter house with a laptop balanced on her knees. Chapter meeting ran long. After it came a committee check-in, then a group chat about Saturday's philanthropy event that is still buzzing at 11:40 p.m. She has an organic chemistry problem set due at nine the next morning, a shift at the campus library on Thursday, and a big sister who has texted twice to ask whether she is coming to the study session that turned into a movie night. She loves these women. She joined because she was lonely in her first semester and this house gave her a place to walk into without feeling like a visitor. She also has not slept more than six hours on a weeknight since recruitment ended, and lately she has felt a flatness in the mornings that she does not have a name for. Nothing in that scene is scandalous. No one is being forced to drink. No one is standing in a basement being yelled at. And yet it contains, in small form, almost everything this book is about: the real pull of belonging, the quiet arithmetic of obligations that were each reasonable on their own, the way sleep becomes the account everyone overdraws, and the uncertainty about whether it is acceptable to say, "I can't tonight." Fraternities and sororities occupy an unusual place in American higher education. They are among the oldest student institutions in the country, and they remain some of the most powerful. For hundreds of thousands of students they supply a ready-made community, leadership roles, alumni networks, service opportunities, and a sense of home at a stage of life when home has just been left behind. They are also where some of the worst harms on campus have happened. Students have died in fraternity houses after drinking rituals. National research has found that roughly three in four members of social fraternities and sororities report experiencing behaviors that meet the definition of hazing. The federal government, after decades of state-by-state legislation, passed a national hazing transparency law in December 2024. Both of those descriptions are accurate, and a useful book about wellness in Greek life has to hold them together. Many writings on the subject pick one. Promotional materials describe a world of sisterhood, brotherhood, scholarship, and service in which risk is an afterthought. Critical journalism, often written in the aftermath of a death, describes an institution so compromised that the only sensible response is to abolish it or stay away. Students who are actually in chapters, or who are thinking about joining, find neither account much help. They need to know how to get the good without paying for it with their health, and how to change the parts that are harmful without being cast out of the thing they came for. The argument of this book The central claim of this book is simple to state and harder to live: the qualities that make fraternity and sorority life valuable are the same qualities that make it risky, and wellness in a chapter depends less on individual willpower than on whether the group has made it safe to say no. Belonging is powerful because it matters to us what the group thinks. That same concern is what makes it so hard to refuse a drink, leave a party at midnight, skip an event that isn't mandatory but somehow feels mandatory, or tell a new member that what is being asked of them is wrong. Loyalty is valuable because it means someone will show up for you; it is dangerous when it means no one will call for help because they are afraid of getting the chapter in trouble. Tradition gives a chapter continuity and identity; it also carries forward practices that no current member chose and many privately dislike. If that is right, then the familiar advice to students ("manage your time better," "know your limits," "just say no") is not wrong but badly incomplete. A member can have excellent time-management skills and still lose sleep to a chapter calendar that assumes everyone is free every night. A member can know her limits perfectly well and still be in a room where refusing a drink carries a social price. What changes outcomes is the combination of personal skill and group norms: individuals who know what they need and how to ask for it, in chapters where asking is treated as ordinary rather than as disloyalty. This is good news, because norms are made by people and can be remade by them. Every chapter has members who set its tone, officers who design its calendar, and new members who notice, more clearly than anyone, what is being asked of them. Most students in a chapter are privately more uncomfortable with dangerous practices than they believe their peers to be. That gap between private belief and perceived norm is one of the best-documented findings in the research on college drinking and hazing, and it is also the opening through which change happens. Who this book is for This book is written first for students: those considering recruitment, new members in their first term, active members trying to balance chapter life with academics and work, and chapter officers who have realized that "wellness chair" is a harder job than it sounded. It is also written for the adults around them. Parents often have little idea what their son or daughter's chapter asks of them week to week and may not know what questions to ask. Advisors, fraternity and sorority life staff, and alumni volunteers have more authority than they sometimes realize and less visibility than they would like. It assumes the reader is intelligent and busy. It does not assume any prior knowledge of the research on hazing, alcohol, sleep, or mental health, and it tries to report that research honestly, including where it is limited. Where it describes students, the scenarios are illustrative composites built to show common patterns; they are not accounts of real individuals. It also tries to respect the variety within Greek life. The word "Greek" covers organizations that differ enormously. Chapters in the North American Interfraternity Conference and the National Panhellenic Conference often have houses, large memberships, and formal recruitment periods. Historically Black fraternities and sororities in the National Pan-Hellenic Council, and culturally based organizations in groups such as the National Association of Latino Fraternal Organizations and the National Multicultural Greek Council, often have smaller chapters, intake rather than recruitment, strong graduate-chapter traditions, and distinctive histories rooted in exclusion from the organizations that came first. Professional and honor fraternities have their own cultures again. Much of what follows applies across all of them, and where the differences matter the text says so. How the book is organized The first chapter looks directly at the double-edged nature of membership: what the evidence says about the benefits of belonging, what it says about the risks, and why both can be true of the same chapter at the same time. The second follows the student through recruitment or intake and the new-member period, the stretch of time when stress is highest, power is most unequal, and patterns are set. Chapters 3 and 4 address the two most serious physical dangers. Chapter 3 covers hazing: what it is, why intelligent people take part in it, what the research by Elizabeth Allan and her colleagues has found, and what the Stop Campus Hazing Act of 2024 now requires of colleges. Chapter 4 covers alcohol and emergencies, with particular attention to medical amnesty and Good Samaritan policies and to the single decision that most often separates a bad night from a death: whether someone calls for help. Chapters 5 and 6 turn to the less dramatic but more universal costs. Chapter 5 is about sleep and time, and about why a chapter calendar is a health document whether anyone treats it as one or not. Chapter 6 is about boundaries: how to decide what you will and will not do, and how to say no in ways that protect both your wellbeing and your relationships. Chapters 7 and 8 move from the individual to the group. Chapter 7 is about bystander intervention inside a chapter, where the people you would need to confront are also your friends. Chapter 8 is written for chapter leaders and those who advise them, and describes what it takes to build a culture in which mental health is treated as ordinary chapter business, drawing on the resources that national organizations, campus offices, and prevention researchers now provide. The conclusion draws these threads into a single test that any member, officer, or parent can apply. A note on health and safety This book discusses alcohol poisoning, hazing injuries, depression, anxiety, and suicidal thinking. It offers general information and practical guidance; it does not replace advice from a physician, counselor, or other professional who knows your situation. Two things are worth saying plainly at the start. If someone is unconscious, cannot be woken, is breathing slowly or irregularly, is having a seizure, or has been injured, call 911 (or your local emergency number) immediately; do not wait to see whether they improve. If you or someone you know is thinking about suicide or is in emotional crisis, in the United States you can call or text 988 to reach the 988 Suicide and Crisis Lifeline at any hour. Most campuses also run a counseling center with after-hours crisis support, and it is worth finding that number before you need it. The rest of the book is about the ordinary weeks, which are where most of the work is done. A chapter that is careful about sleep, respectful of boundaries, honest about mental health, and quick to call for help is not a less loyal chapter. It is a chapter whose loyalty is worth having. Chapter 1: What Belonging Gives and What It Can Cost Consider two first-year students who arrive on the same campus in the same week. The first, a young man from a small town three states away, knows no one. His roommate goes home most weekends. By the fifth week of the semester he is eating dinner alone with his phone, telling his parents things are "fine," and wondering whether he chose the wrong school. The second is a young woman who arrives with a dozen high-school classmates, a crowded social calendar, and a sense that she must be constantly available or be left behind. By the fifth week she is exhausted and anxious, sleeping five hours a night, and saying yes to everything because she is afraid of what saying no might cost. Both might join a fraternity or sorority in the spring. For the first, membership could be the single best decision of his college years: a group of people who expect him at dinner, older members who show him which professors to take, a reason to stay. For the second, it could simply add another layer of obligation to a life already organized around fear of exclusion. Or it could be the place where she finally learns that she can say no and still be loved. Which of these happens depends partly on the students and partly on the chapters they join. That is the starting point for any honest discussion of wellness in Greek life. Membership is not good or bad for health in general. It is a powerful social force, and powerful social forces amplify whatever they are pointed at. The evidence on both sides The need to belong is not a luxury. Psychologists Roy Baumeister and Mark Leary argued in an influential 1995 review that the desire for frequent, positive interactions within stable relationships is a fundamental human motivation, and that its absence is associated with a wide range of negative effects on health and wellbeing. Loneliness among college students is common and persistent. In the 2024–2025 Healthy Minds Study, which surveyed more than 84,000 students at 135 colleges and universities, 52 percent reported high levels of loneliness. That was an improvement from 58 percent in 2022, but it still describes a majority. Fraternities and sororities are, among other things, machines for manufacturing belonging. They give students a defined group, shared rituals, a physical space in many cases, a calendar of events that brings people together without anyone having to organize it from scratch, and an explicit promise that members will look after one another. For students who find it hard to build friendships through classes or residence halls, that structure can be a lifeline. The evidence on outcomes, though imperfect, points in a favorable direction on several measures. A 2014 Gallup survey of more than 30,000 US college graduates, conducted with the National Panhellenic Conference and the North American Interfraternity Conference, found that graduates who had been fraternity or sorority members were more likely than other graduates to be thriving across Gallup's five elements of wellbeing (purpose, social, financial, community, and physical) and more likely to report that they had felt emotionally supported in college. Forty-three percent of members working full time for an employer were engaged at work, compared with 38 percent of other graduates. The same report also acknowledged the serious concerns about hazing, alcohol, and sexual assault in some chapters. More direct evidence on mental health comes from a 2024 report by the Jed Foundation, a national nonprofit focused on the emotional health of teens and young adults. Drawing on data from the Healthy Minds Study, the American College Health Association's National College Health Assessment, and its own survey and concept-mapping research, the report found that fraternity members reported lower rates of depression, anxiety, and serious suicidal thoughts than non-members. In the Healthy Minds data it cited, about 29.8 percent of fraternity members screened positive for depression, compared with 39.1 percent of non-members; 22.5 percent screened positive for anxiety, against 33.2 percent; and 11.8 percent reported serious thoughts of suicide in the past year, compared with 15.5 percent. A 2020 study from the University of Tennessee's Postsecondary Education Research Center, funded by the fraternity industry's own foundation, similarly found that fraternity members reported higher levels of positive mental health. These findings deserve both attention and caution. They are associations, not proof that membership causes better mental health. Students who join are, on average, different from students who do not: they may be more socially confident, more financially secure, or more likely to have been involved in groups in high school. Some of the differences may reflect who joins rather than what joining does. Still, the direction of the evidence is consistent enough that a fair account cannot treat Greek life as simply a hazard to be managed. For many students, it is part of how they stay well. The same body of research that documents benefits also documents risks, and some of them are grave. The most extensive national study of hazing, published in 2008 by Elizabeth Allan and Mary Madden of the University of Maine, surveyed 11,482 students at 53 institutions and conducted more than 300 interviews. It found that 55 percent of students involved in clubs, teams, and organizations had experienced hazing. Among members of social fraternities and sororities, the figure was 73 percent. Hazing is the subject of Chapter 3, but it belongs in any account of what membership can cost. Alcohol is the second major risk. In the 2023 National Survey on Drug Use and Health, 29.3 percent of full-time college students aged 18 to 25 reported binge drinking in the past month. The Jed Foundation report, drawing on National College Health Assessment data, found that between 80 and 87 percent of fraternity and sorority members in the samples it examined reported binge drinking, compared with 64 to 65 percent of non-members. A long-running national study led by Sean Esteban McCabe at the University of Michigan followed high-school seniors into their mid-thirties. Published in 2018 in the Journal of Adolescent Health, it found that men who had lived in a fraternity house for at least one semester had greater odds of binge drinking across young adulthood and higher odds of reporting symptoms of alcohol use disorder at age 35. Women who had lived in sorority houses showed a similar pattern for binge drinking and alcohol use disorder symptoms. Earlier work by the same team found evidence of both selection (heavier drinkers are more likely to join) and socialization (membership is associated with increases in drinking beyond what students brought with them). The Jed Foundation report also found something subtler. Fraternity and sorority members reported greater awareness of campus mental health services than non-members, but while sorority members were more likely to use therapy, fraternity members were not. The benefit of belonging, in other words, did not automatically translate into getting help when help was needed. The report identified shame and secrecy around help-seeking, and norms that equate emotional struggle with weakness, as barriers. There are other costs that rarely make the news. Membership fees and social expectations can place real financial strain on students from lower-income families. The time demands of chapter life, discussed in Chapter 5, can crowd out sleep, study, and paid work. And the experience of rejection during recruitment, discussed in Chapter 2, can be painful in ways that last longer than the few days of the process itself. Why both can be true at once It would be convenient if the benefits and risks belonged to different chapters: good chapters that provide belonging and bad chapters that provide hazing. There is some truth to this. Chapters vary enormously, even within the same national organization on neighboring campuses. But the deeper point is that benefits and risks often come from the same source. Take cohesion. A chapter where members feel intensely bound to one another is a chapter where someone will drive you to the emergency room at 2 a.m., lend you rent money, or sit with you after a breakup. It is also a chapter where members may be reluctant to report wrongdoing for fear of betraying the group, and where the prospect of losing membership feels unbearable. Researchers who study hazing have repeatedly observed that it tends to flourish in groups that prize loyalty and have steep status hierarchies, not in groups that are loosely connected. Take tradition. Traditions give a chapter continuity across generations, and many are genuinely meaningful: rituals that mark commitment, songs, service projects, the passing of symbols from older to younger members. But tradition is also the most common justification for practices that no one would design from scratch. "It's always been done this way" is the sentence that keeps a dangerous practice alive long after everyone involved has privately stopped believing in it. Take shared identity. A strong sense of "we" is protective when it means members look after each other. It becomes risky when it means the chapter's reputation matters more than any one member's safety, or when the group's image, whether as the hardest-partying house or the most polished sorority on campus, creates pressure that individual members struggle to resist. This is why wellness in Greek life cannot be reduced to individual choices, and also why it cannot be reduced to rules imposed from outside. Individual choices happen inside group norms. Rules imposed from outside are followed, evaded, or resented depending on whether they match what members actually value. The most durable improvements come when members themselves decide that the chapter's strengths (loyalty, cohesion, tradition, identity) will be put to work protecting members rather than endangering them. One idea from the research on college drinking helps explain why this is possible. Beginning in the 1980s, researchers including H. Wesley Perkins and Alan Berkowitz found that college students consistently overestimated how much their peers drank and how comfortable their peers were with heavy drinking. Students then adjusted their own behavior toward what they believed was normal. Later studies by Deborah Prentice and Dale Miller at Princeton described a related phenomenon, "pluralistic ignorance," in which most members of a group privately reject a norm but assume that everyone else accepts it, so the norm persists. Hazing research has found the same pattern. Allan and her colleagues have documented that students often believe their peers are more supportive of hazing than they actually are, and that many who participate do so reluctantly while assuming everyone else is enthusiastic. When private discomfort is widespread but invisible, a single member saying out loud what others are thinking can shift the whole room. This book returns to that insight often. It is the reason that the chapter member who quietly leaves a party at midnight, or tells a new member "you don't have to do that," or suggests moving the Thursday meeting so people can sleep, matters more than it might seem. Norms are not fixed facts. They are shared guesses about what other people think, and they change when people provide better information. Different councils, different pressures The costs and benefits of membership do not fall evenly across the fraternity and sorority world, and a book that treated Greek life as one thing would mislead readers in important ways. Large chapters in the North American Interfraternity Conference and the National Panhellenic Conference, especially on campuses where they own or lease houses, tend to be where the most intensive social calendars, the largest parties, and the most formal recruitment processes are found. The research on residential fraternity drinking applies most directly here. So do the time pressures of living where the chapter lives: when your bedroom is upstairs from the chapter room, it is hard to be absent from anything. Organizations in the National Pan-Hellenic Council, the nine historically Black fraternities and sororities often called the "Divine Nine," have a different history and structure. They were founded, beginning in the early twentieth century, partly because Black students were excluded from existing organizations, and they have long emphasized service, scholarship, and lifelong membership through graduate chapters. Chapters on many campuses are small, sometimes only a handful of members, and membership intake rather than open recruitment is the norm. For Black students at predominantly white institutions, these organizations can provide a community of cultural affirmation that is hard to find anywhere else on campus, and that belonging can be protective in ways that general statistics do not capture. At the same time, these organizations have also struggled with hazing. In 1990 the NPHC organizations formally abolished pledging in favor of a membership intake process, yet hazing has persisted underground, and some of the most serious hazing incidents have occurred in small groups operating away from official oversight. Latino, Asian American, and multicultural fraternities and sororities share many of these features: small chapters, strong cultural purpose, and the risk that intense loyalty and secrecy combine in harmful ways. Small chapters face a particular version of the time problem. When a chapter has eight members, every member holds an office, every event needs every person, and the loss of a single member to burnout or academic trouble is felt immediately. Students in these organizations often describe a strong sense of responsibility to the chapter's history and to the graduate members who came before them, which can make stepping back feel like letting down generations of people. None of this changes the central argument. In every kind of organization, the forces that create belonging can also create pressure. But what that pressure looks like, and what protects against it, varies. A wellness practice that works in a 150-member residential chapter may need adapting for a nine-member intake class, and vice versa. Because chapters differ so much, the most useful thing a prospective member can do is gather information about the particular chapter, not the reputation of Greek life in general. Some of this information is now public. Under the Stop Campus Hazing Act, discussed in Chapter 3, colleges must publish a campus hazing transparency report listing student organizations found responsible for hazing violations. Many campuses also post chapter grade-point averages, conduct histories, and accreditation or standards reports from their fraternity and sorority life offices. The rest comes from asking. Prospective members often feel they cannot ask direct questions during recruitment without seeming difficult, but a good chapter should welcome them, and the way a chapter answers is itself informative. Questions worth asking include: How many hours a week do members typically spend on chapter obligations in the first semester, and in later semesters? Which events are mandatory, and what happens if someone misses one? How does the chapter handle a member who is struggling academically or emotionally? Has anyone ever called for medical help at a chapter event, and what happened? What does the new-member period involve, and can I see a schedule? What does membership cost each semester, including dues, social expenses, and apparel? Parents can ask many of the same questions of their student and, where appropriate, of the campus fraternity and sorority life office. They are rarely asked, and they shift the conversation from whether Greek life is good in general to whether this chapter is good for this person. What wellness means here Wellness is a vague word, and it is often used to sell things. In this book it means something specific: the conditions that allow a student to stay physically safe, sleep enough to function, keep up academically, maintain relationships outside the chapter as well as inside it, notice when they are struggling, and get help without fear. It includes the absence of harm (no hazing, no alcohol poisoning, no assault) and the presence of capacity (enough rest, enough time, enough support). A chapter that promotes wellness in this sense is not necessarily a quiet one. It can have a full social calendar, demanding standards, and serious traditions. What distinguishes it is that its demands are designed with members' health in mind, that members can decline without penalty, and that anyone in trouble can ask for help, or have help called for them, without hesitation. The rest of this book describes what that looks like in practice, beginning where every member's experience begins: the process of being chosen. Chapter 2: Recruitment, Intake, and the New-Member Months Almost every member's story in a fraternity or sorority begins with being evaluated. That is worth pausing on, because it shapes much of what follows. Before a student has any rights as a member, any friends in the chapter, or any sense of how things work, they spend days or weeks being assessed by people who hold the power to admit or refuse them. Then, if admitted, most spend another stretch of weeks or months as new members, in a position that is explicitly junior. The psychology of that period, the stress it creates, the patterns it sets, and the vulnerability it produces, explains a great deal about why wellness in Greek life is harder than it looks from the outside. How students come in The processes differ by council and by campus, but three broad patterns cover most students. In National Panhellenic Conference sororities on many campuses, recruitment is highly structured. Potential new members register in advance, often pay a fee, and move through a series of rounds over several days, visiting chapters and gradually narrowing their options while chapters narrow theirs. The process runs on mutual selection: at the end, a matching system pairs students' ranked preferences with chapters' ranked lists. Students are typically assigned a recruitment counselor, an upper-level sorority member who temporarily disaffiliates from her own chapter so she can guide potential new members impartially. On large campuses, primary recruitment can involve thousands of students, long days in formal clothes, and intense emotion, including for the members who spend a week or more before and during it rehearsing, decorating, and talking to hundreds of strangers. Interfraternity Council recruitment is usually less formal. Students attend events, meet members, and may receive bids (formal invitations to join) at any point over a recruitment period, and many chapters recruit continuously throughout the year. The absence of structure has advantages, but it also means less oversight, and events that serve as recruitment can blur into parties. National Pan-Hellenic Council organizations, and many culturally based fraternities and sororities, use a membership intake process instead of open recruitment. Interested students typically attend an informational meeting, submit an application with academic records and letters of recommendation, may be interviewed, and if selected complete a defined education program before initiation. Intake is often less visible and less frequent. Some chapters take a new line only once every year or two. Whatever the format, the experience shares a core feature: a student who wants something very much is being judged, often by people slightly older than themselves, according to criteria that are rarely fully explained. The stress of being chosen Recruitment stress is sometimes dismissed as trivial: it is only a few days, and there are other ways to find friends. But the research on social rejection suggests that it is not trivial at all. In a well-known 2003 study published in Science, Naomi Eisenberger, Matthew Lieberman, and Kipling Williams scanned participants' brains while they played a simple ball-tossing computer game in which other players, apparently, stopped throwing them the ball. Being excluded activated some of the same brain regions associated with the distress of physical pain. Later research has debated the details, but the core observation, that social exclusion hurts in a way that is more than metaphorical, has held up well. Recruitment concentrates that experience. A potential new member may be invited back to fewer chapters than she hoped, may be dropped by the chapter she most wanted, and may compare her results with friends' results in real time on social media. A student going through fraternity rush may attend events for a week and receive no bid while his roommate receives several. Students in intake processes may wait weeks to learn whether they were selected, often without explanation either way. Several features of modern recruitment intensify this stress. The first is visibility. What used to happen privately now often plays out publicly, with students posting outfits, bid-day videos, and reactions. The attention that sorority recruitment at some southern universities has received on social media in recent years has turned what was once a campus ritual into a national spectacle. For students who do not get the outcome they wanted, public visibility adds humiliation to disappointment. The second is timing. Recruitment often happens in the first weeks of a semester, sometimes before classes begin, when students have few other sources of social support on campus. A first-year student who is rejected in week two has not yet had time to build the friendships that would buffer the blow. The third is physical depletion. Formal recruitment can involve very long days, little food, standing for hours, and short nights. Both potential new members and active members working recruitment routinely report exhaustion. Sleep loss, as Chapter 5 describes, amplifies negative emotion and reduces the capacity to regulate it. A disappointment that would be manageable after a good night's sleep can feel devastating after four nights of five hours each. The stress of recruitment is not confined to those being evaluated. Active members working formal recruitment often spend the preceding week in intensive preparation, practicing conversations, songs, and presentations late into the night, and then spend recruitment itself on their feet for twelve hours or more a day, making hundreds of brief conversations sound warm and personal. They then sit through long selection sessions in which the chapter discusses and votes on potential members, sometimes past midnight. These sessions carry their own emotional weight: members may find themselves advocating for a friend's younger sister, disagreeing sharply with chapter sisters, or uncomfortable with the way some candidates are being discussed. Members who have experienced disordered eating, anxiety about appearance, or social anxiety sometimes find that recruitment, with its focus on presentation and first impressions, reactivates those struggles. Chapters that take wellness seriously build rest into the recruitment schedule, make food available and expected, set firm end times for selection sessions, and let members with health concerns take lighter roles without explanation. Intake processes create a different kind of strain. Because intake is less frequent and more selective, interested students may wait a long time for an opportunity and invest heavily in it. The process is often less transparent to outsiders, which can make it harder for students to judge what is legitimate. Some students report informal expectations before any official process begins, such as being seen at events, doing favors for members, or proving interest in ways that are not written down anywhere. National organizations in these councils explicitly prohibit such unofficial "pre-pledging," and a student who encounters it should understand that it is not part of the legitimate process, however common it may be locally. What helps when the answer is no There is no way to make rejection painless, but there are ways to make it less damaging. The most important is to understand what a recruitment outcome is and is not. It is the result of a brief, crowded process in which chapters make quick judgments based on limited information, partly shaped by factors that have nothing to do with the individual: chapter size targets, the number of legacies, a chapter's sense of its own "type," and sometimes luck. It is not a verdict on a person's worth, likability, or future. That is easy to say and hard to believe at 11 p.m. on the night results come out, which is why it helps to hear it before recruitment begins and to decide in advance how you will interpret a disappointing result. The second is to maintain other sources of belonging during recruitment. Students who treat recruitment as the only route to a social life have the most to lose. Keeping up with a club, a team, a job, a faith community, or friends from a residence hall means that a bad result is a setback rather than a catastrophe. The third is to talk to someone. Recruitment counselors exist in part for this reason, and campus counseling centers often see a rise in visits during and after recruitment. There is nothing weak about seeking help to process a painful experience. Parents can help most by listening rather than immediately fixing, and by resisting the temptation to treat the outcome as a family verdict. A parent who was a member of a particular organization can unintentionally add pressure by making clear how much they hoped their child would join it. Finally, students should know that many paths remain open. Many students who are not matched in formal recruitment find a chapter through continuous open bidding later in the year, join a different council's organization, or discover that their social life takes a different shape entirely. Some who were initially disappointed later say it was the best thing that could have happened. For active members, the lesson runs the other way. Chapters make choices that affect people's wellbeing, and they can make those choices with care. That means avoiding cruelty in how potential members are discussed, keeping conversations about potential members off social media, and remembering that the person being evaluated is a nervous eighteen-year-old in her third week of college, not a product. The new-member period and why it matters Once a student accepts a bid or is selected for intake, they enter the new-member period, sometimes still called pledging, though most national organizations have formally abandoned that term. This period typically involves education about the organization's history, values, and rituals; meetings; community service; academic expectations; and social events meant to integrate new members into the chapter. Many national organizations now set limits on its length and specify what it may include. It is also the period of greatest risk, for a simple structural reason. New members are not yet full members. They want very much to be initiated, and they know that others in the chapter will have a say in whether that happens. They are usually younger, newer to campus, and less sure of what is normal. They are, in the language that prevention researchers use, in a position of reduced power, and that asymmetry is precisely what hazing exploits. The national hazing research reviewed in Chapter 3 found that students frequently experienced hazing during this period and rarely identified it as such. Even where there is no hazing, the new-member period can be one of the most demanding stretches of a student's college career. Weekly new-member meetings, study hours, service requirements, chapter events, big-little activities, and informal expectations to "be around" can add up to many hours a week, layered on top of a first- or second-year academic load that the student is still learning to manage. Because new members are eager to show commitment, they rarely push back. Two other features of the new-member period deserve mention. The first is money. New-member fees, initiation fees, dues, required apparel, formal events, and gifts exchanged in big-little traditions can add up quickly, and students from families with fewer resources may feel unable to say that they cannot afford something. Many chapters and national organizations offer payment plans or scholarships, but students have to know to ask, and asking requires a willingness to disclose financial strain to people whose approval they want. Chapters can remove that barrier by publishing full costs in advance and making hardship arrangements confidential and routine. The second is the big-little relationship, in which each new member is paired with an older member who serves as mentor and friend. At its best, this is one of the most valuable features of chapter life: a built-in guide who remembers what it was like to be new and who can answer the questions a new member is embarrassed to ask. At its worst, it becomes a channel for pressure, with the older member passing along expectations that the chapter would never put in writing. A big who asks how classes are going, notices when the little seems tired or withdrawn, and says "you don't have to come tonight" is doing some of the most effective wellness work in the chapter. Table 1 sets out some of the signals that distinguish a healthy new-member program from one that warrants concern. None of the warning signs proves that something is wrong, but each is worth asking about. Table 1. Signals in the new-member period: healthy programs and warning signs. Area Healthy sign Warning sign Schedule Written calendar shared in advance, with end times Last-minute summons; events with no end time Sleep Nothing scheduled late on school nights Late-night or early-morning "mandatory" activities Secrecy Families and advisors can see what the program involves Told not to tell parents, advisors, or other members Status New members treated as members with less experience Separate rules, errands, or demeaning titles for new members Alcohol No alcohol connected to new-member activities Drinking expected at "bonding" events Leaving Clear that a new member may depart without penalty Implied threats, or guilt about "quitting" The first three rows matter especially because they are easy to check. A chapter that runs a healthy program has nothing to hide, and a written schedule with end times is a small but telling sign that someone has thought about members' lives outside the chapter. Secrecy directed at advisors or families is among the most reliable indicators that something is being done that would not survive scrutiny. The new-member period does more than admit people. It teaches them what the chapter expects, and those lessons outlast the period itself. A new member who learns that sleep is optional, that saying no is disloyal, and that problems stay inside the chapter will carry those lessons into active membership and, eventually, into leadership, where they will teach them to the next class. A new member who learns that the chapter plans ahead, respects academic priorities, and wants to know when people are struggling will carry those lessons forward instead. This makes the new-member educator, and the chapter officers who oversee that role, some of the most influential people in the chapter's health. They are not just delivering content. They are setting the norms that the next generation will assume are simply the way things are. For new members themselves, the practical advice is modest but important. Keep a written record of your weekly commitments and actual hours, which makes it easier to see when demands are becoming unreasonable. Protect the academic habits you had before you joined, especially regular study time and sleep. Stay in contact with friends outside the chapter. Learn the name of your campus fraternity and sorority life advisor and your chapter's alumni advisor, so you know whom to contact if something feels wrong. And remember that joining a chapter does not mean giving up the right to decide what you will and will not do. That right is the subject of Chapter 6, but it begins here, in the weeks when it is hardest to exercise. Many students never experience anything worse than a demanding schedule and a lot of new friends. But the new-member period is also where the most dangerous practice in Greek life has historically lived, and it is to that practice that the next chapter turns. Chapter 3: Hazing, Named Plainly On the night of February 2, 2017, Timothy Piazza, a nineteen-year-old sophomore at Penn State, accepted a bid from the Beta Theta Pi chapter. As part of a drinking ritual that members called "the gauntlet," he consumed at least eighteen drinks in about eighty-two minutes. He then fell headfirst down a flight of basement stairs. Over the following hours, as security cameras in the house recorded, he fell repeatedly while members tried to manage the situation themselves. No one called for help until the next morning, roughly twelve hours after the first fall. He died on February 4 of a fractured skull and a lacerated spleen. His father later said that if the members had called for help, the family would not be having the conversation they were having. Timothy Piazza's name is now attached to Pennsylvania's anti-hazing law, passed in 2018, and his parents, along with the families of other students who died in hazing incidents, were among the advocates whose work led Congress to pass the Stop Campus Hazing Act in December 2024. His death is also a case study in almost every theme of this book: an initiation built on alcohol, a group whose loyalty to itself outweighed its care for one person, and a delay in calling for help that turned a medical emergency into a death. Most hazing does not end in death. That is part of why it persists. This chapter describes what hazing is, how common it is, why intelligent people participate in it, what the law now requires, and what students can do when they see it. What counts as hazing, and what the law requires For years, one of the obstacles to preventing hazing was disagreement about what it was. State laws varied, campus policies varied, and students frequently described their own experiences in ways that excluded them from the category. The Stop Campus Hazing Act, signed into law on December 23, 2024, established the first federal definition for the purpose of campus reporting. Under that definition, hazing is any intentional, knowing, or reckless act committed by a person, alone or with others, against another person, regardless of that person's willingness to participate, that is done for the purpose of recruiting, joining, initiating, admitting, affiliating with, or maintaining membership in a student organization, and that causes or creates a risk, above the reasonable risk encountered in the course of participation in the institution or organization, of physical or psychological injury. The law lists examples, including whipping, beating, or striking; causing or requiring sleep deprivation, exposure to the elements, confinement, or extreme physical exertion; compelling someone to consume food, liquid, alcohol, drugs, or other substances; compelling sexual acts; activities that cause reasonable fear of bodily harm through threats; and activities that violate criminal law. Three features of this definition are worth highlighting. First, consent does not make it acceptable. The phrase "regardless of the willingness of such other person" addresses the most common defense of hazing, which is that the new members chose to take part. The law recognizes what research on group pressure has long shown: a person who wants badly to belong, surrounded by people who control whether they will be admitted, is not in a position to give meaningful consent. Second, psychological injury counts. Hazing is not only paddling and forced drinking. Humiliation, degradation, intimidation, and isolation can cause real harm, and the definition includes them. Third, sleep deprivation is named explicitly. This matters for the themes of this book. Keeping new members up all night, calling them out at 3 a.m., or requiring early-morning activities after late-night ones is not an eccentric tradition; it is one of the forms of hazing that federal law now specifically identifies. Researchers at StopHazing, the research and training organization founded by Elizabeth Allan, describe hazing as falling along a spectrum. At one end are intimidation behaviors that are often overlooked, such as deception, silence periods, and demeaning names. In the middle are harassment behaviors that cause emotional or physical discomfort, such as being yelled at, running errands for members, or wearing embarrassing clothing. At the far end are violent behaviors that can cause lasting injury or death, such as forced drinking, beating, and extreme deprivation. The spectrum is useful because it shows that the behaviors at the extreme end are rarely where a group starts. Groups escalate. A chapter that treats demeaning names and servitude as harmless fun has already accepted the premise, that new members may be treated as less than members, on which the more dangerous practices rest. Many real situations fall into gray areas, and it helps to work through a few. Suppose new members are required to attend a 6 a.m. workout three days a week while active members are not. The activity may seem harmless, even healthy, but it applies only to new members, is tied to their admission, and cuts into sleep; under the federal definition it risks becoming hazing, particularly if it follows late-night obligations. Suppose instead that the whole chapter, new and active members alike, signs up together for a charity 5K. That is a shared challenge with a clear purpose and no status distinction. Or consider a scavenger hunt. If it involves visiting campus landmarks and learning chapter history, it is an ordinary team activity. If it requires stealing items, performing embarrassing acts in public, or drinking at each stop, it is hazing. A few questions help sort such cases. Is this activity required only of new members, or would active members do it too? Would the chapter be comfortable if its advisor, national headquarters, or a new member's parents watched it happen? Does it involve alcohol, lost sleep, physical discomfort, secrecy, or humiliation? Would a new member who declined face consequences, formal or informal? Does it serve a purpose connected to the organization's stated values, or does it simply mark new members as lower in status? If the honest answer to any of these is troubling, the activity deserves to be changed. Beyond definitions, the legal landscape has shifted. For most of the history of anti-hazing efforts in the United States, the law was a patchwork of state statutes, most of them passed after a student died. Many states now treat hazing that causes serious injury or death as a felony. Pennsylvania's 2018 law, for example, created tiers of offenses and made hazing that results in bodily injury a felony. The Stop Campus Hazing Act adds a national layer, focused on transparency and prevention rather than criminal penalties. It amends the Clery Act, the federal law requiring colleges that participate in federal student aid programs to disclose crime statistics, which the act also renamed the Jeanne Clery Campus Safety Act. Its main requirements are as follows. Colleges must include statistics on hazing incidents reported to campus security authorities or local police in their annual security reports, with data collection beginning January 1, 2025, and the first annual security reports containing hazing statistics due by October 1, 2026. Colleges must have policies on hazing that describe how to report it, how reports are investigated, and what prevention programs are in place; these were due by June 23, 2025. Prevention programs must be research-informed and campus-wide, and must include primary prevention strategies. Most visibly for students and families, each college must publish a campus hazing transparency report on its website listing student organizations found responsible for hazing violations, with the organization's name, a description of the violation, and relevant dates. The first report was due by December 23, 2025, and the report must be updated at least twice a year. The transparency report is a significant change for prospective members and their families. For the first time, there is a nationally required public record of which organizations on a campus have been found responsible for hazing. It is worth reading before accepting a bid. What the research shows The most comprehensive national data on college hazing come from the study Allan and Mary Madden published in 2008 under the title Hazing in View: College Students at Risk. They surveyed 11,482 undergraduates at 53 colleges and universities and conducted more than 300 interviews with students and staff at 18 of those campuses. Their findings, some of which are summarized in Table 2, remain the most widely cited evidence on the subject. Table 2. Share of students in each group type who reported experiencing hazing behaviors. Group type Experienced hazing behaviors Note Varsity athletics 74% Highest of all groups studied Social fraternities and sororities 73% Drinking games reported by 47% Club sports 64% Often less supervised than varsity All students in clubs, teams, or organizations 55% Across all group types Source: Allan and Madden, Hazing in View (2008). Several other findings from the study matter as much as the prevalence figures. Nine out of ten students who had experienced hazing behaviors in college did not consider themselves to have been hazed. In 95 percent of the cases in which students did identify their experience as hazing, they did not report it to campus officials. In more than half of hazing incidents, a member of the group posted pictures on a public website. In a quarter of cases, alumni were present. A quarter of the time, a coach or advisor was aware. Nearly half of students had experienced hazing before college, in high school teams and groups. And students were more likely to describe positive outcomes of the experience, such as feeling more like part of the group, than negative ones. Taken together, these findings describe a practice that is widespread, frequently public, often known to adults, seldom reported, and usually not recognized by the people it happens to. That combination is what makes hazing so durable. It is not hidden so much as normalized. Later research by Allan, David Kerschner, and colleagues has explored the gap between what students privately believe and what they think their peers believe. Consistent with the research on misperceived norms described in Chapter 1, students often assume others are more accepting of hazing than they are. The researchers have argued that effective prevention must work at many levels at once, from individual knowledge and skills, to group norms, to campus policies and leadership. Their Hazing Prevention Framework, published in 2018, lays out the elements a campus needs: commitment from leadership, capacity to carry out prevention work, cultural competence, assessment, planning, evaluation, sustainability, and the implementation of promising strategies. Why people haze and why people accept it It is tempting to assume that hazing is done by cruel people to naive ones. The research suggests something more uncomfortable: hazing is usually done by ordinary students, many of whom were hazed themselves and some of whom disliked it at the time, to other ordinary students who accept it because they want what the group offers. Several mechanisms help explain this. The first is the psychology of effort. In a classic 1959 experiment, Elliot Aronson and Judson Mills found that people who went through a more severe initiation to join a discussion group later rated the group more favorably than those who went through a mild one. When people suffer for something, they tend to value it more, partly to justify the suffering to themselves. This helps explain why hazed members often defend the practice afterward and why they may later inflict it on others: to question it would be to question the value of what they endured. The second is the belief that hazing builds unity. Members often say that shared hardship bonds a new-member class together. There is something to the idea that shared challenges can build connection, but the evidence that hazing specifically produces cohesion is weak. Research by Aldo Cimino and others has suggested that hazing is often driven more by the status motives of existing members, their desire to establish dominance over newcomers and to ensure that those newcomers are committed, than by any demonstrated effect on group solidarity. Groups build strong bonds in many ways that involve no harm at all. The third is tradition and the cycle it creates. Members who were hazed often feel that the next class should not get off easier than they did. Stopping the practice can feel like a betrayal of what the older members went through. This "cycle of hazing" means that a chapter can continue a practice for decades after most members have privately stopped believing in it. The fourth is diffusion of responsibility. In a group, it is easy for each person to assume that someone else is in charge or that someone else will stop things if they go too far. The Penn State case illustrated this with terrible clarity. Many people were present over many hours. Some expressed concern. No one acted decisively until too late. For the person being hazed, the calculus is also understandable. They want to belong. They have usually already invested time, money, and social identity in the group. They see others enduring the same thing and assume it must be normal. They fear the social cost of leaving or objecting. And, crucially, they often do not have the word for what is happening to them. This is where the finding that nine in ten hazed students do not label their experience as hazing becomes so important. Naming it is the first step toward resisting it. What to do when you see it Knowing what hazing is does not make it easy to respond. A new member who is being hazed may fear losing membership. An active member who objects may fear being ostracized by friends. Chapter 7 addresses bystander intervention in detail, but a few principles belong here. If anyone is in physical danger, the only priority is safety. Call 911. Chapter 4 explains why the fear of getting the chapter in trouble should never delay that call, and how medical amnesty and Good Samaritan policies protect people who seek help. If the situation is not an emergency, there are several routes to report hazing. Campus fraternity and sorority life offices and dean of students offices handle reports, and most campuses now offer online reporting forms that can be submitted anonymously. National fraternity and sorority headquarters take reports directly, and many operate anonymous reporting lines; this is often faster than students expect. Chapter alumni advisors can be approached privately. The Stop Campus Hazing Act requires colleges to publish how reports can be made, so the information should be easy to find. A new member who wants to leave a chapter because of hazing can do so. Membership is voluntary, and departing is always an option, however strongly the group may suggest otherwise. Leaving can be painful, but it is not a failure; it is a judgment that the price was too high. For members who want to change their own chapter's practices, the most effective path is usually to find allies. Because private disapproval of hazing is often more widespread than it appears, a member who raises concerns with two or three trusted friends frequently discovers that they are not alone. A small group that proposes replacing a hazing tradition with a meaningful alternative, such as a challenging service project, a retreat with a real purpose, or a ritual that honors history without degrading anyone, is far more likely to succeed than a lone objector. National organizations and campus staff can often provide support, including training and facilitation, to chapters that want to change. The goal is not a chapter without challenge. Joining something meaningful can and should ask something of people. The goal is a chapter in which the challenge builds members up rather than breaks them down, and in which no one ever has to choose between belonging and safety. Nowhere is that choice more stark than in the moment the next chapter describes: a member in medical danger, and a room full of people deciding whether to call. Chapter 4: Alcohol, Emergencies, and the Call That Matters In most chapters, on most weekends, nothing terrible happens. People drink, some more than they should, and they wake up the next day tired, perhaps embarrassed, perhaps hungover, and otherwise fine. That ordinariness is part of the danger. Because most nights end safely, members come to believe they know how to handle a friend who has had too much: put them to bed, turn them on their side, check on them later. Most of the time, that works. Occasionally it does not, and the difference between those outcomes often comes down to a few minutes and a single decision. This chapter is not a lecture about abstinence. Many college students drink, including many fraternity and sorority members, and some do not, and both groups deserve accurate information. The aim is to describe the risks clearly, explain what alcohol poisoning looks like and what to do about it, and make the case that no chapter rule, no fear of consequences, and no loyalty to the group should ever delay a call for help. The scale of the problem According to the 2023 National Survey on Drug Use and Health, as reported by the National Institute on Alcohol Abuse and Alcoholism, 49.6 percent of full-time college students aged 18 to 25 drank alcohol in the past month, 29.3 percent engaged in binge drinking (defined as five or more drinks on one occasion for men, four or more for women), and 6.8 percent engaged in heavy alcohol use (binge drinking on five or more days in the past month). About 14 percent of full-time college students met the criteria for alcohol use disorder in the past year. The consequences are substantial. NIAAA estimates that 1,519 college students between 18 and 24 die each year from alcohol-related unintentional injuries, including motor vehicle crashes, and that 696,000 students in that age group are assaulted by another student who has been drinking. A majority of sexual assaults in college involve alcohol or other substances. About one in four college students report academic consequences from drinking, such as missing class or falling behind. Chapter 1 described the evidence that fraternity and sorority members, particularly those living in chapter houses, drink more heavily on average than their peers, and that the pattern can persist well beyond college. This is not true of every chapter or every member, and some chapters are entirely substance-free. But the average matters because it describes the environment in which many members make decisions. National organizations have responded with policy. In August 2018, members of the North American Interfraternity Conference voted to prohibit alcohol products above 15 percent alcohol by volume, which in practice means hard liquor, in any chapter facility or at any chapter event unless served by a licensed third-party vendor. The standard took effect for chapters in the United States on September 1, 2019. Many individual fraternities and sororities have gone further, with substance-free housing policies, restrictions on events with alcohol, and requirements for sober monitors. Policies alone do not change behavior, but they signal what the organization considers acceptable and give members who want a safer culture something to point to. How alcohol poisoning happens Understanding the mechanics of alcohol poisoning helps explain why the usual responses are dangerous. A standard drink in the United States contains about 14 grams of pure alcohol. That is roughly 12 ounces of regular beer at about 5 percent alcohol, 5 ounces of wine at about 12 percent, or 1.5 ounces of distilled spirits at about 40 percent. Mixed drinks, punches, and drinks poured at parties frequently contain far more than one standard drink, and people routinely underestimate how much they have consumed. The body can process only a limited amount of alcohol at a time; NIAAA notes that the liver can metabolize roughly one standard drink per hour for many people, and nothing speeds that up. Alcohol consumed quickly continues to be absorbed from the stomach and intestine into the bloodstream after drinking has stopped. This means that a person's blood alcohol concentration can keep rising for some time after their last drink, even while they are asleep. A person who "passed out" may not be sleeping it off; they may be getting worse. At high concentrations, alcohol suppresses the parts of the brain that control basic life-support functions, including breathing, heart rate, body temperature, and the gag reflex. A person can stop breathing, or choke on vomit because the gag reflex no longer works. Rapid drinking, drinking games, and drinking on an empty stomach make these outcomes more likely, which is why so many hazing deaths have involved large amounts of alcohol consumed quickly. Combining alcohol with other depressants, such as opioids, benzodiazepines, or sleep medications, is especially dangerous. Counterfeit pills containing fentanyl have made this risk more acute in recent years, and many campuses now encourage students and chapters to keep naloxone, the medication that reverses opioid overdose, on hand. Alcohol also affects memory and judgment in ways that bear directly on safety. At high blood alcohol levels, many people experience blackouts: periods for which the brain did not form memories, even though the person may have been walking, talking, and appearing to function. Someone in a blackout can seem more capable than they are, which is one reason friends misjudge danger. Blackouts also matter for sexual consent. A person who is incapacitated by alcohol cannot give consent, and a person who is intoxicated may be unable to recognize that another person is incapacitated. Chapters that host events with alcohol carry a particular responsibility here, because the settings they create, with crowded rooms, private upstairs spaces, and a host group that controls the drinks, are settings in which sexual assault is more likely to occur. Drinks left unattended can be tampered with, and students should feel free to pour their own drinks, keep them in hand, and leave with the people they arrived with. Table 3 lists the warning signs of alcohol overdose identified by NIAAA, alongside common responses that do not help. The most important instruction is the simplest: a person does not need to show all of these signs to be in danger. Table 3. Signs of alcohol overdose and responses that do not help. Warning sign What to do What does not work Confusion or stupor; cannot be woken Call 911 immediately Letting them "sleep it off" Vomiting while drowsy or unconscious Turn on side; stay with them Leaving them alone in a room Slow breathing (fewer than 8 per minute) Call 911; monitor breathing Waiting to see if it improves Irregular breathing (10+ seconds between breaths) Call 911; tell responders what was taken Cold showers Seizures Call 911; protect from injury Coffee or "walking it off" Clammy, pale, or bluish skin; low body temperature Call 911; keep warm Giving more food or drink to "absorb" it Source: National Institute on Alcohol Abuse and Alcoholism, "Understanding the Dangers of Alcohol Overdose." While waiting for help, stay with the person. If they are conscious and vomiting, keep them sitting up and leaning forward. If they are unconscious, turn them on their side so that vomit can drain rather than block the airway. Tell emergency responders what the person drank, how much if you know, and anything else they may have taken. Do not try to make them vomit or give them anything by mouth. Why people do not call, and how amnesty helps If the right response is so clear, why do people hesitate? Research at Cornell University in the early 2000s found a striking gap: 19 percent of students surveyed had considered calling for help for an intoxicated peer, but only 4 percent had actually done so. The reasons students gave included uncertainty about whether the situation was serious enough and fear of the consequences. Both reasons are understandable, and both are dangerous. Uncertainty is built into the situation. Friends are rarely trained to judge whether someone's breathing is slow enough to be a problem, and they are often intoxicated themselves. The only safe rule is to resolve uncertainty in favor of calling. An ambulance that turns out not to have been needed is a minor inconvenience; a delay that turns out to have mattered can be permanent. Fear of consequences is especially powerful in chapters. A member who calls for help at a chapter event may worry not only about getting themselves or their friend in trouble but about bringing sanctions on the whole organization: suspension, loss of housing, or the end of a chapter with decades of history. Members may feel enormous pressure, sometimes explicit, to handle things internally. The Penn State case is the most notorious example, but it is far from the only one. In 2004, Lynn Gordon "Gordie" Bailey Jr., an eighteen-year-old first-year student at the University of Colorado, died of alcohol poisoning after a fraternity event. He had been left to "sleep it off." His family later established the Gordie Center, now based at the University of Virginia, which works to prevent hazing and alcohol deaths and teaches students to recognize alcohol poisoning and act. Medical amnesty policies exist to remove the fear of consequences from the decision to call. Their logic is straightforward: if students hesitate to call for help because they fear punishment, then reducing the punishment for calling will increase the number of calls, and more calls will save lives. Cornell University implemented one of the first well-studied medical amnesty protocols in 2002. Under the policy, students who called for help for themselves or a friend in an alcohol emergency would not face formal judicial action for alcohol violations, though they would be expected to meet with health staff afterward. A 2006 evaluation by Deborah Lewis and Timothy Marchell, published in the International Journal of Drug Policy, found that calls for emergency help in alcohol-related situations increased after the policy was introduced, and that the share of students who received follow-up education after such incidents rose substantially, from 22 percent to 52 percent over two years. A 2018 study of a policy change at another university found that medical amnesty did not increase drinking or alcohol-related problems, answering the concern that such policies might encourage risky behavior. Medical amnesty now takes two forms. Campus medical amnesty policies protect students from institutional discipline for alcohol or drug violations when they seek help. State medical amnesty or Good Samaritan laws protect people from certain criminal charges, such as underage possession, when they call for help in an alcohol or drug emergency. According to the Gordie Center, as of 2022, 40 states and the District of Columbia had medical amnesty laws covering alcohol; many states also have Good Samaritan laws covering drug overdoses. There is no federal medical amnesty law. The details of these protections vary, and every member should look up their own campus policy and state law. Some important variations include whether the policy covers only the person who needed help or also the caller and others present; whether it extends to the chapter as an organization; whether it applies to drugs as well as alcohol; and whether it covers other violations, such as hazing or assault, that may have occurred. Many campus policies require the student who received help to complete an educational program or follow-up meeting. Some national fraternities and sororities have adopted their own policies stating that a chapter that calls for help will be treated more leniently than one that does not, and that a chapter that fails to call will face the most serious consequences. That is exactly the right set of incentives. Even where protections are limited or unclear, the answer is the same. Call. No disciplinary consequence is worse than a death, and chapters that have been sanctioned for an incident in which they sought help can recover. Chapters in which a member died because no one called often do not, and neither do the people who were in the room. Building a chapter that calls Individual knowledge is necessary but not sufficient. What makes calling routine is a chapter culture in which calling is expected and praised. Several practices help. Chapters can make "call first" an explicit, repeated standard, stated at the start of each semester and at every event where alcohol may be present, and reinforced by officers who make clear that members who call will be supported, not blamed. Chapters can designate sober monitors at events who have been trained to recognize the signs in Table 3 and who have the authority to call for help without seeking anyone's permission. They can put emergency numbers, the campus medical amnesty policy, and the local non-emergency and campus police numbers somewhere visible in the house. They can make water and food available at every event, avoid drinking games, and ensure that people who do not drink are fully included in social life, which also reduces pressure on those who are drinking less. And after an incident in which someone called for help, chapter leaders can thank the caller publicly, which sends a clearer message than any policy document. Consider how this plays out in a single night. It is 1 a.m. at a chapter party. A junior who has been serving as a sober monitor notices a first-year guest slumped on a couch in a back room. Her friends say she is fine, that she just had too many shots and needs to sleep. The monitor kneels, says the guest's name loudly, and gets only a mumble. Her breathing seems slow, though the monitor is not sure. Two older members wander in and say that if an ambulance comes to the house, the chapter will be on probation for the rest of the year. In that moment, everything depends on whether the monitor believes three things: that she does not need anyone's permission to call, that uncertainty is itself a reason to call, and that her chapter and her campus will stand behind her if she does. If the chapter has spent the semester saying "call first," thanked previous callers, and made sure every monitor knows the medical amnesty policy, she will call. If it has not, she may hesitate, and hesitation is the most dangerous thing in the room. Whatever happens afterward with the chapter's standing, a phone call at 1:04 a.m. is a far better outcome than a phone call at 9 the next morning. For individual members, the guidance is modest and practical. Know what a standard drink is and count them. Eat before and while drinking. Alternate alcoholic and non-alcoholic drinks. Decide in advance how much you intend to drink, and have a friend who will hold you to it. Never leave a drunk friend alone, and never assume that someone who is unconscious is simply asleep. And remember that choosing not to drink, for a night or for good, is a legitimate choice that a healthy chapter will respect. The next chapter turns to the resource that alcohol, late nights, and a crowded calendar all draw down at once: sleep. Chapter 5: Sleep and the Chapter Calendar Ask chapter members what they sacrifice to keep up with everything, and the answer is almost always the same. Not grades, at least not at first. Not friendships inside the chapter, because the chapter is where their friendships are. Not the job, because rent is due. What goes is sleep. It goes quietly, a half-hour here and an hour there, and because the loss is gradual, most people do not notice how much they have given up until they are running on five and a half hours a night and wondering why everything feels harder. Sleep is the most undervalued health resource on a college campus and, for fraternity and sorority members, often the most contested. It is contested not because anyone sets out to deprive members of rest, but because a chapter calendar is built by many hands, each adding something reasonable, and no one is responsible for adding up the total. This chapter makes the case that a chapter's calendar is a health document, whether anyone treats it as one or not, and describes what individuals and chapters can do to protect sleep. What sleep does and what happens without it In 2015 the American Academy of Sleep Medicine and the Sleep Research Society published a joint consensus statement, developed by a panel of experts after reviewing the scientific evidence, recommending that adults sleep seven or more hours per night on a regular basis to promote optimal health. The statement noted that sleeping more than nine hours a night may be appropriate for young adults, for people recovering from sleep debt, and for people with illnesses. Traditional-age college students fall squarely in the young-adult group, and many of them need more than seven hours to feel and function at their best. Most do not get it. In a 2014 review in the journal Nature and Science of Sleep, Shelley Hershner and Ronald Chervin of the University of Michigan reported that about 50 percent of college students experience daytime sleepiness and about 70 percent get insufficient sleep. They documented the consequences: lower grade point averages, greater risk of academic failure, compromised learning, impaired mood, and increased risk of motor vehicle crashes. The connection between sleep and mental health is especially important. Research presented at the 2019 annual meeting of the Associated Professional Sleep Societies, led by researchers at the University of Arizona and drawing on data from more than 110,000 students in the American College Health Association's National College Health Assessment, found that each additional night of insufficient sleep per week was associated with an average increase of more than 20 percent in the likelihood of experiencing mental health symptoms. The associations covered depressed mood, anxiety, hopelessness, anger, functional problems, and thoughts of self-harm and suicide. These are correlations, and poor mental health also disrupts sleep, so the relationship runs in both directions. But the broader experimental literature on sleep deprivation makes clear that insufficient sleep worsens emotional regulation directly. People who are short on sleep react more strongly to negative events and have a harder time calming down afterward. Sleep also matters for learning. During sleep, the brain consolidates memories formed during the day, transferring them into more durable storage. A student who studies until 3 a.m. and sleeps four hours before an exam has given their brain little opportunity to do that work. And sleep matters for safety. Drowsy driving impairs reaction time in ways comparable to alcohol, which is particularly relevant for students driving home from late chapter events, road trips to other campuses, or early-morning philanthropy shifts. Two features of young adult biology make the problem worse. First, the internal body clock tends to shift later during adolescence and early adulthood, so many college students naturally feel alert late at night and struggle to fall asleep early. Second, alcohol, often present on the same nights that run late, disrupts sleep. It may help people fall asleep faster, but it fragments sleep in the second half of the night and suppresses the rapid eye movement stage, so that a night of drinking often produces a night of poor-quality rest even when the hours look adequate. How the calendar eats the night Consider an illustrative week for a sorority member in her second year. Monday has a two-hour chapter meeting that starts at 8 p.m. and often runs until 10:30. Tuesday is a committee meeting and a study-hours requirement at the house. Wednesday is a sisterhood event. Thursday is the night most of the social life happens, and although it is not mandatory, her friends will all be there. Friday has a mixer with a fraternity. Saturday is a philanthropy event beginning at 8 a.m. Sunday is when she catches up on the week's coursework, and her big has asked her to brunch. Scattered through all of this are group chats that start buzzing after 10 p.m., when everyone is finally free, and continue until well past midnight. None of these commitments is unreasonable on its own. Each was scheduled by someone trying to do something good: to run the chapter, support academics, build friendships, raise money for a cause. But together they leave very few nights on which she can be in bed by midnight, and several on which getting seven hours before an early class is impossible. Many chapters also use points or attendance systems to encourage participation, in which members earn credit for attending events and face consequences, such as fines or restrictions on social privileges, if they fall short. These systems exist for understandable reasons. A chapter with low attendance cannot function, and officers who plan events want people to come. But points systems can quietly convert optional events into obligatory ones, and they tend to reward the members with the most free time, not necessarily the most committed ones. Members who work paid jobs, have demanding majors, care for family members, or manage health conditions are the ones most likely to be squeezed. Officers are especially vulnerable. Chapter presidents, recruitment chairs, and treasurers often describe their roles as consuming the equivalent of a part-time job, with meetings, emails, and crises arriving at any hour. They are also the people least likely to complain, because complaining feels like admitting they cannot handle the role they asked for. Table 4 sets out some common calendar practices, how they affect sleep, and alternatives that serve the same purpose with less cost. Table 4. Chapter calendar practices, their effect on sleep, and alternatives. Common practice Effect on sleep Alternative Weeknight meetings that start at 8 or 9 p.m. with no end time Pushes bedtime past midnight Earlier start; firm 90-minute limit; written agenda Early-morning events after late-night ones Cuts sleep from both ends At least ten hours between scheduled activities Group chat traffic late at night Delays sleep; wakes people Quiet hours for chapter channels; scheduled messages Points systems that reward sheer attendance Makes optional events feel mandatory Fewer, better events; flexible ways to meet expectations Heavy programming during exam periods Competes with study and sleep Protected exam weeks with no required events Officers on call at all hours Chronic sleep loss for leaders Shared duties; clear off-hours; a backup contact The alternatives are not radical. A chapter meeting that starts at 7 p.m. and ends at 8:30 accomplishes the same business as one that runs from 8:30 to 11. A group chat can have quiet hours. An exam week can be kept free of required events. What makes these changes hard is not their difficulty but the fact that no one is responsible for making them. The chapter calendar rarely has an owner who asks, "Taken together, what does this week cost our members?" Protecting your own sleep Individual members cannot redesign the chapter calendar alone, but they have more control than they often assume. The first step is to see the week clearly. A simple exercise helps: take a typical week and write down every commitment, including class, study, work, chapter obligations, and travel, alongside the time you actually went to sleep and woke up. Many students are surprised by what they find. The point is not to feel guilty but to have accurate information. It is much easier to decide what to change when you can see where the hours go. The second is to decide on a sleep floor, the minimum number of hours you will protect on most nights, and treat it as a commitment rather than a leftover. A student who decides that, five nights a week, she will be in bed by 12:30 a.m. with the phone on silent, has something concrete to protect when the invitations arrive. Chapter 6 describes how to decline gracefully, but the decision about what to protect comes first. The third is to keep a consistent wake time. Sleep researchers generally advise that waking at roughly the same time each day, including weekends, helps stabilize the body clock and makes it easier to fall asleep at night. Sleeping until noon on Saturday after a late Friday feels restorative but can make Sunday night's sleep harder and Monday morning worse. The fourth is to manage the phone. Notifications from active group chats are among the most common reasons students give for delayed or interrupted sleep. Muting chapter channels after a set hour, keeping the phone out of reach of the bed, and letting close friends know that you will reply in the morning are small changes with outsized effects. The fifth concerns caffeine and alcohol. Caffeine remains active in the body for many hours, so an afternoon energy drink can still be interfering with sleep at midnight. Mixing caffeine with alcohol is particularly risky because the stimulant can mask how intoxicated a person feels. And, as noted above, alcohol may knock you out but will not give you good sleep. Finally, if you live in a chapter house, the environment matters. Houses can be loud late at night, and sharing a room with someone on a different schedule is difficult. Earplugs, an eye mask, and an honest conversation with roommates about quiet hours are worth more than any sleep-tracking app. Houses can adopt quiet hours policies, especially during exam periods, and residents can ask house managers to enforce them. Return to the illustrative second-year member whose week opened this section, and consider what a realistic repair might look like. She cannot skip chapter meeting, and she does not want to. But she can ask the chapter president whether meetings could start at 7:30 rather than 8, and in the meantime she can go straight to bed afterward rather than joining the hour of conversation in the kitchen that usually follows. She can do her Tuesday study hours in the morning at the library, which the chapter's academic chair agrees counts just as well, freeing the evening. She can go to Wednesday's sisterhood event and leave when it officially ends. She can decide that Thursday is a night she will go out, and plan for it by scheduling nothing before 11 a.m. on Friday; or she can decide that this particular Thursday, with a problem set due Friday morning, she will stay in, and tell her friends she will see them at the mixer. She can go to the Saturday philanthropy event and decline the Friday mixer, or go to the mixer and leave by midnight. She can mute the chapter group chat at 11 p.m. and tell her big that she will answer in the morning. None of these choices requires her to love the chapter less or participate less in what matters to her. What they require is a clear view of the week, a decision about what she will protect, and the confidence to act on it. By the end of such a week, she might have gained six or seven hours of sleep without missing anything she truly cared about. That is roughly an extra night, recovered from the gaps between obligations. A word about naps, which many students use to compensate for short nights. A short nap of twenty to thirty minutes in the early afternoon can improve alertness without much effect on nighttime sleep. Long naps, or naps late in the day, can make it harder to fall asleep at night and start a cycle in which the student is awake at 2 a.m. and exhausted at 10 a.m. Naps are a patch, not a substitute for adequate nighttime sleep. If sleep problems persist despite these steps, including trouble falling or staying asleep most nights for several weeks, loud snoring or gasping, or excessive daytime sleepiness even after enough hours in bed, it is worth talking to campus health services. Insomnia is highly treatable, and the first-line treatment recommended by major medical organizations, cognitive behavioral therapy for insomnia, is available on many campuses. Making sleep a chapter value The changes that matter most happen at the chapter level, and they begin when someone decides to treat sleep as a legitimate concern rather than a personal failing. A chapter can review its calendar at the start of each term with sleep in mind, asking of each recurring event whether it needs to happen that often, that late, or at all. It can adopt a few standing rules: no required events after 10 p.m. on school nights, a firm end time for chapter meetings, a minimum gap between late-night and early-morning obligations, and quiet hours on chapter communication channels. It can make exam periods genuinely protected. It can distribute officer responsibilities more evenly so that no one person is on call all the time. And it can audit its points or attendance system, asking whether it is rewarding commitment or simply rewarding free time. The new-member period deserves particular attention. As Chapter 3 noted, the Stop Campus Hazing Act specifically lists sleep deprivation among the behaviors that can constitute hazing. A new-member program that routinely keeps new members up late or wakes them early is not just unkind; it may be unlawful under federal reporting rules and campus policy. A chapter that wants to demonstrate its commitment to new members' wellbeing can start with a simple promise: nothing scheduled for new members after 10 p.m. or before 8 a.m. Leaders set the tone here as elsewhere. A chapter president who says at the first meeting of the semester, "I go to bed by midnight on weeknights, and I don't answer chapter messages after 11," gives every member permission to do the same. A president who is visibly exhausted and answers messages at 3 a.m. teaches the opposite lesson, however often they tell members to take care of themselves. Protecting sleep is sometimes dismissed as soft, a lesser concern next to hazing and alcohol poisoning. But the three are connected. Sleep-deprived students are more emotionally volatile, more susceptible to peer pressure, worse at judging risk, and more likely to rely on alcohol and caffeine to get through the week. A well-rested chapter is a safer chapter. It is also, not incidentally, a chapter whose members have the energy to enjoy one another. That is what the calendar is supposed to be for. Chapter 6: Boundaries and the Art of Saying No Every chapter of this book so far has come back to a single capacity: the ability to decline. The new member who can say no to an activity that feels wrong, the guest who can say no to another round of shots, the member who can say no to a 10 p.m. event before an 8 a.m. exam, the officer who can say no to one more responsibility. In each case, the skill is simple to describe and difficult to practice, because the person saying no wants very much to stay in good standing with the people they are saying it to. This chapter is about that skill. It treats boundaries not as walls that keep people out but as the conditions under which a person can stay in a community for the long run without losing themselves. Members who cannot set boundaries tend to burn out, pull away abruptly, or quietly resent the chapter. Members who can set them tend to stay, and to stay well. What a boundary is A boundary is a decision about what you will and will not do, give, or accept. It is not a rule you impose on others; you cannot control what other people ask of you. It is a rule about your own response. "My chapter should not schedule meetings so late" is a request, and a reasonable one. "I leave chapter meetings at 10 on weeknights" is a boundary. Boundaries in chapter life tend to fall into a few categories. Time boundaries concern how many hours you give, when, and to what. Physical and substance boundaries concern alcohol, drugs, physical activity, and bodily safety. Financial boundaries concern what you spend on dues, events, clothes, trips, and gifts. Emotional boundaries concern how much of others' distress you take on and how much of your own you share. Digital boundaries concern when you are reachable and what you post or allow to be posted about you. And academic boundaries concern the study time you protect, whatever else is happening. The most important thing about boundaries is that they work best when decided in advance. In the moment, standing in a crowded room with friends urging you to stay, it is very hard to reason clearly about what you want. Social pressure, fatigue, alcohol, and the desire to be liked all push toward yes. A boundary decided on a calm Sunday afternoon, when you can think about your week and your priorities, is much easier to hold at midnight on Thursday. Research on goal pursuit supports this. The psychologist Peter Gollwitzer and his colleagues have studied what they call implementation intentions: specific if-then plans that link a situation to a response, such as "If someone offers me a drink after my second, then I'll ask for water." A 2006 meta-analysis by Gollwitzer and Paschal Sheeran, drawing together results from many studies, found that forming such plans had a meaningful positive effect on whether people actually achieved their goals. The mechanism is intuitive. When the situation arises, the response has already been chosen, so there is no need to deliberate under pressure. Why saying no feels so hard Most students overestimate the social cost of declining. Several lines of research help explain why. The first is what the psychologists Thomas Gilovich, Victoria Medvec, and Kenneth Savitsky called the spotlight effect. In a set of studies published in 2000, they found that people consistently overestimate how much others notice their appearance and behavior. A student who leaves a party early may imagine that everyone registers the departure and judges it. In practice, most people are absorbed in their own experience and barely notice. The second concerns the person making the request. Research by Vanessa Bohns and her colleagues has found that people who ask others for things tend to underestimate how hard it is for the other person to say no. In other words, the older member who casually asks a new member to drive her across town at midnight probably does not realize how much pressure the request carries. That is not an excuse for the request, but it does mean that a polite refusal is less likely to surprise or offend the asker than the refuser fears. The asker often never imagined that the request was difficult to decline. The third is the misperceived norms problem described in Chapter 1. Members often believe that their peers are more enthusiastic about late nights, heavy drinking, and constant availability than those peers actually are. The person who says no first frequently gives others permission to do the same. None of this means saying no carries no cost. Sometimes it does. A member who consistently declines social events may be seen as less engaged. A new member who objects to an activity may face real hostility. But the cost is usually smaller than feared, and it has to be weighed against the cost of saying yes: lost sleep, lower grades, compromised safety, and the slow erosion of a person's sense that their own judgment matters. How to say it Declining well is a skill, and like any skill it improves with practice. A few principles help. Keep it short. The instinct when saying no is to explain at length, in the hope that a good enough reason will make the refusal acceptable. But long explanations invite negotiation. "I can't tonight, I've got an exam tomorrow" is enough. So is "I'm heading home, see you Saturday." You do not owe anyone a detailed justification for protecting your sleep, health, or safety. Be warm and clear at once. Warmth and firmness are not opposites. "I love that you guys are going, and I'm going to sit this one out" communicates both affection and a decision. What undermines a boundary is vagueness: "I'll probably come by later" when you do not intend to, which leaves the door open for follow-up pressure. Offer an alternative when you mean it. "I can't make brunch, but can we get coffee Tuesday?" preserves the relationship without abandoning the boundary. Only offer it if you mean it, because an alternative you do not follow through on becomes its own source of stress. Use the language of habit rather than exception. "I don't drink on weeknights" is more stable than "I don't feel like drinking tonight," because it describes who you are rather than a passing mood, and people are less likely to argue with it. The same applies to "I'm in bed by midnight during the week," "I don't do anything before my 9 a.m. lab," and "I don't post photos of other people without asking." Repeat without escalating. If someone presses, the most effective response is often to repeat the same boundary calmly, perhaps with slightly different words, without adding new justifications or becoming defensive. "Honestly, I'm good. I'm heading out." Most people stop pressing after the second or third calm repetition. Some situations call for specific approaches. Declining alcohol at a social event is easier with a drink in hand, even a non-alcoholic one, because it removes the most common prompt ("Why aren't you drinking?"). A simple, habitual answer ("I'm good for tonight," "I'm driving," "I've got an early morning") is usually sufficient. A friend who knows your plan in advance can back you up. Declining a request from an older member, a big, or an officer is harder because of the difference in status. It helps to address the request, not the relationship: "I can't do that tonight, but I'd really like to help with the event next week." If an older member makes a request that seems inappropriate, such as running personal errands, performing tasks only new members are asked to do, or anything involving alcohol, secrecy, or humiliation, the new member has every right to decline, and the request is itself something a chapter officer or advisor should know about. Declining a leadership role is sometimes the healthiest decision a member can make. Chapters often pressure capable, reliable members to take on more and more, precisely because they are capable and reliable. A member who is already stretched can say, "I'm honored you asked, and I need to focus on my classes this semester. Can I help in a smaller way?" Chapters that respect such answers are more likely to have members willing to take on roles in future semesters. Declining an activity that may be hazing is the hardest case, and it is discussed further in Chapter 7. A new member can say, "I'm not comfortable with this, and I'm not going to do it," and then leave if necessary. Afterward, contacting a trusted older member, the chapter advisor, the campus fraternity and sorority life office, or the national organization is not disloyalty. It is exactly what those people are there for. Financial boundaries are among the least discussed and most stressful. Beyond dues, chapter life can involve a steady stream of smaller costs: themed outfits for events, formals and date parties, trips, big-little gifts, charity fundraisers where members are expected to contribute, and meals out with friends who have more money. Students from families with fewer resources often absorb these costs silently, working extra shifts or taking on credit card debt rather than admitting that they cannot keep up. A clear boundary, such as a monthly amount set aside for chapter-related spending beyond dues, is easier to hold if it is decided in advance and treated as fixed. Declining can be simple: "That's outside my budget this month, but I'll be at the next one." Chapters can help by keeping optional costs genuinely optional, providing affordable or free alternatives, and making it easy to raise financial hardship privately with the treasurer or the national organization. A member should never have to choose between chapter participation and rent. Digital boundaries concern both availability and visibility. On availability, the chapter group chat can create an expectation that members respond at any hour, and the pressure to keep up can be constant. Muting channels, setting notification schedules, and telling close friends when you will and will not reply are all legitimate. On visibility, members have a right to decide what images of them are posted and where. Photos taken at parties, especially of people who are intoxicated, can follow a person for years, and the hazing research described in Chapter 3 found that images from hazing incidents frequently end up online. "Please don't post that one" should be enough, and a chapter norm of asking before posting photos of others protects everyone. Some members have boundaries rooted in identity or health that others may not know about. A member in recovery from a substance use disorder, a member whose faith prohibits alcohol, a member managing an eating disorder, a diabetic member, or a member taking medication that interacts with alcohol all have reasons to decline that are nobody else's business. They should not have to explain, and a healthy chapter will not ask. Many campuses now operate collegiate recovery programs that offer community and support for students in recovery, and some students find that these programs complement chapter membership well. Chapters can make these members' lives easier simply by planning events where not drinking is unremarkable. Consider how these principles might come together across one ordinary weekend for an illustrative first-year fraternity member. On Friday afternoon he looks at his calendar and makes three decisions: he will go to the Friday party and leave by 1 a.m. with the friend he is walking home with; he will not drink more than three drinks; and he will be at Saturday's philanthropy car wash at 10 a.m. At the party, an older member hands him a fourth drink and he says, "I'm good, thanks, I'm on the car wash tomorrow," and takes a water instead. The older member shrugs and moves on, which is what usually happens. At 12:45 his friends want to go to another party. He says, "I'm heading back, have fun," and leaves. On Saturday, his big asks whether he can help move furniture into the house on Sunday morning. He has an essay due Monday, and he says, "Sunday morning's my writing block, but I can do Sunday at four." His big says fine. None of these exchanges took more than a few seconds, and none cost him anything with the people involved. What made them possible was that he had decided in advance what mattered to him that weekend. Emotional boundaries and supporting others One of the gifts of chapter life is that members look after one another. That can also become a burden. A member who is known as a good listener may find that friends bring every crisis to her, including serious ones involving depression, self-harm, eating disorders, or trauma, sometimes late at night and often in confidence. She may feel that she cannot step back without abandoning someone who needs her. Supporting friends is good. But no student is equipped to be another student's only source of help in a serious mental health crisis, and trying to fill that role can harm both people. The supporter can become exhausted, anxious, and depressed. The friend in crisis may not get the professional help they need, because they have someone to talk to and so never reach out further. A healthy emotional boundary in this situation does not mean withdrawing support. It means supporting the friend in getting additional help: "I'm really glad you told me, and I want to keep being here for you. I also think you deserve more support than I can give. Can we look at the counseling center together?" It means knowing that if a friend talks about wanting to die or has a plan to hurt themselves, the right response is to involve others immediately, including calling or texting 988, contacting campus counseling crisis lines, or calling 911 if the danger is immediate, even if the friend asked for secrecy. Keeping a friend alive matters more than keeping a promise. It also means recognizing when your own reserves are low and saying so. "I want to help, and I'm really wiped out tonight. Can we talk tomorrow? If things get worse before then, please call 988 or the counseling center's crisis line." Chapter 8 describes how chapters can structure mental health support so that it does not fall on a few informal caregivers. When the boundary is the chapter itself For some members, the question eventually becomes whether to stay in the chapter at all. This is a legitimate question, and it deserves to be asked without shame. People join organizations at eighteen or nineteen based on limited information, and they change. The chapter may change too. A member who finds that membership consistently costs more than it gives, in health, grades, money, or integrity, is entitled to reconsider. There are often options short of leaving entirely. Some organizations allow members to take a temporary leave, reduce their involvement during a difficult semester, or change membership status for reasons such as study abroad, financial hardship, or health. The details vary widely by organization, and members should ask their chapter officers, advisors, or national headquarters what is available. For others, leaving is the right choice. Members who resign should understand that their national organization's rules govern that process, including any financial obligations, and they can ask the campus fraternity and sorority life office for help navigating it. Leaving can be painful, especially when a member's closest friends remain in the chapter. But friendships that are real usually survive the change. A member who leaves a chapter because it was harming them has not failed at belonging. They have made a judgment about their own wellbeing, which is precisely the capacity this chapter has been about. For members who stay, the goal is a relationship with the chapter that is sustainable: one in which they give generously, receive support in return, and can decline when they need to without fearing that their place is at risk. That kind of membership does not happen by accident. It depends on individuals who know their limits and on chapters that honor them. It also depends on members being willing to act when someone else's safety is at stake, which is where the next chapter turns. Chapter 7: Stepping In When It Is Your Own Chapter Most bystander intervention training begins with strangers. A person is harassed on a train, a stranger collapses on the sidewalk, someone at a bar is being steered toward the door by a man she does not seem to know. The question is whether onlookers will act, and the research on that question is substantial and, on the whole, encouraging. Chapter life poses a harder version of the problem. The people whose behavior you might need to interrupt are not strangers. They are your friends, your big, the chapter president, the older members whose approval you want, and the alumni who will be at homecoming. The people who might need your help are also people you know. And the situations are rarely as clear as a collapse on the sidewalk. They are a drinking game that is getting out of hand, a new-member activity that seems a bit much, a comment about a woman at the party that is not quite a threat, a friend who has been saying worrying things in the group chat. Stepping in means risking relationships you care about, often in a room where everyone else seems to think things are fine. This chapter explains why people do and do not intervene, what makes intervention more likely, and how to do it in ways that are effective and, as far as possible, preserve relationships. Its central claim is that intervention inside a chapter is less about individual heroism than about preparation and permission: members who have practiced what they would say, in chapters that have explicitly decided that stepping in is what members do. Why bystanders hesitate The modern study of bystander behavior began with the psychologists Bibb Latané and John Darley in the late 1960s. In a series of experiments, they found that people were less likely to help, or slower to help, when other people were present. In one well-known study, participants filling out questionnaires in a room that began to fill with smoke were much slower to report it when they were sitting with others who did not react than when they were alone. Latané and Darley proposed that intervention requires a person to pass through a series of steps, and that the presence of others can derail any of them. First, the bystander must notice that something is happening. In a loud, crowded party, or when a person is focused on their own conversation, this is not guaranteed. Second, the bystander must interpret the situation as one that requires action. Here, other people's reactions matter enormously. If everyone else seems calm, a bystander may conclude that there is no problem, even if privately worried. Each person looks to the others for cues, and if everyone is doing the same, the group collectively concludes that nothing is wrong. This is the same mechanism of pluralistic ignorance described in Chapter 1. Third, the bystander must decide that it is their responsibility to act. In a group, responsibility is diffused: everyone assumes someone else, perhaps someone older, more senior, or closer to the person, will handle it. Fourth, the bystander must know how to help. A person who does not know what to say to a friend who is being pressured, or does not recognize the signs of alcohol poisoning, may freeze even if they want to act. Fifth, the bystander must actually act, which means overcoming the fear of embarrassment, of being wrong, of social consequences, or of physical danger. Every one of these steps is harder in a chapter setting than in a stranger setting. Noticing is harder because chapter events are often loud, crowded, and involve alcohol. Interpreting is harder because what looks alarming to an outsider may be framed as tradition or fun by people you trust. Taking responsibility is harder because there is almost always someone more senior present. Knowing how to help is harder because the usual advice, "tell someone in charge," does not work when the person in charge is part of the problem. And acting is harder because the social cost of being wrong, or even of being right, falls on relationships you will live with every day. It would be easy to conclude from this that bystanders rarely act. But more recent research offers a more hopeful picture. A 2020 study published in American Psychologist, led by Richard Philpot with colleagues including Mark Levine, analyzed real-life public conflicts captured by surveillance cameras in Lancaster in the United Kingdom, Amsterdam, and Cape Town. In nine out of ten of those conflicts, at least one bystander intervened, and the likelihood that someone would intervene increased as the number of bystanders grew. Earlier work by Levine and colleagues found that shared identity mattered too: in one study, soccer fans were more likely to help an injured stranger wearing their own team's shirt than one wearing a rival team's or a plain shirt. These findings suggest that the problem is not that people are unwilling to help, but that the conditions for helping have to be right. The second finding is especially relevant to chapter life. Shared identity increases helping. Fraternity and sorority members share an intense identity. The question is whether that identity has been defined to include looking out for one another in ways that sometimes mean confronting one another. The ways to step in Most bystander training programs used on college campuses teach a small set of intervention options, often summarized as the "Ds." The Green Dot program and others commonly teach three: direct, distract, and delegate. Some add delay (checking in afterward) and document (recording what happened, when it is safe and useful). The value of having several options is that no one has to choose between a dramatic confrontation and doing nothing. There is almost always something in between. Direct intervention means addressing the situation head-on. "Hey, he's done. No more drinks for him." "That's not okay, stop." "We're not doing this to the new members." Direct intervention is often the most effective when the bystander has status in the group and the situation is not physically dangerous to the intervener. It can also be the hardest, because it puts the bystander in open disagreement with others. Distraction means interrupting a situation without confronting it. A member who notices a friend being cornered by someone at a party can walk up and say, "There you are, I need you for a second," and take the friend away. A member who sees a drinking game escalating can announce that the pizza is here or put on music that changes the room's energy. Distraction works because it changes the dynamics without requiring anyone to admit that anything was wrong, which makes it especially useful when direct confrontation would be socially costly or unsafe. Delegation means getting someone else involved, usually someone with more authority or capacity. In a chapter, that might be a sober monitor, a chapter officer, a house director, the chapter advisor, campus police, or emergency services. Delegation is not a lesser form of intervention. In a medical emergency, calling 911 is the most important intervention there is. In a hazing situation, contacting an advisor or the national organization may be more effective than confronting the members directly. Delaying means acting after the fact: checking in with the person who was targeted, asking if they are okay, offering support, and helping them decide what to do next. It matters because not every situation can be interrupted in the moment, and because people who have been harmed often feel most alone afterward. Documenting means recording what happened, when doing so is safe, will not further harm the person targeted, and is done to support them rather than to shame them. A written note of dates, times, and what was said can later help a person who decides to report. The best choice depends on the situation, the bystander's relationship to the people involved, and safety. There is no single right answer, and the choice to delegate or distract rather than confront directly is not cowardice. It is often good judgment. Chapter situations and what stepping in looks like The general principles become more useful when applied to specific situations members are likely to face. When a member or guest is dangerously intoxicated, the relevant intervention is primarily medical, and Chapter 4 describes it in detail. The bystander's job is to recognize the signs, stay with the person, and call for help without seeking permission. If others in the room resist calling, the bystander can say, "I'm calling. If it turns out to be nothing, that's fine," and do so. Nothing about chapter reputation or policy justifies waiting. When a new-member activity appears to be crossing into hazing, the bystander's options depend on their position. An active member who sees something wrong can often intervene directly, especially if they recruit one or two others first: "Hey, let's wrap this up. This isn't what we're about." An active member who feels unable to intervene in the moment can delegate afterward by speaking with the chapter president, the new-member educator, or the chapter advisor, or by reporting to the campus or national organization. A new member who is being hazed is a bystander as well as a target, and can speak up for fellow new members, which is often easier than speaking up for oneself: "I don't think he's okay. He needs to stop." The most effective intervention against hazing often happens long before any activity takes place, in the meeting where the new-member program is planned. A member who says, at that meeting, "Why are we doing this? What's it for? Is there another way to get the same result?" can stop a practice before it starts. This kind of upstream intervention is less dramatic than stepping into a hazing event, but it is also less risky and often more successful, because it gives others the chance to agree without having to back down publicly. When a sexual assault may be about to happen, for instance when someone who is clearly intoxicated is being led away from a party, or when a member is pressuring a guest who seems uncomfortable, distraction and direct intervention are both useful. A bystander can approach the person being targeted and check in: "Hey, are you okay? Your friends are looking for you." They can get the person's friends involved. They can tell the other person directly that the target is too drunk to be making decisions. They can involve a sober monitor or chapter officer. And if the situation seems dangerous, they can call campus police. Chapters that host events have a particular obligation here, because the host group controls the space. When a member is struggling with their mental health, the intervention is different in kind. It usually means noticing changes, such as withdrawal, missed events, changes in mood, talk of hopelessness, or giving away belongings, and then having a private, caring conversation. "I've noticed you haven't been around much, and you seem really down. I care about you. How are you doing, really?" If the person indicates that they are thinking about suicide, the most important thing is to ask directly, stay with them, and connect them with help. Asking about suicide does not put the idea in someone's head; that concern is a common myth. Programs such as QPR (Question, Persuade, Refer) teach exactly this sequence and are offered on many campuses. The 988 Suicide and Crisis Lifeline, reached by calling or texting 988 in the United States, can help both the person in crisis and a friend who is trying to support them. If the danger is immediate, call 911. When a member makes a racist, sexist, homophobic, or otherwise demeaning comment, the stakes may seem lower, but the pattern of intervention is similar. A direct response can be brief: "Not cool, man." "We don't talk like that here." A distraction can change the subject. A delayed response can be a private conversation later, or checking in with anyone who was hurt by the comment. These moments matter for the chapter's broader culture, because they show members what is tolerated, and a chapter in which demeaning comments pass without challenge is usually one in which more serious harms can go unchallenged too. Making intervention a chapter norm Individual members can learn to intervene, but the most powerful changes happen when the chapter as a whole decides that stepping in is what members do. That decision has to be explicit, repeated, and backed by leaders. The first step is to name it. A chapter that says, at the start of each term and at every event, "In this chapter, if you see something wrong, you step in, and you'll have our full support when you do," changes the third step of Latané and Darley's model for every member. Responsibility is no longer diffused; it is assigned to everyone. The second step is to train. Many campuses offer bystander intervention workshops for student organizations, and many national fraternities and sororities require or provide them. Programs such as Step UP!, Green Dot, and others teach the steps described above and give participants practice. Practice matters. A member who has actually said, "Hey, he's done, no more for him," in a role-play is far more likely to say it at a real party than one who has only heard that it is a good idea. The third step is to prepare specific plans. Chapters can decide in advance who will serve as sober monitors at events and what authority they have, what the process is for calling for medical help, whom members should contact if they are worried about hazing, and how members can raise concerns about a brother or sister's mental health. Plans remove the need to improvise under pressure. The fourth step is to honor intervention after the fact. When a member steps in, calls for help, or raises a concern about a tradition, the chapter's response determines whether anyone will do it again. A chapter that thanks and protects the intervener sends a clear signal. A chapter that blames them for bringing trouble, even implicitly, ensures that the next person will stay silent. Chapters and individuals also need to take the intervener's own safety seriously. Stepping in should never mean putting oneself in physical danger. If a situation involves violence, weapons, or a group that is out of control, the right intervention is to get to safety and call for help. Members who report hazing or other misconduct sometimes fear retaliation from the people they reported. Most campus conduct codes explicitly prohibit retaliation against people who report in good faith, and federal law prohibits retaliation in matters covered by Title IX, such as sexual harassment and assault. A member who experiences retaliation, whether exclusion, threats, or harassment, should document it and tell the campus office that received the original report. Retaliation is itself a violation, and in many cases it is taken as seriously as the original conduct. It also helps to remember that intervention does not have to succeed perfectly to matter. A member who says "that's not cool" and is laughed at has still told everyone in the room that at least one person disagrees, and research on norms suggests that others who were privately uncomfortable will have noticed. The first objection is often the hardest. The second person to object usually finds it much easier, because the first has already broken the illusion that everyone agrees. Finally, chapters can harness the identity that makes intervention hard. The same loyalty that makes members reluctant to confront one another can be redirected: "We look out for each other. That's what brotherhood means. That's what sisterhood means." Framed this way, stepping in is not a betrayal of the chapter's values but the most direct expression of them. The next chapter describes how leaders can build that framing into everything the chapter does, starting with the way it handles the mental health of its members. Chapter 8: Leading a Chapter Toward Mental Health A chapter president has an unusual amount of influence over the wellbeing of other people. At twenty or twenty-one years old, often with little training, she presides over an organization of dozens or hundreds of members, many of whom live together, socialize together, and look to her for cues about what matters. She controls, directly or indirectly, the calendar, the agenda, the tone of meetings, the treatment of new members, and the chapter's response to crises. She will serve for a year, perhaps less, and then hand the whole thing to someone else. This chapter is written for people in that position and for those who advise them: chapter officers, wellness and risk-management chairs, new-member educators, alumni advisors, campus fraternity and sorority life staff, and the national organizations that support them. It argues that mental health can and should be ordinary chapter business, as routine as finances or recruitment, and it describes what that looks like in practice. Why leaders matter most Student mental health has become one of the central concerns of American higher education. The Healthy Minds Study, which surveys students across the country every year, has found encouraging trends recently. In its 2024–2025 report, based on more than 84,000 students at 135 institutions, 37 percent of students screened positive for moderate to severe depression, down from 44 percent in 2022; 32 percent screened positive for moderate to severe anxiety, down from 37 percent; and 11 percent reported suicidal ideation in the past year, down from 15 percent. About 37 percent of students had received therapy or counseling in the past year, and among students with symptoms of depression or anxiety, 60 percent had received some form of treatment. These numbers describe improvement, and they also describe a large population of students who are struggling at any given moment. In a chapter of one hundred members, the Healthy Minds figures suggest that dozens may be experiencing significant symptoms of depression or anxiety, and several may have had thoughts of suicide in the past year. They are in the chapter meeting. They are at the formal. They are in the group chat. Some of them are officers. Chapter 1 described the Jed Foundation's 2024 report on fraternities and mental health. Its findings are a useful guide for leaders because they point both to strengths and to gaps. Fraternity members, on average, reported lower rates of depression, anxiety, and suicidal ideation than non-members, which suggests that the belonging chapters provide is genuinely protective. But fraternity members were also no more likely to use therapy than non-members, despite greater awareness of campus services, and the report identified shame, secrecy, and norms that equate struggle with weakness as barriers to help-seeking. Sorority members were more likely than non-members to use therapy, a difference that suggests the barriers are cultural and changeable rather than inherent to chapter life. Leaders matter because they shape those norms more than anyone else. Members take their cues about what is acceptable from the people at the front of the room. A president who mentions in a chapter meeting that she sees a counselor, or a fraternity president who tells the new-member class that he went through a hard semester and got help, does more to reduce stigma than any poster in the house. Leaders also control the practical conditions described throughout this book, including the calendar, the new-member program, the risk-management procedures, and the response to crises. When they treat these as mental health decisions, the chapter's culture follows. What a mentally healthy chapter does The Jed Foundation report offers eight recommendations for fraternities, and they apply equally well to sororities and to culturally based organizations. They are to make mental health a priority and set up systems for assessment and tracking; to affirm and build on what is already working; to address clear challenges to mental health, especially substance misuse, hazing, and exploitative sexual norms; to reduce shame and secrecy around help-seeking; to enhance self-awareness, capacity for vulnerability, and opportunities for self-care; to conduct regular screenings to identify students at risk; to develop and follow crisis-management procedures; and to make it easy to access resources. Translated into chapter practice, several concrete steps follow. Define the wellness role carefully. Many chapters now have a wellness or mental health chair. This is a valuable position, but it can easily become an informal counselor role, with one student absorbing the crises of dozens of others. The role works best when it is defined as a connector and organizer rather than a caregiver. The wellness chair's job is to know the resources, bring in trainers and speakers, make sure the chapter's crisis procedures are current, advocate for sleep-protecting and stress-reducing changes to the calendar, and help members find professional help. It is not to be the person everyone calls at 2 a.m. The chair should have backup, including another officer and the chapter advisor, and should know that they can hand off any situation that is beyond their capacity. Train more than one person. Programs such as Mental Health First Aid and QPR (Question, Persuade, Refer) teach laypeople how to recognize signs that someone is struggling, how to start a conversation, and how to connect them with help. Many campus counseling centers offer these trainings for free to student organizations. A chapter in which a dozen members have completed such training is far more resilient than one that relies on a single wellness chair. Some national organizations require or subsidize this training for officers. Make access easy. Barriers to help are often mundane: not knowing where the counseling center is, not knowing how to make an appointment, not knowing whether it costs money, fearing that someone will see you walk in. A chapter can lower these barriers by putting the counseling center's phone number, crisis line, and hours in the chapter's shared documents and in the house, inviting a counselor to speak at a chapter meeting so that members meet a real person, organizing a group visit to the center, and normalizing talk about appointments. The simplest version is a single page, posted where every member will see it, listing the resources in Table 5. Table 5. Where chapter members can turn for help. Resource What it offers How to reach it 988 Suicide and Crisis Lifeline 24/7 crisis support by phone, text, or chat Call or text 988 (US) Crisis Text Line 24/7 text-based crisis support Text HOME to 741741 (US) Campus counseling center Counseling, crisis appointments, after-hours line Campus health or counseling website Fraternity and sorority life office Advising, hazing reports, chapter support Campus student affairs website National organization headquarters Health and safety staff, anonymous reporting Organization's website Emergency services Medical, safety, and overdose emergencies Call 911 Build a crisis protocol and practice it. Chapters should have a written procedure for the most serious situations: a medical emergency at an event, a member who expresses suicidal intent, a sexual assault report, a hazing allegation, and the death of a member. The procedure should specify who calls whom, which campus offices and advisors are notified, and how the chapter communicates with members. It should be reviewed with every new set of officers. The death of a member, whether by suicide, accident, or illness, is among the most difficult events a chapter can face, and campus counseling centers and national organizations can help chapters respond in ways that support grieving members and reduce the risk to others. Organizations such as the American Foundation for Suicide Prevention and the Jed Foundation have published guidance on responding after a suicide in a school or campus community. Check in regularly. Screening in the Jed Foundation's sense can be as simple as encouraging members to use campus or online self-assessment tools, and as structured as brief wellbeing check-ins built into big-little relationships or small-group "families" within the chapter. The goal is that no member's struggles go unnoticed for months. Protect academic life. Academic stress is among the most common triggers for mental health difficulties. Chapters that maintain study support, protect exam periods, and treat a member's academic difficulty as a problem to be solved together rather than a standards violation to be punished tend to see better outcomes on both fronts. Where chapters have minimum grade requirements, which many do, the response to a member who falls below them can be supportive before it is disciplinary. Changing a culture without breaking the chapter Many chapter leaders reading this will recognize the gap between the chapter described above and their own. The question is how to close it. Culture change is slow, and it faces particular obstacles in fraternity and sorority life: one-year officer terms, deep attachment to tradition, older members and alumni who remember things differently, and the ever-present fear that changing too much will make the chapter less attractive in recruitment. Several principles help. Start with the chapter's own values. Nearly every fraternity and sorority has a ritual, creed, or founding statement that speaks of care for members, honor, integrity, scholarship, and service. Change framed as a return to those values is far more persuasive than change framed as compliance with outside rules. "Our creed says we will care for each other. Is this how we care for each other?" is a question few members can easily dismiss. Find allies before you need them. Because private discomfort with harmful practices is usually more common than it appears, a leader who talks privately with several respected members before proposing a change often finds support that was invisible in chapter meetings. Change proposed by a group is harder to dismiss than change proposed by one person. Replace rather than just remove. Traditions persist because they meet real needs: marking transitions, building bonds, creating stories. A chapter that simply bans a tradition leaves a void that is often filled by an underground version. A chapter that replaces a hazing ritual with a meaningful alternative, such as a demanding service project, a retreat built around the chapter's history, or a mentoring program that pairs new members with alumni, meets the same needs without the harm. Work with alumni rather than around them. Alumni often have enormous influence, and some resist changes to practices they remember fondly. Many more are supportive when asked, particularly those who are now parents. Chapters can involve alumni advisors early, explain the reasons for changes, and invite alumni to take part in new traditions. The Stop Campus Hazing Act's transparency requirements, which make hazing violations part of the public record, have given many alumni a new reason to support prevention. Write it down. The single biggest threat to culture change in chapters is officer turnover. A president who transforms the chapter's approach to mental health may see the changes erode within two years if they live only in her leadership. Changes that are written into bylaws, officer handbooks, the new-member curriculum, event-planning procedures, and the chapter's annual calendar are much more likely to survive. So are changes that the new-member class experiences as normal from their first day, because they will teach them to the classes after them. Use outside help. Campus fraternity and sorority life offices can provide training, facilitation, and data. National organizations increasingly employ health and safety staff, provide online training, and operate anonymous reporting systems. Umbrella organizations, including the North American Interfraternity Conference, the National Panhellenic Conference, the National Pan-Hellenic Council, and councils for culturally based organizations, publish standards and resources. Independent organizations offer specialized support: the Jed Foundation on mental health, the Gordie Center on alcohol and hazing, StopHazing and the Hazing Prevention Network on hazing prevention, and many campus-based programs on bystander intervention. Under the Stop Campus Hazing Act, every college must now run research-informed, campus-wide hazing prevention programs, so chapters that want help with prevention should find campuses more ready to provide it than before. An illustrative example shows how these principles fit together. Imagine a fraternity chapter of about sixty members whose new-member period has long ended with a week of late nights, early wake-ups, and a final night of heavy drinking, which older members describe as the thing that "made us brothers." A newly elected president, who privately hated that week when he went through it, spends his first month talking one-on-one with a dozen respected members. He finds that most of them have mixed feelings, and several share his view but assumed they were alone. With three of them, he drafts a proposal: the final week will be replaced with a weekend retreat at a campsite, led by two alumni advisors, built around the chapter's founding story and ritual, with a demanding day of service at a local food bank and an evening in which each older member tells the new members, sincerely, what the chapter has meant to him. The proposal is introduced at a chapter meeting not as a ban but as a return to the chapter's creed. There is pushback, some of it heated, and a few alumni call the president to complain. But the proposal passes, the retreat happens, and the new members describe it afterward as the moment they felt most connected to the chapter. The president writes the retreat into the chapter's new-member manual and the officer handbook. Two years later, most members of the chapter have never experienced anything else, and the old week has become a story that older alumni tell. Nothing in that example is guaranteed. Some chapters resist change more fiercely, and some changes fail. But the pattern of private conversations, allies, a replacement rather than a ban, grounding in the chapter's own values, involvement of alumni, and written institutionalization is the pattern that prevention researchers and experienced campus staff describe again and again when change succeeds. The role of advisors, staff, and parents Students lead chapters, but they do not do it alone, and the adults around them have responsibilities that are easy to underestimate. Chapter advisors and alumni volunteers have continuity that students lack. They can carry lessons from one set of officers to the next, remind a new president of the chapter's crisis protocol, and notice when a practice that was stopped has quietly resumed. They are also, as the hazing research showed, sometimes present when harm occurs; an advisor who treats hazing or heavy drinking as harmless tradition gives it cover. Advisors who make clear that they expect members to call for help, protect sleep, and treat new members with respect are among the most powerful allies a reforming chapter can have. Campus fraternity and sorority life staff operate at the level of the whole community. The Hazing Prevention Framework developed by Allan and her colleagues describes what effective campus prevention requires, from sustained leadership commitment and adequate staffing to assessment, planning, and evaluation. Staff can use data from campus surveys to show chapters how their members' actual attitudes compare with perceived norms, which is one of the most effective tools for shifting behavior. They can also make sure that the medical amnesty policy is well known and consistently applied, because a policy that students doubt will not change what they do. Parents occupy a quieter but important position. They cannot and should not manage their student's chapter life, but they can ask good questions. How much time is the chapter taking? Are you sleeping? What does the new-member period involve? Do you know what to do if someone at a party needs help? Is there anything happening that you would not want me to know about? That last question, asked calmly and without an immediate threat of consequences, gives a student permission to talk. Parents can also model the attitude toward help-seeking that this book argues for: a student who knows that a parent will respond to "I'm struggling" with support rather than alarm or disappointment is much more likely to say it. None of this requires any single person to be heroic. It requires many people, students and adults, to make small decisions consistently in the same direction: a meeting that ends on time, a counselor invited to speak, a new-member program with nothing after 10 p.m., a sober monitor who knows she has the authority to call, a president who says out loud that she sees a therapist. Over a few years, those decisions add up to a different chapter. Conclusion Go back through the situations this book has described and a pattern appears. The potential new member deciding whether to ask a chapter how many hours its new-member program really takes. The new member standing at a 6 a.m. workout that only new members attend. The first-year handed a fourth drink. The sober monitor kneeling beside a guest who will not wake up, while two older members warn about probation. The sophomore muting the group chat at 11 p.m. The member telling a friend in crisis that she wants to help and that the friend deserves more help than she can give. The junior at the planning meeting asking, "What is this tradition actually for?" The president telling the chapter that he sees a therapist. Each of these is a moment in which someone either declines something or asks for something, and each carries a price. Sometimes the price is trivial: a shrug, a few seconds of awkwardness. Sometimes it is high: a friendship strained, a chapter sanctioned, a status lost. What the chapters of this book suggest, taken together, is that the size of that price is the single most revealing fact about a chapter's health. The cost of no Here, then, is a test that any member, officer, advisor, or parent can apply to any chapter: How much does it cost a member to say no? To say no to a drink. To say no to an event on a school night. To say no to a new-member activity that feels wrong. To say no to a leadership role in a hard semester. To say no to keeping a dangerous situation inside the chapter, which means calling 911. To say no to the idea that struggling is weakness, which means saying "I'm not okay." In a chapter where these refusals are cheap, where declining a drink draws no comment, where leaving early is unremarkable, where calling for help is praised, where officers talk openly about their own sleep and their own counseling, members are protected from most of the harms this book has described. Hazing cannot take hold where new members can decline without consequence and older members can object without being shunned. Alcohol emergencies end in phone calls rather than funerals where calling is expected. Sleep survives where the calendar is designed by people who assume members have lives. Mental health struggles are caught early where saying so carries no shame. In a chapter where these refusals are expensive, the opposite holds, and it holds no matter how many policies are on paper, how many trainings members have completed, or how sincerely the chapter's creed speaks of care. A new member who fears that declining will cost her the bid will endure things she should not. A member who fears that calling 911 will destroy the chapter will hesitate. A president who fears that admitting exhaustion will look like weakness will keep going until something breaks. This test is useful because it cuts through the argument about whether Greek life is good or bad. That argument has no general answer, as Chapter 1 showed, because the same qualities produce both belonging and harm. The cost of no is specific. It can be observed, asked about, and changed. It explains why two chapters of the same organization on neighboring campuses can be so different. And it points directly to what needs to change. Why a cheap no makes a stronger yes There is an objection that chapter members sometimes raise, and it deserves an answer. If saying no becomes too easy, won't commitment collapse? Won't members skip everything, drink with no one, and treat the chapter as a club they drop into when convenient? Isn't some cost to declining what makes membership mean something? The answer, drawn from everything in the preceding chapters, is that this gets the relationship backward. A yes that is extracted under pressure is not commitment. It is compliance, and compliance is brittle. Members who stay because leaving is too costly, who attend because missing is punished, who drink because refusing is mocked, are not bound to the chapter by loyalty. They are held by fear, and fear produces the resentment, burnout, secrecy, and sudden departures that chapters often puzzle over. A yes that is freely given is different. When a member who could easily decline chooses to show up at the 8 a.m. philanthropy event, it means something. When a new member who knows he could walk away chooses to stay, his commitment is real. When a chapter's traditions are ones that members would choose even if they could opt out, those traditions have earned their place. The research on hazing found that hazing does not reliably build the cohesion its defenders claim. The belonging that lasts is the kind people choose. This is also why the chapter's strengths, its loyalty, its identity, its traditions, are not the enemy of wellness. They are its best resource. Loyalty can mean "we don't let anyone leave" or "we always come for each other." Identity can mean "we're the hardest-partying house" or "we're the chapter that looks out for its people." Tradition can mean "we did it to them because it was done to us" or "we pass on what matters." The chapters that change are rarely the ones that abandon their identity. They are the ones that redefine it. What the law can do and what it cannot The Stop Campus Hazing Act marks a real shift. For the first time, colleges across the country must count hazing incidents, publish the names of organizations found responsible, and run research-informed prevention programs. The first annual security reports including hazing statistics arrive this fall. Families choosing a chapter can see a public record. Chapters now have one more reason, grounded in reputation, to take prevention seriously. But the law operates at the level of institutions and records, and the cost of no is set in rooms. No statute can decide whether the older member shrugs or sneers when a first-year turns down a drink. No transparency report can decide whether the president answers messages at 3 a.m. No policy can make a sober monitor believe that her chapter will stand behind her. Those things are decided by students, one moment at a time, and by the advisors, staff, and parents who shape what students believe is normal. That is why this book has spent so much of its length on ordinary weeks and small decisions. The dramatic cases, the deaths that give laws their names, are the visible end of a long chain. Their beginnings lie in things like a new-member schedule with no end times, a group chat that never goes quiet, a comment that no one challenged, a tradition that everyone privately doubted and no one questioned aloud. A chapter worth belonging to Return, finally, to the sophomore on the floor of her chapter house at 11:40 on a Tuesday night, with a problem set due in the morning and a big sister texting about movie night. What she needs is not a lecture on time management. She needs to believe that she can text back, "I'm going to bed, see you Thursday," and that her big will reply with a heart and nothing more. She needs a chapter meeting that ended at 9:30 instead of 10:30. She needs to know that if the flatness in the mornings continues, she can mention it to a friend in the chapter and be walked to the counseling center rather than told everyone feels that way. Every one of those things is within the power of the people around her. None of them requires the chapter to be less ambitious, less fun, or less close. They require only that the chapter decide, deliberately and repeatedly, that its members' ability to say no is something to protect rather than something to overcome. A chapter that makes that decision is not a weaker version of Greek life. It is the version that delivers on what fraternities and sororities have always promised: a group of people who will look after you, who expect the best of you, and who will still be there when you need them. That promise is worth keeping. Keeping it starts with making no an easy word to say. Notes Introduction 1. Elizabeth J. Allan and Mary Madden, Hazing in View: College Students at Risk. Initial Findings from the National Study of Student Hazing (2008). https://stophazing.org/research/national-hazing-study-hazing-in-view/ 1. Stop Campus Hazing Act, Public Law 118-173, signed December 23, 2024. Summary: Clery Center, "The Stop Campus Hazing Act: What You Need to Know." https://www.clerycenter.org/scha-what-you-need-to-know 2. 988 Suicide and Crisis Lifeline. https://988lifeline.org/ Chapter 1: What Belonging Gives and What It Can Cost 1. Roy F. Baumeister and Mark R. Leary, "The Need to Belong: Desire for Interpersonal Attachments as a Fundamental Human Motivation," Psychological Bulletin 117, no. 3 (1995): 497–529. 2. Healthy Minds Network, The Healthy Minds Study: 2024–2025 Data Report (2025); summary figures from University of Michigan School of Public Health, "Healthy Minds Study: College Student Depression, Anxiety Decline for Third Consecutive Year" (2025). https://sph.umich.edu/news/2025posts/college-student-mental-health-third-consecutive-year-improvement.html 3. Gallup, Fraternities and Sororities: Understanding Life Outcomes (2014); summary at https://news.gallup.com/opinion/gallup/173630/fraternity-sorority-membership-linked-higher-college-grads.aspx 4. The Jed Foundation, Fraternities and Mental Health: Supporting Emotional Well-Being (December 2024). https://jedfoundation.org/wp-content/uploads/2024/12/Fraternities-and-Mental-Health-ReportFinalDec-3-2024.pdf 5. Postsecondary Education Research Center, University of Tennessee, Knoxville, 2020 Mental Health Study (Foundation for Fraternal Excellence research grant report, 2021); summary at https://nicfraternity.org/fraternity-members-report-higher-levels-of-positive-mental-health-and-support/ 6. National Institute on Alcohol Abuse and Alcoholism, "College Drinking: Prevalence," citing the 2023 National Survey on Drug Use and Health. https://www.collegedrinkingprevention.gov/statistics/prevalence 7. Sean Esteban McCabe, Philip Veliz, and John E. Schulenberg, "How Collegiate Fraternity and Sorority Involvement Relates to Substance Use During Young Adulthood and Substance Use Disorders in Early Midlife: A National Longitudinal Study," Journal of Adolescent Health 62, no. 3 (2018): S35–S43. 8. Sean Esteban McCabe et al., "Selection and Socialization Effects of Fraternities and Sororities on US College Student Substance Use: A Multi-Cohort National Longitudinal Study," Addiction 100, no. 4 (2005): 512–524. 9. H. Wesley Perkins and Alan D. Berkowitz, "Perceiving the Community Norms of Alcohol Use Among Students: Some Research Implications for Campus Alcohol Education Programming," International Journal of the Addictions 21, nos. 9–10 (1986): 961–976. 10. Deborah A. Prentice and Dale T. Miller, "Pluralistic Ignorance and Alcohol Use on Campus: Some Consequences of Misperceiving the Social Norm," Journal of Personality and Social Psychology 64, no. 2 (1993): 243–256. Chapter 2: Recruitment, Intake, and the New-Member Months 1. Naomi I. Eisenberger, Matthew D. Lieberman, and Kipling D. Williams, "Does Rejection Hurt? An fMRI Study of Social Exclusion," Science 302, no. 5643 (2003): 290–292. 2. On the hazing findings referred to in this chapter, see note 1. Chapter 3: Hazing, Named Plainly 1. On the death of Timothy Piazza and Pennsylvania's 2018 anti-hazing law: "'Why Didn't You Call for Help?' Parents of Timothy Piazza, Penn State Community Grapple with Loss 5 Years Later," The Daily Collegian (2022). 2. Federal definition of hazing under the Stop Campus Hazing Act, as reproduced by the University of Florida, "Federal and State Definitions." https://stophazing.ufl.edu/what-is-hazing/federal-and-state-definitions/ ; and Kettering University, "Stop Campus Hazing Act." https://www.kettering.edu/stop-campus-hazing-act 3. Implementation dates: Clery Center (note 2); StopHazing, "Federal Anti-Hazing Law: Stop Campus Hazing Act." https://stophazing.org/policy/stop-campus-hazing-act/ 4. Allan and Madden, Hazing in View (note 1). Full report: https://stophazing.org/wp-content/uploads/2014/06/hazinginview_web1.pdf 5. Elizabeth J. Allan, Jessica M. Payne, and David Kerschner, "Transforming the Culture of Hazing: A Research-Based Hazing Prevention Framework," Journal of Student Affairs Research and Practice 55, no. 4 (2018): 412–425. 6. Elizabeth J. Allan, David Kerschner, and Jessica M. Payne, "College Student Hazing Experiences, Attitudes, and Perceptions: Implications for Prevention," Journal of Student Affairs Research and Practice 56, no. 1 (2019): 32–48. 7. Elliot Aronson and Judson Mills, "The Effect of Severity of Initiation on Liking for a Group," Journal of Abnormal and Social Psychology 59, no. 2 (1959): 177–181. 8. Aldo Cimino, "The Evolution of Hazing: Motivational Mechanisms and the Abuse of Newcomers," Journal of Cognition and Culture 11 (2011): 241–267. 9. StopHazing, "Hazing Prevention Framework." https://stophazing.org/resources/hazing-prevention-framework/ Chapter 4: Alcohol, Emergencies, and the Call That Matters 1. National Institute on Alcohol Abuse and Alcoholism, "College Drinking: Prevalence" (note 9) and "College Drinking: Consequences." https://www.collegedrinkingprevention.gov/statistics/consequences 2. North American Interfraternity Conference, "NIC Fraternities Ban Hard Alcohol in Decisive Action" (2018). https://nicfraternity.org/nic-fraternities-vote-to-ban-hard-alcohol/ 3. National Institute on Alcohol Abuse and Alcoholism, "Understanding the Dangers of Alcohol Overdose." https://www.niaaa.nih.gov/publications/brochures-and-fact-sheets/understanding-dangers-of-alcohol-overdose 4. The Gordie Center, University of Virginia, "Medical Amnesty," including the Cornell survey findings on calling for help, the number of states with medical amnesty laws as of 2022, and the 2018 evaluation of a medical amnesty policy change. https://gordie.studenthealth.virginia.edu/learn/medical-amnesty 5. Deborah K. Lewis and Timothy C. Marchell, "Safety First: A Medical Amnesty Approach to Alcohol Poisoning at a U.S. University," International Journal of Drug Policy 17, no. 4 (2006): 329–338. Chapter 5: Sleep and the Chapter Calendar 1. Nathaniel F. Watson et al., "Recommended Amount of Sleep for a Healthy Adult: A Joint Consensus Statement of the American Academy of Sleep Medicine and Sleep Research Society," Journal of Clinical Sleep Medicine 11, no. 6 (2015): 591–592. 2. Shelley D. Hershner and Ronald D. Chervin, "Causes and Consequences of Sleepiness Among College Students," Nature and Science of Sleep 6 (2014): 73–84. 3. American Academy of Sleep Medicine, "Insufficient Sleep Linked to Mental Health in College Students" (2019), reporting an abstract presented at SLEEP 2019 by Thea Ramsey, Michael Grandner, and colleagues. https://aasm.org/insufficient-sleep-college-student-athletes/ Chapter 6: Boundaries and the Art of Saying No 1. Peter M. Gollwitzer and Paschal Sheeran, "Implementation Intentions and Goal Achievement: A Meta-Analysis of Effects and Processes," Advances in Experimental Social Psychology 38 (2006): 69–119. 2. Thomas Gilovich, Victoria Husted Medvec, and Kenneth Savitsky, "The Spotlight Effect in Social Judgment: An Egocentric Bias in Estimates of the Salience of One's Own Actions and Appearance," Journal of Personality and Social Psychology 78, no. 2 (2000): 211–222. 3. Francis J. Flynn and Vanessa K. B. Lake (Bohns), "If You Need Help, Just Ask: Underestimating Compliance with Direct Requests for Help," Journal of Personality and Social Psychology 95, no. 1 (2008): 128–143; see also Vanessa K. Bohns, You Have More Influence Than You Think (W. W. Norton, 2021). Chapter 7: Stepping In When It Is Your Own Chapter 1. Bibb Latané and John M. Darley, "Group Inhibition of Bystander Intervention in Emergencies," Journal of Personality and Social Psychology 10, no. 3 (1968): 215–221; and The Unresponsive Bystander: Why Doesn't He Help? (Appleton-Century-Crofts, 1970). 2. Richard Philpot, Lasse Suonperä Liebst, Mark Levine, Wim Bernasco, and Marie Rosenkrantz Lindegaard, "Would I Be Helped? Cross-National CCTV Footage Shows That Intervention Is the Norm in Public Conflicts," American Psychologist 75, no. 1 (2020): 66–75. 3. Mark Levine, Amy Prosser, David Evans, and Stephen Reicher, "Identity and Emergency Intervention: How Social Group Membership and Inclusiveness of Group Boundaries Shape Helping Behavior," Personality and Social Psychology Bulletin 31, no. 4 (2005): 443–453. 4. Step UP! bystander intervention program. https://stepupprogram.org/ Chapter 8: Leading a Chapter Toward Mental Health 1. Healthy Minds Study 2024–2025 figures (note 5). 2. The Jed Foundation, Fraternities and Mental Health (note 7), for the findings on help-seeking and the eight recommendations. 3. Crisis Text Line. https://www.crisistextline.org/ 4. On campus prevention requirements under the Stop Campus Hazing Act, see notes 2 and 18; on the Hazing Prevention Framework, see notes 20 and 24. Conclusion 1. On the reporting timeline for hazing statistics in annual security reports, see note 2. Further Reading Elizabeth J. Allan and Mary Madden, Hazing in View: College Students at Risk (2008).** The foundational national study of college hazing, freely available from StopHazing. Short, readable, and essential for anyone who wants to understand how common hazing is and why it goes unrecognized. StopHazing (stophazing.org). The research and training organization founded by Elizabeth Allan. Its site includes the Hazing Prevention Framework, plain-language explainers on the Stop Campus Hazing Act, and resources for students, parents, and campus professionals. Hank Nuwer, Hazing: Destroying Young Lives (Indiana University Press, 2018).** An edited volume by the journalist who has documented hazing deaths for decades, bringing together researchers and practitioners on hazing in fraternities, sororities, athletics, the military, and beyond. Walter M. Kimbrough, Black Greek 101: The Culture, Customs, and Challenges of Black Fraternities and Sororities (Fairleigh Dickinson University Press, 2003).** A clear introduction to the history and culture of historically Black Greek-letter organizations, including an honest treatment of the move from pledging to membership intake and the persistence of underground hazing. The Jed Foundation, Fraternities and Mental Health: Supporting Emotional Well-Being (2024).** A research-based report with practical recommendations for chapters, national organizations, and campuses. Useful for any officer building a wellness program. The Gordie Center, University of Virginia (gordie.studenthealth.virginia.edu). Free materials on recognizing alcohol poisoning, medical amnesty laws by state, and hazing prevention, created in memory of Gordie Bailey. National Institute on Alcohol Abuse and Alcoholism, College Drinking Prevention (collegedrinkingprevention.gov). Reliable statistics on college drinking, guidance for parents, and CollegeAIM, a guide that rates campus alcohol interventions by evidence. Matthew Walker, Why We Sleep (Scribner, 2017).** A popular account of sleep science by a neuroscientist. Some of its claims have been criticized as overstated, so read it alongside more cautious sources, but it conveys vividly why sleep matters for learning and emotion. Vanessa K. Bohns, You Have More Influence Than You Think (W. W. Norton, 2021).** A social psychologist's accessible account of how people underestimate their influence on others and how hard it is for others to say no, with direct relevance to boundaries and peer pressure. 988 Suicide and Crisis Lifeline (988lifeline.org) and the American Foundation for Suicide Prevention (afsp.org). Crisis support available by call or text at 988 in the United States, and educational materials on recognizing warning signs and supporting someone at risk. Hashtags: #GreekLifeWellness #FraternityAndSororityLife #GreekLifeBalance #CollegeStudentWellness #StudentMentalHealth #CampusMentalHealth #SleepHealth #HealthySleepHabits #SleepDeprivation #TimeManagement #ChapterObligations #AcademicLifeBalance #StudentWellbeing #MentalHealthAwareness #CollegeStressManagement #BurnoutPrevention #HealthyBoundaries #PeerPressure #StudentSelfCare #FraternityWellness #SororityWellness #ChapterLeadership #CampusSafety #HazingPrevention #AlcoholSafety #BystanderIntervention #StudentBelonging #SupportiveCommunities #MentalHealthSupport #HealthyCampusCulture

Download the Book (PDF): Introduction Every year, in every American hospital, some patients are hurt by the care meant to help them. A surgeon nicks a bowel during a routine laparoscopy and the leak goes unnoticed for two days. A radiologist reads a chest film as clear when a small shadow at the edge of the lung is the first sign of cancer. An emergency physician sends home a forty-year-old with chest pain and a normal first troponin, and he dies that night of a heart attack. A nurse hangs the wrong concentration of heparin. A child is born with brain damage after a labor in which the fetal heart tracing was, or was not, read correctly. Some of these events are the product of carelessness. Some are the unavoidable price of doing difficult things to sick bodies under uncertainty. Many sit in a zone where reasonable clinicians would disagree about whether anything went wrong at all. The law of medical malpractice is the machinery American society uses to sort these cases: to decide which injuries were wrongful, who must pay for them, and how much. It is also, whether it intends to be or not, a regulator of medical behavior, a source of anxiety for physicians, a line item in the national health budget, and the subject of one of the longest-running political fights in American law. This book is about how that machinery works and how well it does its job. It concentrates on three questions that together determine almost everything about a malpractice case and about the policy debate that surrounds malpractice law. The first is the standard of care. Negligence law ordinarily asks what a reasonable person would have done. Medicine is different: the law has long let the profession define its own standard, asking not what a reasonable person would do but what a reasonable physician in the same specialty would do. That choice has large consequences. It means the standard is found in professional custom rather than in the jury's common sense, and it means that almost every malpractice case turns into a contest between experts. The second is causation. Patients who seek medical care are, by definition, already sick or at risk. When things go badly, the law must separate the harm the provider caused from the harm the disease would have caused anyway. That is often harder than proving the error. A missed diagnosis of cancer is a clear mistake; whether earlier detection would have saved the patient is a statistical question that the traditional all-or-nothing rules of tort law handle awkwardly. The third is defensive medicine: the tests, referrals, procedures and avoidance behaviors that physicians adopt not because they help patients but because they reduce, or are thought to reduce, legal risk. Defensive medicine is the main reason malpractice law matters to people who will never be injured or sued. It is also the main argument offered for tort reform, and particularly for caps on damages. Whether the argument holds up is an empirical question, and there is now a substantial body of evidence on it. The argument of this book These three subjects are usually discussed separately: the standard of care and causation by lawyers, defensive medicine by economists and health policy scholars. They belong together, because they are connected by a single chain of reasoning. Because the standard of care is defined by professional practice and proven by experts, liability in any given case is uncertain and expensive to establish. Because causation must be proven against the background of disease, many genuine injuries caused by negligence are never compensated, while some claims succeed or settle in cases where the care was defensible. The result is a system that performs poorly as a sorting mechanism. The best empirical studies show that most patients injured by negligence never file a claim, that a meaningful share of claims involve no error at all, and that over half of every dollar the system spends goes to lawyers, experts, insurers and courts rather than to injured patients. Physicians experience that inaccuracy as randomness, and randomness breeds fear. Fear, far more than actual liability exposure, drives defensive medicine. That is why the most popular reform, capping noneconomic damages, has done less than its advocates hoped. Caps reduce what plaintiffs recover, sometimes sharply, and they reduce the number of cases lawyers are willing to bring, especially on behalf of children, the elderly and others with modest earnings. But they do little to make liability more predictable for the physician deciding whether to order a scan, and the best studies find that their effect on overall health spending is modest. The evidence points toward a different conclusion: reforms that make the system more accurate and less adversarial, rather than merely less generous, are the ones that stand a chance of changing medical behavior. That argument is developed across nine chapters. Chapter 1 sets out the anatomy of a malpractice claim: its elements, the parties, the procedure, and what actually happens to claims once they are filed. Chapter 2 examines the standard of care itself, from the old locality rule to the modern national standard, and the uneasy role of custom, clinical guidelines and emerging technology. Chapter 3 treats informed consent, the branch of malpractice law that asks not whether the treatment was done well but whether the patient agreed to it on adequate information. Chapter 4 turns to the expert witness, the figure on whom nearly everything depends, and to the screening devices, qualification rules and evidentiary gatekeeping that states and federal courts use to police expert testimony. Chapter 5 addresses causation, including the loss-of-chance doctrine that courts have developed, and disagreed about, for patients whose odds of survival were reduced rather than destroyed. Chapter 6 explains how damages are valued, and why noneconomic damages became the battleground. Chapter 7 traces the history of tort reform, from the insurance crises of the 1970s to the damage caps now in force in many states, and the constitutional litigation that has struck some of those caps down. Chapter 8 weighs the evidence on defensive medicine. Chapter 9 surveys the alternatives: communication-and-resolution programs, apology laws, administrative compensation schemes and the models used in other countries. The conclusion draws out what follows. Scope and a word on method The focus is American law. Malpractice in the United States is almost entirely a creature of state law, so there are fifty variations on every rule described here. The book explains the dominant approaches, notes the important minority positions, and uses particular states, especially California and Texas, as worked examples because their reforms have been the most influential and the most studied. Where other countries have taken instructive paths, particularly Britain on informed consent and New Zealand and the Nordic countries on no-fault compensation, the comparison is drawn. The book relies throughout on the empirical literature, which is unusually rich. Researchers have reviewed hospital records to count negligent injuries, followed closed claims from filing to payment, analyzed the federal National Practitioner Data Bank, surveyed physicians about their practices, and exploited natural experiments, such as the immunity from malpractice suits that applies to care delivered on military bases, to measure what liability actually does to clinical behavior. Readers who want to follow the evidence further will find the key studies identified in the Notes and in Further Reading. Nothing in this book is legal advice. It is written for readers who want to understand how one of the most contested areas of American civil law actually works: patients and families trying to make sense of a bad outcome, clinicians who want to know what the law does and does not demand of them, students, policymakers, and anyone who has wondered whether the extra test their doctor ordered was for their benefit or for the doctor's. Malpractice law is often discussed as if it were only about blame and money. It is really about knowledge: what the medical profession knows about how care should be delivered, what a court can know about what happened in a particular operating room, and what anyone can know about what would have happened otherwise. The system's failures are, at bottom, failures of knowledge, and so are the most promising reforms. Chapter 1: The Anatomy of a Malpractice Claim Medical malpractice is not a separate body of law so much as a specialized application of ordinary negligence. The plaintiff must prove the familiar four elements: that the defendant owed a duty of care, that the defendant breached it, that the breach caused an injury, and that the injury produced legally compensable damages. What makes malpractice distinct is how each of those elements is filled in, who proves them, and the dense layer of procedural rules that legislatures have built around them. The elements, the questions each asks, and the kind of proof each typically demands are set out in Table 1. The rest of this chapter explains how they operate in practice, who the defendants are, and what happens to a claim once it enters the system. Table 1. The elements of a medical malpractice claim. Element Core question Typical proof Common points of dispute Duty Was there a provider-patient relationship? Records, appointments, consultation notes Curbside consults, on-call physicians, telehealth Breach Did care fall below the professional standard? Expert testimony on standard of care Competing schools of thought, judgment calls Causation Did the breach cause the harm? Expert medical and statistical testimony Preexisting disease, delayed diagnosis, loss of chance Damages What losses resulted? Economists, life care planners, testimony Noneconomic damages, caps, future care costs Duty and the provider-patient relationship In most negligence cases duty is barely contested: everyone owes everyone else reasonable care not to cause physical harm. In malpractice, the professional duty arises from the relationship between provider and patient, and that relationship must exist before the specialized standard applies. Usually its existence is obvious. A patient who books an appointment, is admitted to a hospital or is examined in an emergency department is plainly in a treatment relationship. The difficult cases lie at the edges. A physician who gives an informal opinion to a colleague in a hallway, the so-called curbside consult, generally does not assume a duty to the colleague's patient, though courts have found a relationship where the consulting physician reviewed records, gave specific advice and knew it would be relied on. An on-call specialist who is telephoned by an emergency physician and declines to come in presents a closer question, and results vary with the facts and with hospital bylaws that may oblige on-call physicians to respond. Physicians who examine people on behalf of employers or insurers owe a narrower duty than a treating physician, typically limited to not causing harm in the examination itself, though some courts extend it to disclosing serious findings. Telehealth, which expanded dramatically during the COVID-19 pandemic and has remained a large part of practice, has added new variants. A virtual visit creates a treatment relationship just as an office visit does. The harder questions concern which state's law governs when patient and physician are in different states, and whether the standard of care for a video consultation should account for the physician's inability to touch or examine the patient directly. The general answer emerging from practice and from licensing boards is that the medium does not lower the standard: if a condition cannot safely be evaluated remotely, the reasonable course is to say so and arrange an in-person examination. Emergency departments carry an additional federal overlay. The Emergency Medical Treatment and Active Labor Act of 1986 requires hospitals that participate in Medicare and have emergency departments to screen anyone who comes seeking care and to stabilize emergency conditions before transfer or discharge. The statute creates a federal cause of action against hospitals, but it is not a federal malpractice law: courts have consistently held that it guarantees an appropriate screening, applied equally to all patients, not a correct diagnosis. Who is sued The individual physician is the classic defendant, but modern malpractice litigation typically names several parties, and the institutions often matter more than the individuals. Hospitals are liable in two distinct ways. First, they are vicariously liable for the negligence of their employees, including nurses, pharmacists, technicians and, increasingly, employed physicians. A growing share of American physicians are now employees of hospitals and health systems rather than independent practitioners, which makes the institution the natural target. Second, where the physician is an independent contractor, as many emergency physicians, anesthesiologists and radiologists still are, courts frequently impose liability through the doctrine of apparent or ostensible agency: if the hospital held the physician out as part of its services and the patient reasonably relied on the hospital rather than choosing the physician, the hospital answers for the physician's negligence. Hospitals also owe duties of their own. In Darling v. Charleston Community Memorial Hospital, decided by the Illinois Supreme Court in 1965, a young man's broken leg was put in a cast that was too tight; nurses failed to report the signs of circulatory compromise, no one reviewed the general practitioner's handling of the case, and the leg was eventually amputated. The court held the hospital liable for its own failures of supervision, laying a foundation for what became known as corporate negligence. Later decisions, such as Thompson v. Nason Hospital in Pennsylvania in 1991, articulated specific institutional duties: to maintain safe facilities and equipment, to select and retain only competent physicians, to oversee all persons who practice medicine within its walls, and to formulate and enforce adequate rules and policies. The negligent credentialing claim, alleging that a hospital granted or renewed privileges to a physician it knew or should have known was incompetent, has become an important tool where the physician's own insurance is inadequate. Other defendants include physician group practices, nurse practitioners and physician assistants (whose standard of care is a subject of active debate as their scope of practice expands), pharmacists, nursing homes, and manufacturers of drugs and devices, though claims against manufacturers sound in product liability rather than malpractice and face their own obstacles, including federal preemption for many devices. A special category of defendant is the federal government. Physicians employed by the Department of Veterans Affairs, by military hospitals and by federally funded community health centers are covered by the Federal Tort Claims Act, under which the United States is the defendant, the case is tried to a judge without a jury, punitive damages are unavailable, and the claimant must first present an administrative claim within two years. Service members themselves have long been barred from suing for injuries incident to service, including negligent military medical care, under the doctrine of Feres v. United States (1950). In 2019 Congress created a limited administrative claims process allowing service members to seek compensation from the Department of Defense for malpractice by military providers, without disturbing the Feres bar on lawsuits. Procedure: from injury to courtroom Malpractice cases pass through a series of filters before any jury hears them, and most filters are more restrictive than those in ordinary negligence cases. The first filter is the lawyer. Nearly all malpractice plaintiffs are represented on contingency, with the lawyer receiving a percentage of any recovery, commonly a third or more, and nothing if the case fails. Because malpractice cases are expensive to prepare, often requiring hundreds of thousands of dollars in expert fees and costs advanced by the lawyer, plaintiff's attorneys screen aggressively. Surveys of malpractice lawyers have consistently found that they decline the large majority of cases brought to them, usually because damages are too small to justify the cost, even where negligence seems likely. Several states cap contingency fees in malpractice cases on a sliding scale; California's MICRA, for example, has long limited fees, and its 2022 amendments set the limits at 25 percent of a recovery obtained before a lawsuit is filed and 33 percent after. The second filter is time. Statutes of limitations for malpractice are generally shorter than for other torts, commonly one to three years, and many states add a statute of repose that extinguishes claims after a fixed period, often four to ten years, regardless of when the injury was discovered. The discovery rule, which starts the clock only when the patient knew or should have known of the injury and its likely cause, softens the limitations period; the statute of repose limits that softening. Special rules usually apply to minors, to foreign objects left in the body and to fraudulent concealment. The third filter is pre-suit screening. Many states require a plaintiff to give notice to the defendant before filing, sometimes with a waiting period during which the parties exchange records and may negotiate. Many require a certificate or affidavit of merit, in which a qualified expert attests that the claim has a reasonable basis; Texas requires a detailed expert report within 120 days of the defendant's answer, on pain of dismissal with fees. Some states refer claims to medical review panels whose opinions may be admitted at trial. These requirements, discussed further in Chapter 4, are meant to weed out weak claims early. They also raise the cost of bringing any claim at all. Once filed, a malpractice case proceeds through discovery (the exchange of records, depositions of the treating clinicians and expert disclosures), motions to exclude or limit expert testimony, summary judgment motions and, in the rare case that goes the distance, trial. The period from injury to resolution is long. Studies of closed claims have found averages of roughly four to five years from incident to closure, with litigated cases taking longer. What happens to claims The public image of malpractice litigation is shaped by headline verdicts, but the reality is different. Most claims are dropped or dismissed without any payment. Of those that are paid, the overwhelming majority are settled. Only a small fraction, commonly estimated at under 10 percent of claims, reach a jury verdict, and physicians win most of them: studies spanning several decades have found that defendants prevail in roughly three-quarters or more of malpractice trials. Plaintiffs' lawyers try cases they believe are strong, and defendants settle cases they believe are weak, so the cases that reach verdict are disproportionately close ones; even so, juries are markedly more favorable to physicians than popular accounts suggest. Neil Vidmar's research on malpractice juries found that jurors tend to approach claims skeptically and give physicians the benefit of the doubt. Payment data come mainly from the National Practitioner Data Bank, created by the Health Care Quality Improvement Act of 1986, to which every payment made on behalf of a licensed practitioner in settlement or satisfaction of a malpractice claim must be reported. An analysis of Data Bank records from 1992 through 2014 by Adam Schaffer and colleagues, published in JAMA Internal Medicine in 2017, found that the rate of paid claims per physician fell by more than half over the period, from about 20 to about 9 per thousand physician-years, while the average payment rose modestly in real terms to around $350,000. About 8 percent of paid claims exceeded $1 million, roughly a third involved a patient's death, and diagnostic error was the single most common allegation. Paid claims were also concentrated: a small share of physicians accounted for a disproportionate share of payments. Claims are not evenly distributed across medicine. A study by Anupam Jena and colleagues in the New England Journal of Medicine in 2011, using data from a large national insurer, found that about 7.4 percent of physicians faced a claim in any given year, but the annual rate ranged from under 3 percent in psychiatry to about 19 percent in neurosurgery. The authors projected that by age 65, nearly every physician in a high-risk specialty and about three-quarters of those in low-risk specialties would have faced at least one claim. Most claims did not result in payment. This combination, a high chance of being sued and a low chance of having to pay, is central to understanding physicians' experience of the system. For most physicians, being sued is not a financial catastrophe, since they are insured and most claims close without payment. It is a professional and emotional ordeal that can last years. The accuracy problem The most important empirical fact about malpractice litigation is how imperfectly claims track negligence. The Harvard Medical Practice Study, which reviewed more than 30,000 hospital records in New York State from 1984, found that adverse events, meaning injuries caused by medical management, occurred in 3.7 percent of hospitalizations, and that about 28 percent of those events were attributable to negligence. When the researchers matched the records against malpractice claims, they found that only a small fraction of patients injured by negligence ever filed a claim: roughly one claim for every seven or eight negligent injuries. At the same time, many of the claims that were filed involved no negligence the reviewers could detect. A later study in Utah and Colorado found broadly similar patterns. The most careful study of what happens to claims once filed came from David Studdert and colleagues, published in the New England Journal of Medicine in 2006. Physician reviewers examined 1,452 closed claims from five liability insurers. They judged that about 3 percent of claims involved no verifiable injury and that about 37 percent involved no error. Encouragingly, most claims without error received no payment, and most claims with error did. But the system was far from precise: about a quarter of claims were decided in a way that did not match the reviewers' judgment of whether an error occurred, and it spent heavily to get there. For every dollar paid to claimants, the researchers estimated, about 54 cents went to administrative expenses, mainly legal fees, expert costs and insurers' overhead. These findings cut against both sides of the political debate. They refute the claim that the system is dominated by frivolous suits: most claims involve real injuries, and most payments go to cases where something went wrong. They also refute the claim that the system efficiently compensates injured patients: the great majority of negligently injured patients receive nothing, and the process consumes more than half its resources in transaction costs. The rest of this book is, in large part, an investigation of why a system can be so imperfect in both directions at once. The answer begins with the standard of care. Chapter 2: The Standard of Care In ordinary negligence law, the jury decides what a reasonable person would have done. It draws on its own experience: jurors know what careful driving looks like, what a prudent landlord does about an icy step. Custom is relevant but not decisive. In the most famous statement of the point, Judge Learned Hand wrote in The T.J. Hooper (2d Cir. 1932) that an entire industry could lag behind in adopting a cheap precaution, in that case radio receivers on tugboats, and still be negligent, because "a whole calling may have unduly lagged." Medicine has been treated differently. For well over a century, American courts have held that a physician's conduct is measured against the customary practice of the profession: what a reasonably prudent physician in the same field would have done in similar circumstances. The profession, in other words, largely sets its own standard, and the jury learns what that standard is from expert witnesses. This deference to medical custom is the most consequential choice in the whole of malpractice law. It shapes who must testify, what counts as proof, how predictable outcomes are, and why the system behaves as it does. Why the law defers to medical custom Several justifications support the professional standard. The first is competence: jurors cannot judge from ordinary experience whether a surgeon should have converted a laparoscopic gallbladder removal to an open procedure or whether a pediatrician should have ordered a lumbar puncture on a feverish infant. The second is legitimacy: a standard derived from what physicians actually do is one that physicians can know in advance and comply with, rather than one announced after the fact by laypeople. The third is professional autonomy: medicine is self-regulating in many respects, through licensure, specialty certification and peer review, and the law has extended that autonomy to the definition of negligence. The deference has costs. It makes the standard of care dependent on expert testimony in almost every case, which makes litigation expensive and outcomes dependent on the quality and credibility of the experts each side can afford. It risks entrenching outdated practices, since a physician who does what most colleagues do is protected even if what they do is wrong. And it creates an awkward circularity when practice varies widely, as it often does: if half of competent surgeons do one thing and half do another, what is "the" standard? In practice, most jury instructions frame the test in terms of reasonableness informed by custom, rather than custom alone. A typical formulation asks whether the physician exercised the degree of skill, care and learning expected of a reasonably prudent health care provider in the same profession or specialty under similar circumstances. Several states, notably after decisions like Nowatske v. Osterloh (Wis. 1996), have said explicitly that the standard is reasonable care, and that custom is evidence of it rather than its definition. The practical difference is usually small, since the jury will still hear what physicians customarily do from experts, but it leaves room to argue that a widespread practice was itself unreasonable. The most famous instance of a court overriding medical custom is Helling v. Carey, decided by the Washington Supreme Court in 1974. A woman in her twenties had consulted ophthalmologists for years about vision problems, and they had not tested her eye pressure for glaucoma. The accepted practice was not to perform the test routinely on patients under forty, because the disease was rare in younger people. When she was eventually diagnosed, she had suffered permanent vision loss. The court, applying reasoning close to The T.J. Hooper, held as a matter of law that the defendants were negligent: the test was simple, cheap and harmless, and the consequence of missing the disease was grave. The decision provoked a storm in the medical profession and a legislative response in Washington that attempted to reinstate the professional standard. It is still cited, but rarely followed. Its lasting significance is as a reminder that the professional standard is a legal choice, not a law of nature, and that courts retain residual authority to say a custom is unreasonable. From the locality rule to a national standard For much of the nineteenth and twentieth centuries, many American courts applied a locality rule: a physician was measured against the practice of physicians in the same or a similar community. The rule had a sensible origin. A country doctor in the nineteenth century lacked the training, equipment and access to specialists available in a city teaching hospital, and it seemed unfair to hold him to the urban standard. It also had a serious drawback: in a small community, the only people qualified to testify about local practice were the defendant's colleagues and friends, who were often unwilling to testify against him. Plaintiffs' lawyers called this the conspiracy of silence, and it made many meritorious claims impossible to prove. As medical education standardized, specialty boards certified physicians against national criteria, and communication and transport improved, the rationale for the locality rule eroded. The Massachusetts Supreme Judicial Court's decision in Brune v. Belinkoff (1968), involving an anesthetic dose given during childbirth in New Bedford, rejected the argument that local practice could justify a dose that Boston specialists regarded as excessive. The court held that the standard was that of the average qualified practitioner, taking into account advances in the profession and the medical resources available. The Mississippi Supreme Court went further in Hall v. Hilbun (1985), adopting a national standard for both general practitioners and specialists, while allowing the jury to consider the resources actually available to the physician. The resources qualification is the key to the modern approach. Most states now apply a national standard of skill and knowledge, qualified by the circumstances in which the physician practiced. A rural emergency physician without an on-site cardiac catheterization laboratory is not negligent for failing to perform an angioplasty; but he is expected to recognize a heart attack as well as an urban colleague and to know when to give clot-dissolving drugs or arrange transfer. A minority of states retain versions of the locality or similar-locality rule, often by statute, but the practical gap between the approaches has narrowed. The shift has been studied for its effect on clinical behavior. Michael Frakes, in a 2013 article in the American Economic Review, exploited the fact that states adopted national-standard rules at different times. He found that such adoption was associated with a narrowing of the gap between local practice patterns and national norms, for example in cesarean section rates. The finding matters for Chapter 8: it is evidence that the content of the legal standard, and not just the size of damage awards, influences what physicians do. Specialists, generalists and respectable minorities Specialists are held to the standard of their specialty. A board-certified cardiologist is judged by what a reasonable cardiologist would do; a family physician treating a heart condition is judged by what a reasonable family physician would do, which includes knowing when to refer. A generalist who undertakes a procedure normally performed by specialists may be held to the specialist's standard. Non-physician clinicians present a newer version of the question. Nurse practitioners and physician assistants now provide a large share of primary and urgent care, and in many states nurse practitioners practice independently. Courts generally measure them against the standard of their own profession, but where they perform the same diagnostic functions as physicians, plaintiffs argue the patient is entitled to the same level of competence. The issue is unsettled, and it will become more pressing as scope-of-practice laws continue to expand. Medicine rarely has a single right answer, and the law accommodates legitimate disagreement through the doctrine known variously as the two schools of thought or the respectable minority rule. A physician who follows an approach endorsed by a reputable body of practitioners is not negligent merely because another body of practitioners would have chosen differently. The Pennsylvania Supreme Court in Jones v. Chidester (1992) held that the doctrine applies where a considerable number of recognized and respected professionals support the approach, rejecting a stricter test that looked only to whether the school was respected and a looser one that required merely a few adherents. The doctrine protects innovation and clinical judgment, but it also invites each side to assemble experts claiming that its approach is the mainstream. A related principle protects errors of judgment. A physician who, having gathered the appropriate information, chooses between reasonable alternatives is not liable because the choice turned out badly. Courts differ on whether juries should be given a separate "error in judgment" instruction; several state supreme courts have disapproved of such instructions on the ground that they confuse jurors into thinking that any mistake labeled a judgment call is immune. The better view is that the ordinary standard already accounts for judgment: the question is whether the process of deciding, including the history taken, the tests considered and the risks weighed, met the standard, not whether the outcome was good. Guidelines, protocols and the codification of care Over the past four decades, medicine has produced an enormous volume of clinical practice guidelines: recommendations by specialty societies, government agencies and health systems about how to diagnose and treat particular conditions. Guidelines seem to offer an answer to the vagueness of the professional standard. If the American College of Cardiology recommends a particular approach to chest pain, why not treat adherence as compliance with the standard of care and deviation as negligence? The law has resisted making guidelines conclusive, for good reasons. Guidelines vary in quality and in the strength of the evidence behind them. Different bodies issue conflicting guidelines on the same question; breast cancer screening intervals are a well-known example. Guidelines are written for typical patients, and good medicine often requires departing from them for an atypical one. And guidelines may be shaped by the interests of the organizations that write them. Courts therefore generally treat guidelines as evidence of the standard of care, admissible through expert testimony or, in many jurisdictions, as learned treatises, but not as the standard itself. Legislatures have occasionally tried to give guidelines more force. Maine's Medical Liability Demonstration Project, begun in 1990, allowed physicians in certain specialties who followed state-approved guidelines to use that compliance as an affirmative defense. Evaluations found that the defense was almost never invoked, and the project was allowed to lapse. Subsequent federal proposals for "safe harbor" protection for guideline adherence have not been enacted. The difficulty is structural: a safe harbor that protects guideline compliance only helps if guidelines are precise enough to be applied to the facts of a case, and precise guidelines are exactly the ones most likely to be inappropriate for some patients. Guidelines matter more in practice than in doctrine. Plaintiffs use them to show that a physician departed from recognized practice; defendants use them to show compliance. Empirical studies of litigated cases have found that guidelines are more often used against physicians than in their defense, which gives physicians an incentive to follow them closely, sometimes more closely than good medicine would warrant. Hospital protocols raise a similar issue. A hospital's internal policy, for example requiring a physician to be notified whenever a fetal heart tracing shows certain patterns, is generally admissible as evidence of the standard of care, but violation of the policy is not negligence per se, and compliance is not a complete defense. Emergencies, resources and the standard under stress The standard of care is always applied in context, and context includes emergencies. Good Samaritan statutes in every state protect clinicians who volunteer aid outside their ordinary duties, typically by removing liability for ordinary negligence. Several states go further and raise the bar for liability in emergency care within hospitals. Texas, as part of its 2003 reforms, requires proof of "willful and wanton negligence" for claims arising from emergency medical care in a hospital emergency department or obstetrical unit, a standard far more demanding than ordinary negligence. Georgia adopted a gross-negligence standard with a clear-and-convincing evidence requirement for emergency department care in 2005. The COVID-19 pandemic brought the question of standards under stress to national attention. Many states enacted statutes or issued executive orders granting health care providers immunity from ordinary negligence claims arising from pandemic care, and the federal Public Readiness and Emergency Preparedness Act provided broad immunity for the use of covered countermeasures such as vaccines, tests and certain treatments. Courts have since had to decide how far these immunities extend, particularly to nursing home deaths and to care that was delayed because of the pandemic. The underlying principle is not new: the law has always recognized that what is reasonable depends on circumstances, and crisis standards of care, developed by public health authorities to guide the allocation of scarce resources, reflect that principle in formal terms. Resource constraints raise a quieter and more persistent question. Should the standard of care vary with what a patient's insurance covers or what a hospital can afford? Courts have been reluctant to say so explicitly. The classic cautionary case is Wickline v. State of California (Cal. Ct. App. 1986), in which a Medi-Cal patient was discharged earlier than her physician wanted after the state program authorized only a shorter stay; the court held that the physician, not the payer, remained responsible for the medical decision to discharge, and warned that cost-containment programs could be liable if their design corrupted medical judgment. The general rule remains that physicians must advocate for the care their patient needs, and that a payer's refusal does not excuse a clinician's failure to exercise independent judgment. Technology and the moving standard The standard of care moves as medicine moves. A practice that was optional a decade ago may be expected today, and courts evaluate conduct by the standard at the time of treatment, not the time of trial. Experts must be careful to testify about what was expected when the events occurred, and defendants often argue, sometimes persuasively, that plaintiffs' experts are applying hindsight. The most significant current example is artificial intelligence. Diagnostic algorithms now read retinal images, flag suspicious findings on mammograms and chest scans, and predict clinical deterioration from electronic health record data. They raise two opposite liability questions. First, if a physician follows an algorithm's erroneous recommendation, is the physician negligent for relying on it? Second, if a physician ignores a correct algorithmic warning, or fails to use a tool that has become widely adopted, is that negligence? Because the standard of care is tied to customary practice, the law will likely treat AI tools as it has treated earlier diagnostic technologies: at first their use is optional and a physician is judged on independent judgment; as they become standard, failing to use them, or unreasonably overriding them, may itself fall below the standard. Scholars including W. Nicholson Price and Michelle Mello have observed that this dynamic may push physicians toward following algorithmic outputs even when their own judgment differs, since deviation from a recommendation leaves a record that a plaintiff can exploit, while following it offers the cover of common practice. The deeper lesson of this chapter is that the standard of care is not a fixed rule but a contested, evolving account of good practice that must be reconstructed in each case. The instrument for that reconstruction is expert testimony, which Chapter 4 examines. Before turning to it, there is a second, distinct duty that physicians owe their patients: the duty to obtain their informed consent. Chapter 3: Informed Consent A surgeon may perform an operation flawlessly and still be liable for the harm it causes. If the patient did not agree to the procedure, or agreed without being told of a risk that then materialized, the law may treat the injury as a wrong regardless of technical skill. This is the domain of informed consent, a branch of malpractice law that asks a different question from the standard of care. The standard of care asks whether the physician treated the patient competently. Informed consent asks whether the physician respected the patient's right to decide. The doctrine is conceptually important out of proportion to its role in litigation. Informed consent claims are rarely the heart of a malpractice case; they are usually pleaded alongside negligence claims and rarely succeed on their own. But the doctrine expresses the law's view of what the relationship between physician and patient is, and its evolution, from battery to negligence and from physician-centered to patient-centered disclosure, tracks a broader transformation in medical ethics. From battery to negligence The starting point is bodily autonomy. In Schloendorff v. Society of New York Hospital (1914), Judge Benjamin Cardozo, then on the New York Court of Appeals, wrote that every human being of adult years and sound mind has a right to determine what shall be done with his own body, and that a surgeon who operates without the patient's consent commits an assault. The case itself was about hospital liability, and the statement was not strictly necessary to the result, but it became the foundation of the modern law. An operation performed without any consent, or one that goes beyond the consent given, remains a battery. The classic examples involve a surgeon who, with the patient under anesthesia, operates on a different part of the body than the one agreed to, or performs an additional procedure the patient never authorized. Battery does not require proof that the physician deviated from professional standards or even that the patient was worse off; the unauthorized touching is itself the wrong, although damages may be modest if no real harm followed. Courts recognize an emergency exception where the patient cannot consent and delay would threaten life or health, and a limited extension doctrine allowing a surgeon to address an unexpected condition discovered during an authorized operation when it would be unreasonable to stop and ask. The harder cases involve patients who did consent, but on incomplete information. A patient agrees to a spinal procedure without being told of a small risk of paralysis; the risk materializes. The patient consented to the touching, so battery fits poorly. By the late 1950s and 1960s, courts began to treat the failure to disclose risks as a form of negligence. The phrase "informed consent" itself entered American law through Salgo v. Leland Stanford Jr. University Board of Trustees, a California appellate decision in 1957 involving a patient paralyzed after an aortography procedure whose risks he said had not been explained. By framing inadequate disclosure as negligence rather than battery, courts brought it within the ordinary structure of malpractice: duty, breach, causation and damages. Whose standard governs disclosure? Once disclosure became a matter of negligence, the obvious question was what standard measured it. Two answers emerged, and American states remain divided between them. The first, the professional or physician-based standard, applies the same logic as the standard of care: a physician must disclose what a reasonable physician in the same field would disclose in similar circumstances. The Kansas Supreme Court adopted this approach in Natanson v. Kline (1960), a case involving radiation therapy after a mastectomy that caused severe burns. Under the professional standard, the plaintiff needs an expert to testify about what physicians customarily disclose, and the physician's own profession decides how much patients should be told. The second, the patient-based or material-risk standard, was announced in Canterbury v. Spence, decided by the United States Court of Appeals for the District of Columbia Circuit in 1972. Jerry Canterbury, a nineteen-year-old clerk, underwent a laminectomy for back pain. He was not told of a roughly one percent risk of paralysis. After the operation he fell from his hospital bed and was left partially paralyzed. In an opinion by Judge Spottswood Robinson, the court held that the scope of the duty to disclose is measured by the patient's need for information, not by medical custom. A risk must be disclosed if a reasonable person in the patient's position would likely attach significance to it in deciding whether to undergo the treatment. The court reasoned that respect for the patient's right of self-determination demands a standard set by law rather than by physicians, and that there was no reason to think professional custom on disclosure reflected any special medical expertise. The same year, the California Supreme Court reached a similar conclusion in Cobbs v. Grant. The material-risk standard asks what the patient needs to know, which usually includes the nature of the proposed procedure, its material risks, its expected benefits, the reasonable alternatives including doing nothing, and the risks of those alternatives. Materiality depends on both the probability and the severity of the harm: a one-in-ten-thousand risk of death may be material, while a one-in-ten risk of temporary soreness may not be. The comparison is set out in Table 2, which also includes the subjective standard that some commentators have advocated and a few courts have gestured toward, and which governs the causation element in a small minority of jurisdictions. Table 2. Standards for the duty to disclose risks. Standard Question asked Leading authority Need for expert on disclosure Professional (physician-based) What would a reasonable physician disclose? Natanson v. Kline (Kan. 1960) Usually required Material risk (reasonable patient) What would a reasonable patient want to know? Canterbury v. Spence (D.C. Cir. 1972); Cobbs v. Grant (Cal. 1972) Needed for risks and alternatives, not for materiality Subjective (particular patient) What would this patient have wanted to know? Advocated in commentary; limited judicial support Not required for materiality Roughly half the states use each of the first two standards, and several have fixed the rule by statute. The split is less stark in practice than in theory, because experts are needed under either standard to establish what the risks and alternatives actually were, and physicians' customary disclosures have moved toward patient-centered norms under the influence of medical ethics and hospital accreditation standards. The comparative view: Britain's reversal The United Kingdom offers an instructive comparison. For decades English law applied the Bolam test, derived from Bolam v. Friern Hospital Management Committee (1957), which judged a doctor's conduct, including disclosure, by whether it accorded with a practice accepted as proper by a responsible body of medical opinion. The House of Lords applied that test to disclosure in Sidaway v. Board of Governors of the Bethlem Royal Hospital (1985), a case whose facts closely resembled Canterbury. Thirty years later, in Montgomery v. Lanarkshire Health Board (2015), the UK Supreme Court abandoned Sidaway. Nadine Montgomery, a diabetic woman of small stature, was not told of the risk of shoulder dystocia in vaginal delivery of a large baby, because her obstetrician believed that if told, women would opt for cesarean section, which she considered not in their interest. The baby suffered severe injuries. The court held that a doctor must take reasonable care to ensure the patient is aware of any material risks and of reasonable alternatives, with materiality measured by what a reasonable person in the patient's position, or this particular patient to the doctor's knowledge, would consider significant. Australia had reached a similar position earlier in Rogers v. Whitaker (1992). Montgomery is now the leading modern statement of the patient-centered approach in the common-law world and is frequently cited in American commentary. Causation: would the patient have refused? Proving that a risk was not disclosed is only half the claim. The patient must also prove that the nondisclosure caused the injury, which means showing that proper disclosure would have led to a different decision. If the patient would have gone ahead anyway, the failure to disclose did not cause the harm. Here the courts face a real difficulty. A patient who has suffered a devastating complication will almost always testify, sincerely, that had she known of the risk she would have declined. That testimony is unavoidably colored by hindsight. Canterbury resolved the problem by adopting an objective test: the question is whether a reasonable person in the patient's position would have declined the treatment if adequately informed. Most American jurisdictions follow the objective approach. A minority, including Oklahoma after Scott v. Bradford (1979), apply a subjective test asking what this particular patient would have done, accepting the risk of self-serving testimony as the price of respecting individual choice. The objective causation test has a practical consequence. If a treatment is clearly beneficial and the undisclosed risk is small, the plaintiff will struggle to show that a reasonable person would have refused, even if the particular plaintiff genuinely would have. This is one reason informed consent claims so rarely succeed on their own. They are strongest where there were real alternatives, including declining treatment, and the undisclosed risk was serious enough to tip a reasonable decision. Frontiers of disclosure Beyond the standard list of risks, benefits and alternatives, courts have confronted a series of harder questions about what physicians must disclose. The first concerns informed refusal. In Truman v. Thomas (1980), the California Supreme Court held that a physician who recommended a Pap smear to a patient who repeatedly declined it had a duty to explain the risks of refusing. She later died of cervical cancer that screening would likely have caught. The principle, that patients must be told the consequences of saying no as well as the consequences of saying yes, has obvious implications for preventive care and is closely related to the documentation practices discussed in Chapter 8. The second concerns the physician's personal characteristics and interests. In Moore v. Regents of the University of California (1990), the California Supreme Court held that a physician must disclose personal interests, whether research or economic, that may affect medical judgment; the defendant had used the patient's cells to develop a commercially valuable cell line without telling him. Courts have been more divided on whether physicians must disclose their own experience or success rates. In Johnson v. Kokemoor (Wis. 1996), the Wisconsin Supreme Court allowed evidence that a neurosurgeon had exaggerated his experience with a difficult aneurysm operation and had failed to disclose that more experienced surgeons elsewhere had markedly better outcomes. Other courts have been reluctant to require such disclosures, fearing an unmanageable expansion of the duty. Most courts have also declined to require disclosure of a physician's own health conditions, such as HIV infection or substance use, absent a showing of material risk to the patient, though the results are fact-dependent. The third concerns statistical information. Arato v. Avedon (Cal. 1993) involved a patient with pancreatic cancer who was not told the grim statistical life expectancy associated with his disease; his family argued that he would have put his business and financial affairs in order had he known. The California Supreme Court declined to require disclosure of specific statistics as a matter of law, holding that the question was whether the physician disclosed information material to the treatment decision, and leaving it to the jury. The fourth concerns the form of consent. The signed consent form is ubiquitous but legally secondary. A form is evidence that disclosure occurred, sometimes strong evidence, but it does not substitute for the conversation, and courts have repeatedly held that a generic form listing every conceivable risk in boilerplate does not demonstrate that the patient was informed. Conversely, a well-documented discussion recorded in the chart may carry more weight than any form. Shared decision-making tools and decision aids, which present options and their outcomes in plain terms, have been endorsed in medical literature as improving patient understanding, and their use may in time become part of the standard for significant elective procedures. Why informed consent matters beyond litigation Informed consent litigation is a small part of malpractice practice, but the doctrine's significance lies elsewhere. It is the one area where the law has deliberately refused to let the medical profession define its own duty, on the ground that the question of what a patient should be told is not a technical medical judgment but a question about respect for persons. It sits in tension with the deference to custom that governs the standard of care, and the tension is productive: it reminds us that the professional standard is a policy choice, and that where the reasons for deference run out, the law can and does set its own standard. It also connects to the empirical themes of this book. Studies of why patients sue have repeatedly found that poor communication, a sense of having been kept in the dark or treated dismissively, is a strong predictor of litigation, often stronger than the severity of the error itself. Informed consent law addresses communication before treatment. As Chapter 9 explains, some of the most promising reforms address communication after an adverse outcome. Both reflect the same insight: much of what drives malpractice litigation is not only what physicians do, but what they say, and fail to say, to the people they treat. Chapter 4: The Expert Witness Because the standard of care is defined by professional practice, and because causation in medicine is a technical question, nearly every malpractice case is decided by a contest of experts. The plaintiff needs at least one qualified physician willing to testify that the defendant's care fell below the standard and that the departure caused the injury. The defendant, typically, produces one or more physicians who say the opposite. The jury, which cannot independently evaluate the medicine, must decide whom to believe. This dependence on experts is the single most important practical feature of malpractice litigation. It drives the cost of cases, and therefore which cases lawyers will take. It shapes the pre-suit screening mechanisms that legislatures have enacted. It raises hard evidentiary questions about which experts may testify and on what basis. And it has spawned a market in professional testimony that critics on both sides regard as corrosive. Why experts are required, and when they are not In most jurisdictions, a malpractice plaintiff who does not present expert testimony on the standard of care and causation will lose on summary judgment or directed verdict. The rule follows from the professional standard: if the question is what a reasonable physician would have done, only someone who knows what physicians do can answer it. There are two exceptions. The first is the common-knowledge exception. Some errors are so obvious that laypeople can recognize them without help: amputating the wrong leg, leaving a surgical sponge inside a patient, failing to respond at all to a patient's calls for help. In such cases courts allow the jury to infer negligence from ordinary experience. The exception is applied narrowly, and courts regularly reject attempts to invoke it where the question involves any clinical judgment. The second is the doctrine of res ipsa loquitur, "the thing speaks for itself." Where an injury is of a kind that does not ordinarily occur without negligence, and the instrumentality was under the defendant's control, the jury may infer negligence without direct proof of what went wrong. The most famous medical application is Ybarra v. Spangard, decided by the California Supreme Court in 1944. Joseph Ybarra went into surgery for an appendectomy and woke with a paralyzed shoulder, an injury to a part of his body far from the operating field. He could not say who among the surgeon, anesthesiologist, nurses and hospital staff had caused it, because he had been unconscious. The court allowed the case to proceed against all of them, reasoning that those in control of an unconscious patient should bear the burden of explaining what happened. Retained surgical items, the paradigm case for res ipsa today, remain a persistent source of claims despite counting protocols and radio-frequency tagging. Outside these exceptions, the plaintiff needs an expert, and so does the defendant as a practical matter, since a jury left with only the plaintiff's expert is likely to believe him. Who may testify Because expert testimony is so decisive, the question of who qualifies has become heavily regulated. The traditional rule under evidence law was generous: any witness with knowledge, skill, experience, training or education sufficient to assist the jury could testify, and weaknesses in qualifications went to the weight of the testimony rather than its admissibility. Under that rule, a general surgeon could often testify against an orthopedic surgeon, or an internist against an emergency physician, if the matter was within the witness's knowledge. Many states have tightened the rule by statute, typically as part of tort reform packages. The most common requirement is that the expert practice in the same or a similar specialty as the defendant. Michigan's statute requires, among other things, that if the defendant is a board-certified specialist, the expert must be board certified in that same specialty; the Michigan Supreme Court in Woodard v. Custer (2006) interpreted the rule to require matching on the specialty relevant to the care at issue. Arizona enacted a similar requirement, which its supreme court upheld in Seisinger v. Siebel (2009) against a challenge that it infringed the court's rulemaking authority. Many states also require that the expert have been in active clinical practice or teaching during the period before the events in question; Kansas, for instance, requires that the expert have devoted at least half of his or her professional time to actual clinical practice in the relevant field in the two years before the incident. These rules have a clear rationale. A defendant should be judged by someone who does what the defendant does, and full-time professional witnesses who no longer practice have little claim to know current practice. But they also have clear costs. In a narrow specialty, the pool of qualified experts may be small, and few will be willing to testify against colleagues. A plaintiff injured by a rare procedure may simply be unable to find a matching expert. And the requirements generate satellite litigation over whether a particular expert's qualifications match, which delays cases and can end meritorious claims on technical grounds. Pre-suit screening: certificates and affidavits of merit The second major regulatory device is the pre-suit certificate or affidavit of merit. More than half the states require some form of early expert support before a malpractice case may proceed. The details vary considerably. Pennsylvania requires the plaintiff's attorney to file, with or shortly after the complaint, a certificate stating that an appropriate licensed professional has supplied a written statement that there is a reasonable probability that the care fell outside acceptable professional standards and caused the harm. New Jersey requires an affidavit of merit from an appropriately licensed person within a set period after the defendant answers. The New Jersey Supreme Court, recognizing that the requirement was ensnaring meritorious claims on technicalities, directed in Ferreira v. Rancocas Orthopedic Associates (2003) that trial courts hold early case management conferences to address compliance before the deadline passes. Texas requires one of the most demanding submissions: an expert report, served within 120 days of each defendant's answer, that provides a fair summary of the expert's opinions on the applicable standard of care, the manner in which the care failed to meet it, and the causal relationship between the failure and the injury. A report that is missing or inadequate results in dismissal with prejudice and an award of attorney's fees to the defendant. The Texas statute has generated an enormous body of appellate decisions on the adequacy of reports. The theory behind these requirements is that they filter out cases that have no expert support before defendants incur the cost of defending them. The evidence on their effects is limited but suggestive. They appear to reduce the number of claims filed, which is their purpose. Whether they filter out weak claims specifically, rather than claims that are simply expensive to prepare, is less clear, since the burden of obtaining an early expert opinion falls hardest on plaintiffs with modest damages. Several state courts have struck down such requirements as violating the right of access to courts or the separation of powers. The Oklahoma Supreme Court, for example, invalidated affidavit-of-merit requirements more than once, including in Zeier v. Zimmer, Inc. (2006), holding that they imposed an unconstitutional monetary barrier on access to the courts; the Washington Supreme Court struck down that state's certificate-of-merit requirement in Putman v. Wenatchee Valley Medical Center (2009) on separation-of-powers and access grounds. A long-running question was whether these state requirements apply when a malpractice case is litigated in federal court, as happens when the parties are from different states or the claim is brought against the United States under the Federal Tort Claims Act. Under the Erie doctrine, federal courts in such cases apply state substantive law but federal procedural rules. The federal circuits divided. The Third Circuit had held that Pennsylvania's certificate-of-merit rule is substantive and applies in federal court. Others, including the Sixth Circuit in Gallivan v. United States (2019) with respect to Ohio's affidavit requirement and the Fourth Circuit in Pledger v. Lynch (2021) with respect to West Virginia's pre-suit certification requirement, held that such rules conflict with the Federal Rules of Civil Procedure, which set out what a complaint must contain and do not require supporting evidence at the pleading stage. The Supreme Court took up the question in Berk v. Choy, a case arising from Delaware's affidavit-of-merit statute, and in early 2026 held that the Delaware requirement does not apply in federal court because the Federal Rules already answer the question of what a plaintiff must file to begin a case. The decision means that in the federal forum, the screening function these statutes serve must be performed, if at all, by ordinary pleading standards, summary judgment and the rules governing expert disclosure. Its practical effect will depend on how many malpractice cases can reach federal court, which is a minority, but it removes a significant filter for those that do. Gatekeeping: Frye, Daubert and Rule 702 Qualification is one question; the reliability of the expert's opinion is another. Two tests govern admissibility of expert testimony in American courts. The older test comes from Frye v. United States (D.C. Cir. 1923), which required that the scientific principle on which an expert relies be generally accepted in the relevant scientific community. The newer comes from Daubert v. Merrell Dow Pharmaceuticals, Inc. (1993), in which the Supreme Court held that the Federal Rules of Evidence superseded Frye in federal courts and made trial judges gatekeepers responsible for ensuring that expert testimony is both relevant and reliable. Daubert identified factors a court might consider, including whether a theory has been tested, whether it has been subjected to peer review, its known error rate and its general acceptance. General Electric Co. v. Joiner (1997) held that appellate courts review these rulings for abuse of discretion and that courts may exclude opinions connected to data only by the expert's say-so. Kumho Tire Co. v. Carmichael (1999) extended the gatekeeping obligation to all expert testimony, including testimony based on technical or experience-based knowledge rather than science. Federal Rule of Evidence 702 was amended in 2000 to codify these decisions and amended again effective December 1, 2023. The 2023 amendment clarified that the proponent of expert testimony must demonstrate to the court that it is more likely than not that the testimony meets each of the rule's requirements, and that each opinion reflects a reliable application of the expert's methods to the facts. The Advisory Committee explained that many courts had been wrongly treating questions about the sufficiency of an expert's basis and the application of methodology as matters of weight for the jury rather than admissibility. Most states have adopted Daubert or a similar standard; a significant minority, including New York, Pennsylvania, Illinois and Washington, retain versions of Frye, and several apply hybrid approaches. In malpractice cases, gatekeeping bites differently than in toxic tort or product cases, where it originated. Most standard-of-care testimony is experience-based rather than scientific: an experienced obstetrician testifies about what obstetricians do. Courts have generally held that such testimony is reliable if the expert has the relevant experience and explains how it supports the opinion, though the requirement that the expert identify the standard with specificity, rather than simply asserting that the defendant was negligent, is enforced with increasing rigor. Gatekeeping matters most on causation, where experts may offer opinions about the natural history of disease, the probable effect of earlier treatment or the mechanism of a birth injury that rest on scientific literature and can be tested against it. Challenges to causation opinions, for example testimony that a delay of a few hours in delivery caused a child's cerebral palsy, are among the most consequential evidentiary fights in malpractice litigation. The market for testimony Expert testimony in malpractice cases is a market, and its incentives are uncomfortable. Experts are paid by the side that retains them, often at substantial hourly rates for record review, deposition and trial. A small number of physicians testify frequently, some for plaintiffs, some for defendants, and a few for both. Experts who are good at testifying, meaning credible, clear and unshakable on cross-examination, are in demand whatever the merits of the cases they testify in. The profession has responded in two ways. The first is professional discipline. Specialty societies have adopted guidelines for expert witness conduct and disciplined members who give testimony considered irresponsible. The leading case is Austin v. American Association of Neurological Surgeons (7th Cir. 2001). Dr. Donald Austin, a neurosurgeon, testified for a plaintiff that a surgeon had negligently injured a nerve and offered a theory of causation that his professional association considered unsupportable. The association suspended him. He sued, and the Seventh Circuit, in an opinion by Judge Richard Posner, upheld the suspension, reasoning that a professional association has a legitimate interest in the quality of its members' testimony and that such discipline serves the public by deterring irresponsible expert evidence. Several state medical boards have also taken the position that expert testimony constitutes the practice of medicine and is subject to board discipline. The difficulty is obvious. Disciplinary processes run by physicians may chill testimony against physicians. Society guidelines typically apply to testimony on either side, but critics have observed that complaints are far more often brought against experts who testify for plaintiffs. Where only a few specialists in a field are willing to testify against a colleague, the threat of professional sanctions can make meritorious claims harder to prove, reviving in modern form the old conspiracy of silence that the national standard was supposed to end. The second response is proposals for neutral experts. Federal Rule of Evidence 706 and state analogues allow judges to appoint their own experts, but courts use the power rarely, partly because of cost and partly because of the adversarial tradition. Some jurisdictions have experimented with medical review panels, discussed further in Chapter 7, and some reform proposals would replace adversarial expert testimony with specialized adjudicators who have medical expertise, the health courts discussed in Chapter 9. What the reliance on experts means The expert is where the book's first theme, the standard of care, and its larger argument meet. A standard defined by professional practice must be reconstructed in each case by paid witnesses who disagree. That makes outcomes uncertain, because they depend on which expert a jury finds more persuasive, and makes cases expensive, because experts are costly and both sides need them. The expense filters out claims with modest damages regardless of their merit, which helps explain why so many negligently injured patients never sue. The uncertainty contributes to the imperfect correlation between claims and actual negligence documented in Chapter 1, and to the physician's sense that liability is a lottery. Those perceptions, as Chapter 8 argues, are what drive defensive medicine. Reforms aimed at experts, such as qualification requirements and certificates of merit, reduce the number of claims but do not obviously make the ones that remain more accurate, and they may make meritorious claims harder to bring. Once the standard of care has been established and breached, the plaintiff still has to prove that the breach made a difference. That is the subject of the next chapter, and it is where many cases with clear errors are lost. Chapter 5: Causation and the Problem of the Already-Sick Patient In most personal injury cases, causation is the easy part. A pedestrian struck by a car was healthy before and injured after; the car caused the injury. Medicine is different in a basic way. Patients come to physicians because they are sick, injured or at risk. When a patient with cancer dies after a delayed diagnosis, the cancer is a cause of death. The question is whether the delay was also a cause, and that question requires comparing what happened with a hypothetical world in which the physician acted properly. No one can observe that world. It must be reconstructed from medical knowledge, statistics and expert opinion. Causation is where many malpractice claims founder, including claims in which the breach of the standard of care is clear. It is also where the traditional structure of tort law fits medicine least comfortably, and where courts have been most willing to innovate, and most divided about whether to. The traditional rule: more likely than not The basic test of causation in American tort law is but-for causation: the plaintiff must show that, but for the defendant's negligence, the injury would not have occurred. The plaintiff must prove this by a preponderance of the evidence, meaning more likely than not. Where several causes combine, most courts also accept a substantial-factor formulation, under which a defendant's conduct is a cause if it was a substantial factor in bringing about the harm, even though other factors contributed. In malpractice cases, experts are typically asked whether their opinions are held to a "reasonable degree of medical certainty" or "reasonable medical probability." The phrase has no fixed scientific meaning, and commentators and some courts have criticized it as confusing, but in most jurisdictions it is understood to mean simply that the expert believes the proposition is more likely than not. Applied to the already-sick patient, the more-likely-than-not rule produces sharp results. Consider a patient whose lung cancer was visible but missed on a chest X-ray. By the time it was diagnosed a year later, it had spread. Suppose the evidence shows that had it been diagnosed promptly, the patient would have had a 60 percent chance of long-term survival, and that after the delay the chance was 10 percent. The patient died. Under the traditional rule, the plaintiff wins: more likely than not, prompt diagnosis would have saved the patient, so the negligence caused the death. The estate recovers full wrongful death damages. Now suppose the chance of survival with prompt diagnosis was 40 percent, falling to 10 percent after the delay. The negligence cut the patient's chance of survival by three-quarters. But more likely than not, the patient would have died anyway, so the plaintiff cannot prove the negligence caused the death. The estate recovers nothing. The rule's all-or-nothing character is obvious. A defendant whose negligence reduced a patient's chance of survival from 51 percent to zero pays full damages; a defendant whose negligence reduced it from 49 percent to zero pays nothing. And for patients with serious diseases whose baseline chances were below even, which describes a large share of cancer patients, the rule means that no amount of negligence in diagnosis or treatment can give rise to liability for death. Physicians who treat the sickest patients are, in effect, immune from liability for their errors in treating them. The loss-of-chance doctrine Beginning in the 1980s, a number of state courts responded by recognizing a claim for the loss of a chance of a better outcome. The leading early case is Herskovits v. Group Health Cooperative of Puget Sound, decided by the Washington Supreme Court in 1983. Leslie Herskovits had complained of chest pain and coughing; his physicians did not pursue the symptoms, and his lung cancer was diagnosed about six months later. Expert evidence suggested that the delay had reduced his chance of five-year survival from 39 percent to 25 percent. He died. The trial court dismissed the claim because he had never had a better-than-even chance of survival. The Washington Supreme Court reversed, allowing the estate to proceed. The plurality opinion held that the reduction in the chance of survival was itself a compensable injury, with damages measured by the proportion of the chance that was lost. The theoretical underpinning came from Joseph King, whose 1981 article in the Yale Law Journal, "Causation, Valuation, and Chance in Personal Injury Torts Involving Preexisting Conditions and Future Consequences," argued that a chance of survival is a thing of value, that its destruction is an injury in its own right, and that damages should be calculated proportionally. On this view, a patient whose chance of survival was reduced from 40 percent to 10 percent has lost 30 percentage points of chance, and the defendant should pay 30 percent of the full value of the death. The most thorough modern statement of the doctrine is Matsuyama v. Birnbaum, decided by the Massachusetts Supreme Judicial Court in 2008. Kimiyoshi Matsuyama had for years complained to his primary care physician of gastric symptoms; the physician did not pursue the possibility of gastric cancer despite risk factors, including the patient's Japanese ancestry, which is associated with higher incidence. By the time the cancer was diagnosed, it was incurable. The jury found that the negligence had reduced his chance of survival and awarded damages. The Supreme Judicial Court affirmed, holding that loss of chance is a recognized injury in Massachusetts malpractice law, and endorsed the proportional method of calculating damages: the jury first determines the full value of the wrongful death, then multiplies it by the percentage of chance lost. The court also held that the plaintiff must prove by a preponderance of the evidence that the defendant's negligence caused the loss of chance, preserving the ordinary burden of proof for the new injury. Courts divided The loss-of-chance doctrine has divided American courts roughly in half, and the division reflects real disagreement about the purposes of tort law. The main positions are summarized in Table 3. Table 3. Approaches to causation where negligence reduced a patient's chance of a better outcome. Approach Rule Illustrative authority Result where baseline chance was below 50% Traditional all-or-nothing Must prove negligence more likely than not caused the final harm Kramer v. Lewisville Memorial Hospital (Tex. 1993); Fennell v. Southern Maryland Hospital Center (Md. 1990) No recovery Relaxed causation Evidence of a substantial reduction in chance may go to jury on causation of the full harm Hamil v. Bashline (Pa. 1978) Full damages possible if jury finds causation Lost chance as a separate injury, proportional damages Reduction in chance is itself the injury; damages scaled to percentage lost Herskovits (Wash. 1983); Matsuyama v. Birnbaum (Mass. 2008); Lord v. Lovett (N.H. 2001) Proportional recovery Statutory rejection Legislature bars recovery for loss of an opportunity of 50% or less Michigan statute enacted after Falcon v. Memorial Hospital (Mich. 1990) No recovery The proportional approach has attracted the widest scholarly support. Courts adopting it argue that it serves both of tort law's core goals. It compensates patients for a real loss, since a chance of survival is something patients value and pay for, and it deters negligence in treating seriously ill patients, who would otherwise be unprotected. It is also more accurate than the traditional rule: over many cases, a defendant pays for the harm its negligence statistically caused, rather than paying all or nothing depending on which side of 50 percent a case happens to fall. The Minnesota Supreme Court adopted the doctrine in Dickhoff v. Green (2013), a case involving a child whose rare cancer was not diagnosed at an early examination, emphasizing that denying recovery would leave a real injury uncompensated. Courts rejecting the doctrine offer several counterarguments. The Texas Supreme Court in Kramer v. Lewisville Memorial Hospital (1993) reasoned that loss-of-chance claims dilute the causation requirement and expose physicians to liability for harms they probably did not cause. Critics also argue that statistical survival figures are too imprecise to support proportional damages, that the doctrine increases malpractice costs, and that its logic is hard to confine. If a patient who died can sue for a lost chance of survival, can a patient who is still alive sue for a reduced chance, before any bad outcome has occurred? Most courts that recognize the doctrine have limited it to cases where the ultimate harm has materialized, though some have been more expansive. And is the doctrine limited to death, or does it extend to lost chances of avoiding disability? Courts have given varying answers. Legislatures have also intervened. After the Michigan Supreme Court recognized a lost-opportunity claim in Falcon v. Memorial Hospital (1990), involving a young woman who died of an amniotic fluid embolism during childbirth, the legislature amended the statute to provide that a plaintiff cannot recover for loss of an opportunity to survive or achieve a better result unless the opportunity was greater than 50 percent. Michigan courts have since struggled with how to apply the statutory language, which is notoriously opaque. The effect in Michigan and similar states is to restore the traditional all-or-nothing rule. Delayed diagnosis and the arithmetic of harm Loss of chance is most often litigated in delayed-diagnosis cases, which are among the most common and costly malpractice claims. As noted in Chapter 1, diagnostic error is the leading allegation in paid claims nationally. Cancer, heart attack, stroke, infection and pulmonary embolism are the conditions most frequently involved. These cases turn on detailed causation evidence. An expert must establish when the disease would have been detectable, what stage it would have been at, what treatment would then have been available, what the outcome of that treatment would likely have been, and how that compares with what actually happened. For cancer, experts often rely on tumor growth rates, estimating backward from the size of a tumor at diagnosis to its likely size and stage at the time of the missed opportunity. These calculations involve assumptions that are contestable. Tumors do not all grow at the same rate, and staging at an earlier date can only be inferred. Survival statistics come from population studies whose subjects may differ from the patient in age, health and other factors. Defense experts in such cases often argue that the disease was already incurable at the time of the alleged error, or that the delay did not materially change the treatment or outcome. Plaintiffs' experts argue that the window of curability was open when the error occurred and closed during the delay. Because the underlying science is uncertain, both sides can usually find qualified experts to support their positions, and the case turns on which is more persuasive. This is one of the reasons the gatekeeping disputes discussed in Chapter 4 are so important in causation: a court that excludes a plaintiff's causation expert as speculative effectively ends the case. Stroke cases illustrate how causation arguments evolve with medicine. When clot-dissolving drugs were first approved for acute ischemic stroke, they had to be given within a few hours of symptom onset, and their benefit, while real, was modest in absolute terms: many patients recovered without them, and many who received them did not recover. Plaintiffs claiming that delayed treatment caused a stroke-related disability had to show that timely treatment more likely than not would have produced a better outcome, which the trial data often did not support for any individual patient. Later developments, including mechanical thrombectomy for large vessel occlusions, which in some trials produced much larger benefits, changed the arithmetic. A failure to recognize a large vessel stroke and transfer a patient for thrombectomy may now be easier to connect causally to a worse outcome than a failure to give clot-dissolving drugs was two decades ago. Birth injuries: causation at its hardest Birth injury cases, particularly those alleging that negligent management of labor caused cerebral palsy or other neurological injury, are among the highest-value malpractice claims and the most difficult on causation. The damages can be enormous, because a child with severe brain damage may need lifelong care. The causation questions are extraordinarily complex. Plaintiffs typically allege that the physician or nurses failed to recognize signs of fetal distress on the electronic fetal heart monitor and delayed a cesarean delivery, allowing the baby to suffer oxygen deprivation. Defendants typically argue that the injury occurred before labor began, or resulted from infection, genetic factors or other causes unrelated to labor management. Medical research over the past several decades has established that only a minority of cerebral palsy cases are attributable to events during labor and delivery, and professional bodies, notably the American College of Obstetricians and Gynecologists, have published criteria for determining when an intrapartum event is likely to have caused neonatal brain injury. Those criteria, which consider factors such as the infant's blood acidity at birth, the timing and pattern of brain injury on imaging, and the presence of multi-organ dysfunction, are frequently at the center of expert disputes. Plaintiffs argue they are drawn too narrowly; defendants argue they reflect the best available science. The mismatch between the frequency of claims and the underlying science has been cited as evidence of the system's inaccuracy. Electronic fetal monitoring, introduced widely in the 1970s in the expectation that it would reduce cerebral palsy, has not done so, while it has contributed to rising cesarean rates. Many obstetricians believe the frequency of cesarean delivery is in part defensive: a physician who performs a cesarean at the first sign of an ambiguous tracing is rarely sued for doing so, while one who waits may be sued if the baby is injured, whatever the true cause. That dynamic is one of the clearest examples of the connection between causation uncertainty and defensive medicine that Chapter 8 examines. Two states have removed the most severe birth injury cases from the tort system altogether. Virginia's Birth-Related Neurological Injury Compensation Program, established in 1987, and Florida's Birth-Related Neurological Injury Compensation Association, established in 1988, provide no-fault compensation for a defined category of catastrophic birth injuries, funded by assessments on participating physicians and hospitals. Claims within the programs' scope are handled administratively and are excluded from ordinary malpractice suits. Chapter 9 returns to these programs as experiments in alternatives to litigation. What causation reveals Causation disputes expose a basic tension in malpractice law. Tort law was built for cases in which one person's act caused another's injury in a way that can be traced. Medicine involves probabilities: treatments that improve the odds rather than guarantee outcomes, diagnoses that are more or less likely, diseases whose course is uncertain. Courts can either force probabilistic evidence into the all-or-nothing framework, producing arbitrary results at the margins, or reshape the framework to accommodate probabilities, as the loss-of-chance doctrine does, at the price of greater complexity and expanded liability. Neither approach solves the underlying problem, which is that what would have happened is unknowable with precision. That irreducible uncertainty is another source of the unpredictability that physicians experience, and another reason that the connection between negligence and liability is looser than either side of the malpractice debate admits. It also bears on damages, because the value of what a patient lost depends on what the patient would otherwise have had. The next chapter turns to how damages are measured, and why their largest and least tangible component became the target of reform. Chapter 6: Valuing Harm: Damages in Malpractice Cases A malpractice plaintiff who proves duty, breach and causation is entitled to damages: a sum of money meant to restore, as far as money can, what the negligence took away. In theory this is straightforward. In practice, damages are where the stakes of malpractice litigation are set, where most of the money in the system flows, and where the political conflict over malpractice law has been fiercest. Understanding how damages are measured is essential to understanding why tort reformers have focused so heavily on them, and what caps actually do. American tort law divides compensatory damages into two broad categories. Economic damages compensate for financial losses that can in principle be measured in the market: medical expenses, lost earnings, the cost of future care. Noneconomic damages compensate for losses that have no market price: pain, suffering, disfigurement, loss of enjoyment of life, emotional distress and, in some cases, loss of companionship. A third category, punitive damages, is meant to punish and deter rather than compensate, and is rare in malpractice cases. Economic damages: the cost of a damaged life Economic damages are often the largest component of major malpractice awards, particularly where a young patient suffers severe, permanent injury. They are proven through a specialized set of experts. Past medical expenses are established from bills and records. Future medical and care costs are established through a life care plan, prepared by a nurse, physician or rehabilitation specialist, that sets out every item of care the plaintiff is expected to need for the rest of life: physician visits, therapies, medications, equipment, home modifications, attendant care, and the frequency and cost of each. A life expectancy expert, often a physician or epidemiologist, testifies about how long the plaintiff is likely to live, which in severe injury cases is itself contested, because the injury may shorten life expectancy and because longer life expectancy increases damages. An economist then projects these costs into the future and reduces them to present value, applying assumptions about inflation in medical costs and the rate of return on investment. For a child with severe brain injury requiring round-the-clock care, the resulting figure may run to many millions of dollars. Lost earnings are calculated in a similar way. For an adult with an established career, the economist projects what the plaintiff would have earned over a working life and subtracts what the plaintiff can still earn. For children, the calculation is inherently speculative, and it has historically depended on statistical assumptions tied to education, and sometimes to race and sex, that have come under criticism. Earnings tables that project lower lifetime earnings for women and for Black and Hispanic children systematically reduce awards for those plaintiffs. Scholars including Martha Chamallas and Jennifer Wriggins have documented the effect, and some courts and legislatures have begun to restrict the use of race-based and sex-based projections; a California statute that took effect in 2022 prohibits reduction of damages for future earnings based on race, ethnicity or gender. The structure of economic damages has a consequence that is central to the reform debate. Plaintiffs with high incomes recover large economic damages when they are injured; plaintiffs without earnings, including children, the elderly, the unemployed and people who work in the home, recover little in lost earnings. For those plaintiffs, noneconomic damages and future care costs are most of what the tort system offers. Noneconomic damages: pricing suffering Noneconomic damages compensate for pain, suffering, mental anguish, loss of enjoyment of life, physical impairment and disfigurement. They are real losses. A person who has lost the use of her legs or the ability to have children or years of expected life has suffered a harm that money cannot measure but that the law, having no other remedy, compensates in money. The difficulty is that there is no standard for translating suffering into dollars. Juries receive instructions that tell them to award a sum that is reasonable and fair, without further guidance. Unsurprisingly, awards for similar injuries vary widely. Empirical studies have found that noneconomic awards do generally rise with the severity of injury, which suggests that juries are not acting randomly, but the variation among cases of similar severity is large. Scholars have proposed ways to reduce it, including schedules of values tied to injury severity, the use of prior awards as benchmarks, and guidance on per diem arguments, in which counsel suggests a daily value for suffering and multiplies it over the plaintiff's life expectancy. Many states restrict or prohibit per diem arguments; some allow counsel to suggest a total figure, a practice known as anchoring, which studies show influences awards. Georgia's 2025 tort reform legislation restricted anchoring arguments in civil cases, reflecting the concern that suggested figures untethered to evidence inflate verdicts. Noneconomic damages are also the most visible source of the very large awards, often called nuclear verdicts, that attract public attention. Such verdicts are frequently reduced on post-trial motions or appeal, through the device of remittitur, in which the court offers the plaintiff a choice between a reduced award and a new trial, or through statutory caps where they apply. The headline verdict often has little relation to what is ultimately paid. Because noneconomic damages are indeterminate, variable and sometimes very large, they became the primary target of tort reform. The logic is straightforward: capping the component of damages that is hardest to predict should make liability more predictable, reduce the size of awards and settlements, reduce insurance premiums and, reformers argue, reduce the fear that drives defensive medicine. Chapters 7 and 8 examine whether that logic holds. Collateral sources, periodic payments and the mechanics of awards Beyond the headline categories, a series of technical rules affects what plaintiffs actually receive and what defendants actually pay. The collateral source rule, a traditional common-law principle, provides that damages are not reduced because the plaintiff received compensation from another source, such as health insurance or disability benefits. The rationale is that a wrongdoer should not benefit from the plaintiff's foresight in buying insurance, and that the insurer typically has a right to be reimbursed from the recovery through subrogation or a lien. Critics argue that the rule allows double recovery. Many states have modified it for malpractice cases, either by allowing the jury to hear evidence of collateral payments or by requiring the court to reduce the award by the amount of such payments. California's MICRA allowed defendants to introduce evidence of collateral source payments and barred the collateral source from seeking reimbursement from the plaintiff. A related and increasingly important dispute concerns the measure of past medical expenses: whether a plaintiff may recover the amount billed by providers or only the amount actually paid, which for insured patients is often much lower. Courts and legislatures have divided, with many, including California in Howell v. Hamilton Meats and Provisions (2011), limiting recovery to the amount paid. Periodic payment statutes allow or require that large awards for future damages be paid over time rather than as a lump sum, often through an annuity. They reduce the defendant's immediate cost and, in some versions, provide that payments for future care stop if the plaintiff dies earlier than expected, reflecting the fact that the award was calculated on the assumption of a longer life. Plaintiffs' advocates argue that such rules shift risk to injured patients and deprive them of control over their recoveries. Joint and several liability determines what happens when several defendants are liable. Under the traditional rule, each is liable for the entire judgment, and the plaintiff can collect it from whichever defendant can pay, leaving that defendant to seek contribution from the others. Many states have modified or abolished the rule, making each defendant liable only for its proportionate share of fault, at least for noneconomic damages. The change matters in malpractice cases because hospitals and well-insured defendants were often the ones who paid when other defendants were underinsured. Prejudgment interest rules, which in some states add interest to damages from the date of injury or filing, affect settlement incentives, since defendants who delay pay more. Wrongful death and the valuation of a life When malpractice causes death, the claims are brought by the estate and survivors under wrongful death and survival statutes. The survival action continues the claim the decedent would have had, including pain and suffering before death and medical expenses. The wrongful death action compensates the survivors for their own losses, which traditionally meant the financial support the decedent would have provided and, in most states, the loss of companionship, guidance and society. The measure of wrongful death damages has a distinctive consequence. In many states, damages focus on the survivors' losses rather than the value of the life itself. Where the decedent was a child, an elderly retiree or an adult without dependents, economic losses to the survivors may be small, and recoverable noneconomic damages may be limited by statute. Several states restrict who may recover for loss of companionship. Florida's wrongful death statute, for example, bars adult children from recovering noneconomic damages for the death of a parent, and bars parents from recovering such damages for the death of an adult child who had no spouse or children, in cases arising from medical negligence. Critics have called this provision the "free kill" law, because it can leave no one entitled to meaningful damages when an unmarried adult without minor children dies from negligent care. Repeal bills have been introduced repeatedly in the Florida legislature and advanced in recent sessions, but opposition from physician and hospital groups and the governor has so far prevented enactment. New York presents a related debate. Its wrongful death statute, largely unchanged since the nineteenth century, limits recovery to pecuniary losses and does not allow damages for grief or loss of companionship. The legislature has repeatedly passed the Grieving Families Act to expand recoverable damages, and Governor Kathy Hochul vetoed it three times, in January 2023, December 2023 and December 2024, citing concerns about insurance costs and health care affordability, while signaling openness to a narrower version. These debates illustrate a broader point about damages and caps. When the law limits or excludes noneconomic damages, the cases most affected are those in which the injured person had little in economic losses. In death cases, that means children, the elderly and adults without dependents. The effect is not incidental: it follows from the structure of damages, and it bears directly on who can find a lawyer, as the next chapter shows. Punitive damages Punitive damages are available in malpractice cases only upon proof of conduct far worse than ordinary negligence: typically malice, fraud, or reckless or wanton disregard for the patient's safety. They are awarded rarely. Examples include cases of physicians who operated while intoxicated, falsified records to conceal errors or performed procedures for profit on patients who did not need them. Many states impose additional limits on punitive damages generally, including heightened proof requirements, caps tied to compensatory damages and requirements that a portion be paid to the state, and the Supreme Court's decisions in BMW of North America v. Gore (1996) and State Farm Mutual Automobile Insurance Co. v. Campbell (2003) set constitutional limits on their ratio to compensatory damages. In malpractice, they are a minor feature of the landscape, and most malpractice insurance policies either exclude punitive damages or are issued in states where public policy bars insuring them. Damages and the logic of the system Damages rules decide not only what injured patients receive but which cases get brought at all. Because plaintiffs' lawyers work on contingency and must advance the high costs of expert-driven litigation, a case is viable only if the expected recovery comfortably exceeds those costs. Research on malpractice lawyers suggests that many will not take a case unless expected damages reach several hundred thousand dollars. A case with clear negligence and modest damages, such as a missed fracture that delays healing or a medication error that causes a week's additional hospitalization, is usually economically impossible to bring. This means that the tort system's deterrent signal is concentrated on catastrophic injuries: deaths, brain damage, paralysis, serious birth injuries. Negligence that causes moderate harm, which is far more common, is largely invisible to the system. And any rule that reduces the value of claims, whether a cap, a collateral source offset or a restriction on wrongful death damages, pushes additional cases below the threshold of viability. Those most affected are the patients whose damages are primarily noneconomic. The next chapter shows how that dynamic played out when states enacted the most consequential reform of all: caps on noneconomic damages. Chapter 7: Crisis and Reform: The Politics and Law of Damage Caps Medical malpractice has been the subject of three major national "crises" in the past half century, each marked by sharply rising insurance premiums, physicians protesting and threatening to leave practice, insurers withdrawing from markets, and legislatures responding with reform. The reforms enacted in those periods, above all caps on noneconomic damages, now define the malpractice landscape in much of the country. This chapter traces how they came about, what they do, how courts have treated them and what the evidence says about their effects on the things they were supposed to change. Three crises The first crisis came in the mid-1970s. Malpractice claims had grown rapidly through the 1960s, and several large commercial insurers either raised premiums dramatically or stopped writing malpractice coverage altogether. In California, premiums for some specialties rose several-fold in a short period, and anesthesiologists and other physicians staged work slowdowns. Legislatures across the country responded. California enacted the Medical Injury Compensation Reform Act of 1975, known as MICRA, which became the template for reform nationally. Physicians also created their own insurers: physician-owned mutual companies, many founded in this period, came to dominate the market and still insure a large share of American physicians. The second crisis came in the mid-1980s, with a similar pattern of rising premiums and legislative response. The third came in the early 2000s, when premiums again rose sharply in many states, some insurers exited, and physician groups in states such as Pennsylvania, West Virginia and Nevada organized protests, including temporary closures of trauma centers. The American Medical Association identified a list of states in "crisis." Congress considered, and the House repeatedly passed, federal legislation modeled on MICRA, including a $250,000 cap on noneconomic damages; the Senate never enacted it. State legislatures acted instead. Texas, in 2003, enacted the most sweeping package, including a constitutional amendment approved by voters to protect caps from challenge in the state courts. Why did these crises happen? Physicians and insurers blamed rising claims and jury awards. Plaintiffs' lawyers and consumer advocates blamed the insurance cycle. Scholars who studied the question, including Tom Baker in The Medical Malpractice Myth (2005), found that both played a part but that the insurance cycle explained much of the timing. Malpractice insurance is a long-tail line: claims are paid years after premiums are collected, so insurers invest premiums and rely heavily on investment income. When interest rates are high and investment returns strong, insurers compete for market share by underpricing; when returns fall, or when earlier underpricing becomes apparent, they raise prices sharply to rebuild reserves. Each malpractice crisis followed a period of soft pricing and coincided with falling investment returns. Claims costs rose over time, but more gradually than premiums did during the crisis years. This does not mean claims costs were irrelevant; it means that the sharp spikes that drove political action owed much to the financial dynamics of insurance. A study by Bernard Black, Charles Silver, David Hyman and William Sage of closed claims in Texas from 1988 to 2002, published in the Journal of Empirical Legal Studies in 2005, found that the number of paid claims per capita and the real value of payments were largely stable over that period, despite the sense of crisis that led to the 2003 reforms. Their title summed up the finding: "Stability, Not Crisis." The architecture of reform Tort reform in malpractice is a menu, and states have ordered different combinations. The main items are caps on noneconomic damages, caps on total damages, limits on attorney contingency fees, modification of the collateral source rule, periodic payment of future damages, shortened statutes of limitations and repose, abolition or modification of joint and several liability, certificates or affidavits of merit, expert qualification requirements, pretrial screening panels, and heightened standards for emergency care. Some states also created patient compensation funds, which provide excess coverage above a primary layer of private insurance and are funded by provider surcharges. Of these, caps on noneconomic damages are the centerpiece, and the most studied. The research described below and in Chapter 8 consistently finds that caps are the one reform with clear effects on payments; the effects of most other items, taken alone, are harder to detect. California's MICRA is the most important example. For nearly fifty years it capped noneconomic damages in malpractice cases at $250,000, a figure never adjusted for inflation. By 2022 that cap was worth, in real terms, only a small fraction of its 1975 value. A proposed ballot initiative to raise the cap sharply led to a negotiated compromise between physician organizations, hospitals and plaintiffs' lawyers, enacted as Assembly Bill 35 in 2022. Beginning in 2023, the cap rose to $350,000 in cases not involving death and $500,000 in death cases, increasing by $40,000 and $50,000 respectively each year until reaching $750,000 and $1 million, after which the caps will be adjusted annually for inflation. Under that schedule, the caps for 2026 stand at $470,000 and $650,000. The new law also allows separate caps to apply to different categories of defendants: health care providers, health care institutions and unaffiliated providers can each be subject to a cap, so that a single case can produce up to three caps' worth of noneconomic damages. The same legislation adjusted contingency fee limits and made statements of sympathy inadmissible as evidence of liability, a provision discussed in Chapter 9. Texas took a different path. Its 2003 legislation capped noneconomic damages at $250,000 per claimant against all physicians collectively, with an additional $250,000 available against each of up to two health care institutions, for a maximum of $750,000. Unlike MICRA's revised caps, the Texas limits are not indexed to inflation. Texas also capped total damages in wrongful death cases, excluding medical expenses, at $500,000 in 1977 dollars, adjusted for inflation. Proposition 12, approved by voters in 2003, amended the Texas Constitution to authorize the legislature to limit noneconomic damages in health care cases, forestalling the kind of state constitutional challenge that has defeated caps elsewhere. Other states chose total caps. Indiana's Medical Malpractice Act, one of the oldest, caps total recovery, now at $1.8 million for injuries occurring after mid-2019, with the individual provider liable for the first $500,000 and a state Patient Compensation Fund liable for the excess. Virginia caps total damages at a level that rises by statute in fixed annual steps toward $3 million. Nebraska and Louisiana have total caps as well. Total caps limit economic damages, including future medical care, and thus bite hardest on the most severely injured patients. The details of caps matter. Some states set higher caps for catastrophic injuries or death. Some apply one cap per claimant, others per defendant, others per occurrence. Some are indexed to inflation and some are not; an unindexed cap tightens every year. And some apply only to certain categories of defendants. Several states have revised their caps in recent years, as California did: Colorado in 2024 enacted legislation substantially raising its long-static medical malpractice limits, phased in over several years, and Nevada in 2023 legislated stepwise increases to its $350,000 cap. Iowa went the other direction in 2023, strengthening its caps and setting a $1 million ceiling on noneconomic damages against hospitals. Caps in the courts Caps have been challenged under state constitutions, typically on the grounds that they violate the right to trial by jury, equal protection, the separation of powers, the right of access to the courts or prohibitions on special legislation. The results vary sharply by state, and they turn more on the text and history of each state's constitution than on any general principle. Table 4 sets out a selection of the leading decisions. Table 4. Selected state high court decisions on caps on noneconomic damages in malpractice cases. State Case Year Result Principal ground California Fein v. Permanente Medical Group 1985 Upheld Rational response to insurance crisis Ohio Arbino v. Johnson & Johnson 2007 Upheld (general tort cap) Jury right and due process not violated Georgia Atlanta Oculoplastic Surgery v. Nestlehutt 2010 Struck down Right to jury trial Illinois Lebron v. Gottlieb Memorial Hospital 2010 Struck down Separation of powers Missouri Watts v. Lester E. Cox Medical Centers 2012 Struck down Right to jury trial Florida North Broward Hospital District v. Kalitan 2017 Struck down Equal protection Wisconsin Mayo v. Wisconsin Injured Patients and Families Compensation Fund 2018 Upheld Rational basis; legislative judgment Courts upholding caps have generally applied deferential review, reasoning that the legislature could rationally conclude that caps would stabilize insurance markets and preserve access to care, and that the jury's role is to find facts, while the legislature may define the legal consequences. Courts striking them down have reasoned that the amount of damages is a factual question within the jury's constitutional province, or that singling out the most severely injured malpractice victims to bear the cost of an insurance problem is irrational or discriminatory. The Georgia Supreme Court in Nestlehutt emphasized that the right to jury trial as it existed when the state constitution was adopted included the determination of damages. The Florida Supreme Court, first in Estate of McCall v. United States (2014) for wrongful death and then in Kalitan for personal injury, held that caps violated equal protection because they imposed the burden on the most severely injured and because there was no longer any evidence of a crisis to justify them. Wisconsin illustrates how the question can shift within a single state. The Wisconsin Supreme Court struck down an earlier $350,000 cap in Ferdon v. Wisconsin Patients Compensation Fund (2005); the legislature responded with a $750,000 cap, which the court upheld in Mayo in 2018, explicitly overruling Ferdon's approach to rational basis review. Missouri illustrates how legislatures can respond to adverse rulings. After Watts held that caps violated the right to jury trial for common-law causes of action, the legislature in 2015 abolished the common-law malpractice claim and replaced it with a statutory cause of action subject to caps, reasoning that the jury right does not protect claims created by statute. The Missouri Supreme Court had already upheld caps on statutory wrongful death claims on similar reasoning. What caps actually do The evidence on caps is now extensive, and some of its conclusions are firm. First, caps reduce payments. Studies consistently find that noneconomic caps reduce average malpractice payments, though estimates vary with the cap's level and design. A study by Seth Seabury and colleagues in Health Affairs in 2014, using national claims data, found that caps reduced average payments by roughly 15 percent, with larger effects in specialties whose claims involve severe injuries. A tight cap, such as MICRA's before 2023 or Texas's, has large effects on cases with major noneconomic damages. In Texas, research by Hyman, Black, Silver and others found that the 2003 cap substantially reduced payments in cases where it bound, and that the number of claims and payments fell. Second, caps reduce malpractice insurance premiums, at least over time, though the effect is modest compared with the swings of the insurance cycle, and premiums in both capped and uncapped states fell substantially from the mid-2000s through the 2010s as the market softened. The broad national decline in paid claim rates documented in Chapter 1 occurred in states with and without caps. Third, caps change which cases are brought. Because they reduce expected recoveries, they make it uneconomic for lawyers to take many cases. The effect falls disproportionately on plaintiffs whose damages are mainly noneconomic: children, the elderly, homemakers, and people with low incomes. Research on Texas found that after the 2003 cap, lawyers largely stopped taking cases involving elderly patients and deaths of people without dependents, since the capped recovery would not cover the cost of litigation. Studies of California under the unadjusted MICRA cap reached similar conclusions, and the difficulty of finding counsel for such cases was one of the arguments that led to the 2022 compromise. Caps also shift the composition of awards toward economic damages, which favor higher earners. Fourth, caps have at most modest effects on physician supply. Reformers argued that caps would attract physicians to a state and keep them from leaving. Some studies have found small positive effects on the supply of certain specialists, particularly in rural areas. But the Texas experience, closely studied, did not support claims of a large effect: David Hyman, Charles Silver, Bernard Black and Myungho Paik found that physician supply in Texas after 2003 grew in line with pre-reform trends and did not show the surge that reform advocates had predicted, despite widely circulated claims of a physician influx. Fifth, and most important for the policy debate, caps have limited effects on health care spending. That question, and the defensive medicine on which it depends, is the subject of the next chapter. Screening panels and other procedural reforms Beyond caps, the reform menu includes several procedural devices, of which pretrial screening panels are the most widely tried. Panels typically consist of a physician, a lawyer and sometimes a layperson, who review a claim before it goes to court and issue an opinion on whether the standard of care was breached. In some states the panel's opinion is admissible at trial. The theory is that an early neutral assessment will encourage plaintiffs to drop weak claims and defendants to settle strong ones. The experience has been mixed. Several states abolished their panels after finding that they added delay and cost without clearly affecting outcomes, and several courts have struck down panel requirements under state constitutions. Indiana's medical review panel, part of its comprehensive act, has operated for decades, but evaluations have not found clear benefits. The broader lesson is that procedural reforms that add steps to litigation tend to reduce the number of claims by raising their cost, without clearly improving the accuracy with which claims are sorted. Caps reduce payments and change the composition of cases. None of these reforms address the underlying problems identified in the earlier chapters: the uncertainty of the standard of care, the difficulty of causation, and the resulting mismatch between negligence and liability. Whether they change what physicians actually do in treating patients is the question to which the reform debate ultimately returns. Chapter 8: The Economics of Defensive Medicine The strongest argument for malpractice reform has never been the cost of malpractice insurance or the size of jury awards. Those costs, while real, are a small fraction of national health spending. The strongest argument is that fear of liability changes how physicians practice, leading them to order tests, procedures and consultations that patients do not need, at great cost to the health system. If that is true on a large scale, then malpractice law imposes costs far beyond the claims it resolves, and reforms that reduce liability could save far more than they cost injured patients. This chapter examines that argument. Defensive medicine is real; almost no one who has studied the question doubts it. The disputed questions are how large it is, what causes it, and whether the reforms usually proposed would reduce it. What defensive medicine is Defensive medicine is conventionally defined as medical practices undertaken primarily to reduce the risk of liability rather than to benefit the patient. It takes two forms. Assurance behavior, sometimes called positive defensive medicine, involves supplying additional services: ordering an extra imaging study, admitting a patient for observation who could safely go home, referring to a specialist, performing a cesarean section at an equivocal fetal heart tracing. Avoidance behavior, or negative defensive medicine, involves withholding services: declining to treat high-risk patients, abandoning high-risk procedures, or leaving a high-risk specialty or practice location. Both forms are hard to measure, for a basic reason: defensive motives are mixed with others. A physician who orders a CT scan for a patient with a headache may be responding to liability fear, to patient demand, to financial incentives under fee-for-service payment, to uncertainty about the diagnosis, to habit, or to genuine concern that the scan might reveal something. Most extra tests are over-determined. Some also have real, if small, clinical value, which complicates the distinction between defensive and prudent medicine. And the distinction between a service that is defensive and one that is merely low-value is often a matter of framing. What physicians say Surveys have consistently found that most physicians report practicing defensive medicine. The most frequently cited is a survey of Pennsylvania physicians in high-risk specialties conducted by David Studdert and colleagues and published in JAMA in 2005, at the height of the most recent malpractice crisis. Of the 824 physicians who responded, 93 percent reported practicing defensive medicine. The most common forms of assurance behavior were ordering more tests than medically indicated, prescribing more medications, referring patients to specialists unnecessarily, and ordering invasive procedures to confirm diagnoses. Forty-two percent reported restricting their practice in the preceding three years, for example by eliminating procedures prone to complications or avoiding patients with complex medical problems or who seemed litigious. Later surveys found similar results. Physicians in all specialties report that liability concerns shape their decisions, and many estimate that a substantial share of the tests and procedures they order are at least partly defensive. Some physician organizations have extrapolated these reports into estimates that defensive medicine costs hundreds of billions of dollars annually. Surveys are a weak basis for such estimates. Physicians' reports of their own motives are subject to recall and reporting bias, and during a period of political conflict over tort reform, physicians have an incentive to emphasize liability costs. Surveys also cannot tell what physicians would do differently if liability risk changed; a physician who says she orders a test partly for defensive reasons may continue to order it for the other reasons if the defensive motive disappeared. For that, one needs to observe behavior. What the behavioral evidence shows The most important studies of defensive medicine measure what happens to physician behavior and health spending when liability risk changes. Table 5 summarizes several of the most influential. Table 5. Selected empirical studies of defensive medicine and health spending. Study Setting and method Principal finding Kessler and McClellan, Quarterly Journal of Economics (1996) Elderly Medicare heart patients; states adopting direct tort reforms Reforms reduced hospital spending 5 to 9 percent with no measurable effect on mortality or complications Congressional Budget Office (2009) Analysis of a national package including a $250,000 cap National health spending lower by about 0.5 percent, of which about 0.3 percent from reduced defensive medicine Mello, Chandra, Gawande and Studdert, Health Affairs (2010) Synthesis of evidence on components of liability costs Total liability system costs about $55.6 billion (2008), about 2.4 percent of health spending; defensive medicine about $45.6 billion of the total Paik, Black, Hyman and Silver, Journal of Empirical Legal Studies (2012) Texas Medicare spending before and after 2003 reforms No evidence that reforms reduced spending relative to control states Frakes and Gruber, American Economic Journal: Economic Policy (2019) Active-duty military patients, who cannot sue for care in military facilities, compared with patients who can Immunity from liability associated with about 5 percent less treatment intensity, without worse outcomes The pioneering study was by Daniel Kessler and Mark McClellan, published in the Quarterly Journal of Economics in 1996. They examined elderly Medicare patients hospitalized for heart attacks and ischemic heart disease, comparing states that adopted "direct" tort reforms, such as caps on damages, with those that did not. They found that direct reforms reduced hospital expenditures for these patients by 5 to 9 percent within a few years, without significant effects on mortality or serious complications. If generalized to all health spending, the savings would be very large. The study was enormously influential in the policy debate. But later researchers who attempted to extend it to other conditions and populations generally found smaller effects, and some found none. The Congressional Budget Office, asked in 2009 to estimate the effects of a package of reforms including a $250,000 cap on noneconomic damages, a $500,000 cap on punitive damages, modification of the collateral source rule, a statute of limitations and replacement of joint and several liability with a fair share rule, concluded that the package would reduce total national health spending by about 0.5 percent, or roughly $11 billion in 2009. About 0.2 percent of that would come from lower malpractice premiums and 0.3 percent from reduced defensive medicine. The CBO estimated the federal deficit would be reduced by about $54 billion over ten years. These are real savings, but they are a small fraction of what advocates of reform had claimed. The most thorough attempt to estimate the total cost of the liability system was by Michelle Mello, Amitabh Chandra, Atul Gawande and David Studdert, published in Health Affairs in 2010. They estimated that the medical liability system, including payments to claimants, administrative costs and defensive medicine, cost about $55.6 billion in 2008, or about 2.4 percent of national health spending. Defensive medicine accounted for the largest share, about $45.6 billion, most of it in hospital services. The authors emphasized the uncertainty of the estimate and noted that it was significant but much smaller than figures in wide circulation. The Texas experience provides a natural test. If caps reduce defensive medicine, the sweeping 2003 reforms in Texas should have produced lower spending relative to other states. Myungho Paik, Bernard Black, David Hyman and Charles Silver examined Medicare spending in Texas and comparison states before and after the reforms and found no evidence that spending fell. If anything, Texas spending rose faster than in comparable states. Other studies of Texas found similar results for particular types of care. The most sophisticated recent study is by Michael Frakes and Jonathan Gruber, published in the American Economic Journal: Economic Policy in 2019. They exploited an unusual natural experiment: active-duty military personnel treated in military hospitals cannot sue for malpractice because of the Feres doctrine, while military dependents treated in the same facilities can, and both groups are also treated by civilian providers outside military facilities, where everyone can sue. Comparing how treatment differed across these groups and settings, the authors found that the absence of liability was associated with about 5 percent less treatment intensity, with no measurable effect on health outcomes. This is strong evidence that defensive medicine exists and is substantial in magnitude. It is also evidence that full immunity, not a cap, is what produces the effect. Reconciling the evidence Taken together, these studies suggest several conclusions. First, defensive medicine is real, and liability affects behavior. The military study and some of the state reform studies find meaningful effects on treatment intensity when liability changes. The profession's reports of defensive practice are not invented. Second, the effect of typical reforms, particularly caps, is modest. The CBO's estimate of 0.3 percent of health spending from reduced defensive medicine for a national cap package is consistent with the bulk of the literature. The Texas studies found little or nothing. The large effects in the military study came from complete immunity, which no state has enacted for civilian care, and which would raise obvious concerns about accountability. Third, there is a gap between physicians' reported fear and the objective risk that caps modify. Caps reduce the size of payments in cases with large noneconomic damages. But physicians' main fear is not of large payments, which their insurance covers, but of being sued at all: the time, stress, reputational damage and professional disruption of litigation, and the report to the National Practitioner Data Bank that follows any payment. Caps do not reduce the chance of being sued, except to the extent that they discourage some claims with modest damages, and they do little to reduce the uncertainty about whether particular conduct will lead to a suit. Surveys have found that physicians in states with caps report defensive practices at rates similar to those in states without them. Research has also found that physicians significantly overestimate their risk of being sued, which suggests that reforms reducing objective risk may not reduce perceived risk proportionally. Fourth, defensive medicine coexists with other incentives. Under fee-for-service payment, extra tests and procedures also generate revenue for the physician or institution. Where liability and financial incentives both point toward more care, reducing liability alone may have little effect, because the financial incentive remains. Conversely, where financial incentives point toward less care, as in capitated systems and some integrated delivery systems, defensive pressures may be offset. This is one reason the military setting is informative: its physicians are salaried, so the effect of removing liability can be observed without the confounding pull of fee-for-service revenue, though for the same reason the size of the effect may not carry over exactly to civilian practice. Fifth, the content of legal standards matters, not just the size of damages. The finding that shifting from local to national standards of care changed practice patterns, described in Chapter 2, suggests that physicians respond to what the law requires, not merely to how much they might have to pay. Frakes and colleagues have also found evidence that when legal standards move toward national norms, regional variations in practice decrease. A reform that made the standard of care clearer might affect behavior more than a cap on damages. Defensive medicine and the standard of care The connection between defensive medicine and the standard of care is closer than it first appears. Because the standard is defined by professional custom, practices adopted defensively can, over time, become the standard. If most emergency physicians order a CT scan for minor head injuries because they fear being sued for missing a bleed, then ordering the scan becomes customary practice, and a physician who does not order it may be found to have breached the standard. Defensive medicine thus tends to ratchet: each round of defensive practice raises the standard against which future practice is judged. Efforts to reverse the ratchet have come from within medicine. The Choosing Wisely campaign, launched in 2012 by the ABIM Foundation with specialty societies, published lists of commonly overused tests and treatments, many of them the kind of low-value services associated with defensive practice, such as imaging for uncomplicated low back pain and routine preoperative testing for low-risk surgery. Clinical decision rules, such as validated rules for when a head injury requires CT scanning, provide evidence-based grounds for not ordering tests. Because the standard of care incorporates professional norms, these efforts may, over time, change what counts as reasonable practice and provide physicians with defensible grounds for doing less. There are limits. A physician who follows a decision rule and misses a rare case still faces the prospect of a jury considering the case with the benefit of hindsight. Hindsight bias, the tendency to judge a decision as worse when one knows it led to a bad outcome, is well documented in experiments involving jurors and physicians reviewing records. It means that a physician who declines a test with a one-in-a-thousand yield, and whose patient turns out to be the one, will face a jury that knows the outcome and may find it hard to accept that the decision was reasonable. The fear of that scenario, more than any calculation of expected payment, is what drives physicians to order the test. What follows The economics of defensive medicine thus returns us to the book's central argument. Defensive medicine is driven less by the size of damage awards than by the unpredictability of liability, and unpredictability is rooted in the features examined in earlier chapters: a standard of care that must be reconstructed case by case through competing experts, causation questions that often cannot be answered with confidence, and a system that sorts negligent from non-negligent care only imperfectly. Caps reduce what injured patients receive, and they reduce the number of claims with modest damages, but they do little about unpredictability, which is why their effects on defensive medicine are small. If this analysis is right, reforms aimed at defensive medicine should aim to make liability more accurate and more predictable, rather than merely smaller. The last chapter examines reforms that attempt to do that. Chapter 9: Beyond the Courtroom: Disclosure, Resolution and No-Fault Alternatives If the central failings of malpractice litigation are that it compensates few injured patients, costs a great deal to do so, sorts negligent from non-negligent care imperfectly and communicates liability risk to physicians in a way that promotes fear rather than learning, then the most promising reforms are those that address those failings directly. Over the past three decades, a set of alternatives has emerged: programs that disclose errors to patients and offer compensation early, laws that encourage apologies, proposals for specialized health courts, and administrative no-fault compensation schemes, some already operating in the United States and many abroad. This chapter examines them, and the evidence on whether they work. Liability as a driver of safety: the anesthesia story Before turning to alternatives, it is worth recognizing that the liability system has, on occasion, driven real improvements in safety. The clearest case is anesthesiology. In the early 1980s, anesthesiology was among the highest-risk specialties for malpractice, with high premiums and frequent catastrophic claims involving brain damage and death from unrecognized oxygen deprivation during surgery. The American Society of Anesthesiologists responded by launching a Closed Claims Project in 1985, which systematically analyzed closed malpractice claims to identify the patterns of injury behind them. The analysis found that a large proportion of the most serious injuries involved problems with ventilation, such as esophageal intubation, that better monitoring could detect. At about the same time, anesthesiologists at the Harvard-affiliated hospitals adopted formal standards for intraoperative monitoring, published in JAMA in 1986, and the American Society of Anesthesiologists soon adopted national monitoring standards, which came to include pulse oximetry and capnography, the continuous measurement of blood oxygen and exhaled carbon dioxide. Anesthesia-related deaths fell substantially over the following years, and anesthesiology malpractice premiums fell with them. The anesthesia story is instructive for two reasons. It shows that malpractice claims, analyzed systematically, can be a rich source of safety information. And it shows that the professional standard of care can be raised deliberately by the profession itself, with liability following rather than leading. The claims data identified the problem; the profession changed the standard; the change reduced both injuries and claims. The alternatives discussed in this chapter attempt, in different ways, to replicate that sequence. Communication-and-resolution programs The most promising reform to emerge in the past quarter century is the communication-and-resolution program. The model's core commitments are that when a patient is harmed by care, the institution promptly discloses what happened, investigates, explains the findings to the patient and family, apologizes where appropriate, takes steps to prevent recurrence and, where the investigation shows that care was unreasonable, offers fair compensation without requiring the patient to sue. Where care was reasonable, the institution explains why and defends vigorously if a claim is filed. The model's origins lie in the Veterans Affairs Medical Center in Lexington, Kentucky, which in 1987 adopted a policy of disclosing errors and offering settlement after losing two major malpractice judgments. Steve Kraman and Ginny Hamm described the program in the Annals of Internal Medicine in 1999 under the title "Risk Management: Extreme Honesty May Be the Best Policy," reporting that its liability payments were moderate compared with similar facilities, despite the conventional fear that disclosure would invite more claims. The best-known program is that of the University of Michigan Health System, which adopted a disclosure-and-offer approach in 2001 under the leadership of its chief risk officer, Richard Boothman. A study by Allen Kachalia and colleagues, published in the Annals of Internal Medicine in 2010, compared the period before and after the program's implementation. The rate of new claims and lawsuits fell, the median time to resolve claims fell substantially, and total liability costs, including payments and legal expenses, declined. The study was observational and could not rule out the influence of other trends, but its results were striking enough to spur wide interest. Other programs followed, including one operated by the Colorado physician-owned insurer COPIC, known as the 3Rs program (Recognize, Respond, Resolve), which offers limited payments for out-of-pocket losses after unexpected outcomes without requiring a determination of fault or a waiver of the right to sue, and programs at Stanford, several Massachusetts hospitals and other systems. In 2016 the federal Agency for Healthcare Research and Quality published the CANDOR toolkit (Communication and Optimal Resolution) to help hospitals implement the approach. Evaluations of programs in Massachusetts led by Michelle Mello and colleagues found that hospitals could implement them successfully and that patients generally reported favorable experiences, though implementation required sustained institutional commitment and cooperation from liability insurers. Why might such programs work? Research on why patients sue suggests several reasons. Patients and families often sue because they want to know what happened, want someone to take responsibility, want to prevent the same thing from happening to others, and feel ignored or deceived after an adverse event. Litigation is a poor instrument for these goals, but it may be the only one available when the institution closes ranks. Communication-and-resolution programs address those motives directly. They also reduce transaction costs by resolving meritorious claims early, without the expert-heavy litigation described in Chapter 4, and they give clinicians a way to learn from errors rather than defend against them. The programs have limits. They work best in integrated institutions that employ their physicians and self-insure, since coordination is easier when the hospital and its clinicians share a single insurer. They depend on patients being treated fairly: if the offer is too low, patients may accept it without legal advice and receive less than they would have in court, which raises concerns about the fairness of settlements with unrepresented patients. Most programs encourage patients to consult counsel for this reason. And they do not reach the large majority of negligently injured patients who never realize they were harmed by error, unless the institution's own disclosure identifies them. Apology laws Related to disclosure programs are apology laws, which make statements of sympathy, and in some states admissions of fault, inadmissible as evidence of liability. The motivation is that physicians and hospitals are reluctant to express sympathy or apologize after adverse events for fear that their words will be used against them. Most states now have some form of apology law, and California's 2022 MICRA amendments added broad protection for expressions of sympathy and benevolence. The laws vary. Most protect only expressions of sympathy ("I am sorry this happened") and not admissions of fault ("I am sorry I made this mistake"); a few, including Colorado's, protect admissions as well. The evidence on their effect is mixed. Some studies have found associations between apology laws and faster resolution or lower settlement amounts; a study by Benjamin McMichael, R. Lawrence Van Horn and W. Kip Viscusi found that apology laws did not reduce, and for some categories of physicians may have increased, liability risk, possibly because disclosure alerted patients to errors they would not otherwise have discovered. Apology laws alone, without the structured investigation and compensation of a communication-and-resolution program, appear to have little effect. Health courts A more ambitious proposal would take malpractice claims out of the ordinary courts altogether and place them before specialized tribunals, often called health courts. Proposals developed by the advocacy organization Common Good, working with researchers at the Harvard School of Public Health, envisioned administrative judges with expertise in medicine, supported by neutral medical experts, applying a standard of avoidability rather than negligence and using schedules to guide noneconomic damages. The goal is to improve accuracy and consistency, reduce transaction costs and make compensation more predictable. The avoidability standard deserves particular attention. Instead of asking whether the care fell below professional custom, the tribunal would ask whether the injury would have been avoided if best practice had been followed. This is broader than negligence, since some injuries that would be avoided by best practice occur in the hands of competent clinicians who follow common practice, and so more patients would be compensated. But with neutral experts and damages schedules, each claim would cost less to resolve. Proponents argue that the standard would also produce better safety information, since it focuses on what could have been done differently rather than on whether someone should be blamed. Health courts face significant obstacles. The main legal one is the right to jury trial under state constitutions, which may make it difficult to require patients to use an administrative forum for claims that would otherwise go to a jury. Proposals typically address this with voluntary participation, offering incentives to choose the administrative route, or with federal legislation. Politically, health courts have been opposed by plaintiffs' lawyers, who see them as a device for reducing compensation, and have not generated sustained enthusiasm among physicians, who are not confident that an avoidability standard would reduce their exposure. The federal Agency for Healthcare Research and Quality funded a set of demonstration projects in 2010 on patient safety and medical liability, including communication-and-resolution programs, but no state has adopted a comprehensive health court. No-fault compensation at home and abroad The most radical alternative abandons fault altogether and compensates injuries caused by medical care without asking whether anyone was negligent. The United States has limited experience with such programs, and several other countries have extensive experience. The two American birth injury programs, in Virginia and Florida, compensate infants with severe neurological injuries sustained during birth, regardless of fault, from funds supported by assessments on physicians and hospitals. They were created during the malpractice crises of the 1980s to stabilize obstetric insurance. Evaluations have found that they provide compensation more quickly and with lower transaction costs than litigation, and that they have helped keep obstetric insurance available. But they have been criticized for narrow eligibility criteria that exclude many injured children, for being underused because families and physicians are unaware of them, and in Florida's case for a history of disputes with families over the adequacy of benefits. They cover only a small category of cases. The National Vaccine Injury Compensation Program, created by the National Childhood Vaccine Injury Act of 1986, is a larger federal no-fault scheme. It compensates people injured by covered vaccines through a special claims process in the Court of Federal Claims, using a table of recognized injuries and time frames that allows compensation without proof of causation in individual cases. It was created to protect vaccine supply after litigation threatened to drive manufacturers from the market, and it has paid thousands of claims. It is often cited as proof that administrative compensation can work in the United States, though it too has been criticized for delays and for the difficulty of claims outside the injury table. Abroad, New Zealand has operated a comprehensive no-fault accident compensation scheme since 1974, administered by the Accident Compensation Corporation, which covers injuries from all causes, including treatment injuries, and in exchange largely bars personal injury lawsuits. Since 2005, New Zealand has compensated treatment injuries without requiring proof of error, defining such injuries as those caused by treatment that are not a necessary part or ordinary consequence of it. Compensation covers medical costs and lost earnings but is limited for noneconomic loss. Sweden established a patient insurance scheme in 1975, now governed by the Patient Injury Act, which compensates avoidable injuries under a standard that asks whether an experienced specialist would have acted differently, or whether an alternative method would have avoided the injury with equal medical benefit. Denmark, Finland and Norway have similar systems. These schemes compensate a much larger share of injured patients than the American tort system does, at much lower transaction costs, and are widely considered successful in their own countries. Researchers have noted, however, that their compensation levels are generally lower than American tort awards and that they operate within comprehensive social welfare systems, including universal health coverage, that cover much of what American tort damages must pay for, such as future medical care. The comparison reveals a tradeoff. The Nordic and New Zealand systems compensate more patients, more quickly and more cheaply, but individual recoveries are lower and the systems depend on social insurance that the United States does not have. They also separate compensation from accountability, which is handled through professional discipline and patient safety systems rather than through the award of damages. Whether such a separation would be acceptable in the United States, where litigation carries expectations of accountability that the other systems do not, is doubtful. What the alternatives show Taken together, the alternatives suggest what a better system would look like. It would compensate more of the patients who are harmed by avoidable error, and it would do so more quickly and at lower cost. It would treat the investigation of adverse events as an opportunity for learning rather than a prelude to litigation. It would communicate honestly with patients, which appears to reduce their need to sue. And it would give physicians a clearer and more predictable sense of what is expected of them and what will happen if something goes wrong, which is what the evidence in Chapter 8 suggests is needed to reduce defensive medicine. No single reform achieves all of this. Communication-and-resolution programs come closest within the existing legal framework and have the strongest evidence behind them. But they depend on voluntary adoption by institutions, and they coexist with the tort system rather than replacing it. The tort system remains the backstop, and the doctrines examined in this book, the standard of care, causation, the rules on experts and damages, continue to define what patients can expect when resolution fails. Conclusion: Accuracy, Not Just Generosity The American debate over medical malpractice has been conducted for half a century largely in terms of generosity. One side argues that the system is too generous: juries award too much, lawyers bring too many claims, insurance costs too much, and physicians practice defensively as a result. The other side argues that it is not generous enough: injured patients are undercompensated, caps punish the most severely harmed, and reform protects negligent providers at the expense of their victims. Each side's preferred reform follows from its diagnosis: caps and procedural hurdles on one side, their repeal on the other. The evidence reviewed in this book suggests that the debate has been conducted along the wrong axis. The central problem of malpractice law is not that it is too generous or too stingy, though in particular respects it is both. It is that it is inaccurate. Most patients injured by negligence never file a claim. A meaningful share of claims involve no error. Claims with and without error are resolved in ways that match the merits only imperfectly. More than half the money spent goes to the process rather than to patients. And physicians experience the result as something close to a lottery, in which the chance of being sued bears an uncertain relationship to the quality of their care. Why the system is inaccurate The sources of that inaccuracy lie in the doctrines examined in the first half of this book. The standard of care is defined by professional custom, which is often unclear and contested, and it must be reconstructed in each case through the testimony of paid experts who disagree. The qualification rules and screening requirements that legislatures have imposed on experts make claims more expensive to bring without clearly improving the accuracy of the ones that survive. Causation requires comparing what happened with a counterfactual that no one can observe, using probabilistic evidence that the all-or-nothing rules of tort law handle poorly and that the loss-of-chance doctrine handles only partly. Damages rules concentrate the system's attention on catastrophic injuries and on plaintiffs with high economic losses, leaving moderate injuries and people without earnings largely outside the system. These are not failures of bad faith. They follow from the attempt to apply a framework designed for traceable, one-off accidents to an activity that is probabilistic, highly technical and conducted on people who are already sick. The uncertainty is partly irreducible. But the way the law handles it can make it better or worse. What caps can and cannot do The second half of the book examined the principal reform of the past fifty years, the cap on noneconomic damages, and the principal argument for it, the reduction of defensive medicine. The evidence supports clear conclusions. Caps reduce payments, particularly in cases involving severe injuries with large noneconomic components. They reduce the number of claims, by making cases with modest economic damages unprofitable to bring. Their burden falls disproportionately on children, the elderly, homemakers and low earners, whose damages are mostly noneconomic. They reduce insurance premiums over time, though less than the insurance cycle does. They have at most modest effects on physician supply. And they have small effects on health spending through reduced defensive medicine, on the order of a fraction of one percent of national spending. Caps have these limited effects on defensive medicine because they do not address its main cause. Physicians practice defensively not primarily because they fear large payments, which are insured, but because they fear being sued at all, and because they cannot predict with confidence which decisions will lead to a suit. Caps leave both of those fears largely intact. The one natural experiment that removed liability entirely, the immunity of military facilities from suits by active-duty personnel, found meaningful reductions in treatment intensity, but complete immunity is not a policy option for civilian care, and it would abandon the accountability that the liability system, for all its flaws, provides. What would help If the problem is inaccuracy and unpredictability, the reforms most likely to help are those that improve accuracy and predictability. Several follow from the evidence. The first is to expand communication-and-resolution programs. They have the strongest evidence of any reform: faster resolution, lower transaction costs, fewer lawsuits and better experiences for patients and clinicians. They work within the existing legal system and require no change in doctrine. Their spread depends on hospitals, insurers and physicians committing to them, and on safeguards ensuring that patients who accept offers are treated fairly. Legislatures can encourage them by protecting the investigations and communications they involve and by supporting their adoption by smaller institutions that lack the resources of large academic centers. The second is to make the standard of care clearer where it can be made clearer. Well-constructed clinical guidelines and validated decision rules can give physicians defensible grounds for not ordering unnecessary tests, and the evidence that legal standards shape practice patterns suggests that clarity matters. Rigid safe harbors have proved unworkable, but courts and legislatures can recognize the evidentiary weight of high-quality guidelines, and professional societies can write guidelines with their legal use in mind, specifying when departure is appropriate. The third is to reduce the adversarial character of expert evidence, which is at the root of the unpredictability. Measures that encourage neutral expert review, whether through court-appointed experts, early neutral evaluation or the independent review that communication-and-resolution programs perform, may do more for the accuracy of outcomes than qualification rules that simply narrow who may testify. The fourth, for states that retain caps, is to set them at levels that do not simply exclude whole categories of patients from the system, and to index them to inflation. California's 2022 compromise, reached after nearly five decades of an unadjusted cap, is a model of how the political stalemate can be broken. States considering caps should recognize that a cap whose effect is to make claims by children and the elderly economically impossible to bring is not a neutral measure. The fifth, more ambitious, is to experiment seriously with administrative compensation for defined categories of injury, as the birth injury and vaccine programs have done. The Nordic and New Zealand experience shows that such systems can compensate more patients at lower cost, but it also shows that they depend on social insurance and on separate mechanisms for accountability, and their lessons must be adapted rather than transplanted. What remains unsettled Several questions remain open. Artificial intelligence is entering clinical practice faster than the law has decided how to treat it, and the way courts handle the first cases in which an algorithm's recommendation was followed or ignored will shape how physicians use such tools for years. The expanding scope of practice for nurse practitioners and physician assistants will force the law to decide what standard of care applies to clinicians performing the same functions with different training. The Supreme Court's 2026 decision that state affidavit-of-merit requirements do not govern in federal court may prompt states to find other ways to screen claims, or may shift some litigation toward federal forums where possible. Debates over wrongful death damages in states such as New York and Florida will continue to test whose losses the law counts. And the long-term effect of rising average payments, documented even as claim frequency has fallen, remains to be seen. The deepest question is the one the book began with: what the law can know. Malpractice litigation asks courts to determine what good medicine required, what happened in a clinical encounter and what would have happened otherwise, and it asks them to do so through an adversarial contest between paid experts, years after the fact, in front of jurors who know how the story ended. It is not surprising that the answers are often imprecise. What is surprising is how much of the policy debate has proceeded as if the only question were how much money should change hands at the end. For patients, for physicians and for the health system as a whole, the more important question is whether the system gets the answer right, and whether it helps medicine learn from the answer. On that measure, the most useful reforms are those that make the system more honest and more accurate, not merely smaller or larger. Notes Introduction and Chapter 1 1. Troyen A. Brennan et al., "Incidence of Adverse Events and Negligence in Hospitalized Patients: Results of the Harvard Medical Practice Study I," New England Journal of Medicine 324 (1991): 370–376. 1. A. Russell Localio et al., "Relation Between Malpractice Claims and Adverse Events Due to Negligence: Results of the Harvard Medical Practice Study III," New England Journal of Medicine 325 (1991): 245–251. 2. David M. Studdert et al., "Claims, Errors, and Compensation Payments in Medical Malpractice Litigation," New England Journal of Medicine 354 (2006): 2024–2033. 3. Adam C. Schaffer et al., "Rates and Characteristics of Paid Malpractice Claims Among US Physicians by Specialty, 1992–2014," JAMA Internal Medicine 177, no. 5 (2017): 710–718. 4. Anupam B. Jena et al., "Malpractice Risk According to Physician Specialty," New England Journal of Medicine 365 (2011): 629–636. 5. Neil Vidmar, Medical Malpractice and the American Jury: Confronting the Myths about Jury Incompetence, Deep Pockets, and Outrageous Damage Awards (University of Michigan Press, 1995). 6. Darling v. Charleston Community Memorial Hospital, 33 Ill. 2d 326, 211 N.E.2d 253 (1965). 7. Thompson v. Nason Hospital, 527 Pa. 330, 591 A.2d 703 (1991). 8. Emergency Medical Treatment and Active Labor Act, 42 U.S.C. § 1395dd. 9. Feres v. United States, 340 U.S. 135 (1950); Federal Tort Claims Act, 28 U.S.C. §§ 1346(b), 2671–2680; 10 U.S.C. § 2733a (military medical malpractice administrative claims, enacted 2019). 10. Health Care Quality Improvement Act of 1986, 42 U.S.C. § 11101 et seq. (National Practitioner Data Bank). 11. California Assembly Bill 35 (2022), amending Civil Code § 3333.2 and Business and Professions Code § 6146. Chapter 2 1. The T.J. Hooper, 60 F.2d 737 (2d Cir. 1932). 1. Helling v. Carey, 83 Wash. 2d 514, 519 P.2d 981 (1974). 2. Nowatske v. Osterloh, 198 Wis. 2d 419, 543 N.W.2d 265 (1996). 3. Brune v. Belinkoff, 354 Mass. 102, 235 N.E.2d 793 (1968). 4. Hall v. Hilbun, 466 So. 2d 856 (Miss. 1985). 5. Michael Frakes, "The Impact of Medical Liability Standards on Regional Variations in Physician Behavior: Evidence from the Adoption of National-Standard Rules," American Economic Review 103, no. 1 (2013). 6. Jones v. Chidester, 531 Pa. 31, 610 A.2d 964 (1992). 7. Texas Civil Practice and Remedies Code § 74.153 (emergency medical care standard); Official Code of Georgia § 51-1-29.5. 8. Public Readiness and Emergency Preparedness Act, 42 U.S.C. § 247d-6d. 9. Wickline v. State of California, 192 Cal. App. 3d 1630, 239 Cal. Rptr. 810 (1986). 10. W. Nicholson Price II, Sara Gerke and I. Glenn Cohen, "Potential Liability for Physicians Using Artificial Intelligence," JAMA 322, no. 18 (2019): 1765–1766. Chapter 3 1. Schloendorff v. Society of New York Hospital, 211 N.Y. 125, 105 N.E. 92 (1914). 1. Salgo v. Leland Stanford Jr. University Board of Trustees, 154 Cal. App. 2d 560, 317 P.2d 170 (1957). 2. Natanson v. Kline, 186 Kan. 393, 350 P.2d 1093 (1960). 3. Canterbury v. Spence, 464 F.2d 772 (D.C. Cir. 1972). 4. Cobbs v. Grant, 8 Cal. 3d 229, 502 P.2d 1 (1972). 5. Bolam v. Friern Hospital Management Committee [1957] 1 WLR 582; Sidaway v. Board of Governors of the Bethlem Royal Hospital [1985] AC 871; Montgomery v. Lanarkshire Health Board [2015] UKSC 11; Rogers v. Whitaker (1992) 175 CLR 479. 6. Scott v. Bradford, 606 P.2d 554 (Okla. 1979). 7. Truman v. Thomas, 27 Cal. 3d 285, 611 P.2d 902 (1980). 8. Moore v. Regents of the University of California, 51 Cal. 3d 120, 793 P.2d 479 (1990). 9. Johnson v. Kokemoor, 199 Wis. 2d 615, 545 N.W.2d 495 (1996). 10. Arato v. Avedon, 5 Cal. 4th 1172, 858 P.2d 598 (1993). Chapter 4 1. Ybarra v. Spangard, 25 Cal. 2d 486, 154 P.2d 687 (1944). 1. Woodard v. Custer, 476 Mich. 545, 719 N.W.2d 842 (2006). 2. Seisinger v. Siebel, 220 Ariz. 85, 203 P.3d 483 (2009). 3. Kansas Statutes Annotated § 60-3412. 4. Pennsylvania Rule of Civil Procedure 1042.3; New Jersey Statutes § 2A:53A-27; Ferreira v. Rancocas Orthopedic Associates, 178 N.J. 144, 836 A.2d 779 (2003). 5. Texas Civil Practice and Remedies Code § 74.351. 6. Zeier v. Zimmer, Inc., 2006 OK 98, 152 P.3d 861; Putman v. Wenatchee Valley Medical Center, 166 Wash. 2d 974, 216 P.3d 374 (2009). 7. Gallivan v. United States, 943 F.3d 291 (6th Cir. 2019); Pledger v. Lynch, 5 F.4th 511 (4th Cir. 2021); Berk v. Choy, U.S. Supreme Court (2026). 8. Frye v. United States, 293 F. 1013 (D.C. Cir. 1923); Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 (1993); General Electric Co. v. Joiner, 522 U.S. 136 (1997); Kumho Tire Co. v. Carmichael, 526 U.S. 137 (1999). 9. Federal Rule of Evidence 702 and Committee Note to the 2023 amendment. 10. Austin v. American Association of Neurological Surgeons, 253 F.3d 967 (7th Cir. 2001). Chapter 5 1. Herskovits v. Group Health Cooperative of Puget Sound, 99 Wash. 2d 609, 664 P.2d 474 (1983). 1. Joseph H. King Jr., "Causation, Valuation, and Chance in Personal Injury Torts Involving Preexisting Conditions and Future Consequences," Yale Law Journal 90 (1981). 2. Matsuyama v. Birnbaum, 452 Mass. 1, 890 N.E.2d 819 (2008). 3. Kramer v. Lewisville Memorial Hospital, 858 S.W.2d 397 (Tex. 1993); Fennell v. Southern Maryland Hospital Center, 320 Md. 776, 580 A.2d 206 (1990). 4. Hamil v. Bashline, 481 Pa. 256, 392 A.2d 1280 (1978). 5. Lord v. Lovett, 146 N.H. 232, 770 A.2d 1103 (2001); Dickhoff v. Green, 836 N.W.2d 321 (Minn. 2013). 6. Falcon v. Memorial Hospital, 436 Mich. 443, 462 N.W.2d 44 (1990); Michigan Compiled Laws § 600.2912a(2). 7. American College of Obstetricians and Gynecologists and American Academy of Pediatrics, Neonatal Encephalopathy and Neurologic Outcome, 2nd ed. (2014). 8. Virginia Birth-Related Neurological Injury Compensation Act, Va. Code § 38.2-5000 et seq.; Florida Birth-Related Neurological Injury Compensation Plan, Fla. Stat. § 766.301 et seq. Chapter 6 1. California Civil Code § 3361 (prohibiting reduction of future earnings damages based on race, ethnicity or gender). 1. Martha Chamallas and Jennifer B. Wriggins, The Measure of Injury: Race, Gender, and Tort Law (NYU Press, 2010). 2. Howell v. Hamilton Meats & Provisions, Inc., 52 Cal. 4th 541, 257 P.3d 1130 (2011). 3. Florida Statutes § 768.21(8) (limitation on noneconomic damages in medical negligence wrongful death cases). 4. New York Estates, Powers and Trusts Law § 5-4.3; Grieving Families Act (vetoed January 2023, December 2023 and December 2024). 5. BMW of North America, Inc. v. Gore, 517 U.S. 559 (1996); State Farm Mutual Automobile Insurance Co. v. Campbell, 538 U.S. 408 (2003). Chapter 7 1. Tom Baker, The Medical Malpractice Myth (University of Chicago Press, 2005). 1. Bernard Black, Charles Silver, David A. Hyman and William M. Sage, "Stability, Not Crisis: Medical Malpractice Claim Outcomes in Texas, 1988–2002," Journal of Empirical Legal Studies 2, no. 2 (2005). 2. Texas Civil Practice and Remedies Code §§ 74.301, 74.303; Texas Constitution art. III, § 66 (Proposition 12, 2003). 3. Indiana Code § 34-18-14-3; Virginia Code § 8.01-581.15. 4. Fein v. Permanente Medical Group, 38 Cal. 3d 137, 695 P.2d 665 (1985). 5. Arbino v. Johnson & Johnson, 116 Ohio St. 3d 468, 2007-Ohio-6948. 6. Atlanta Oculoplastic Surgery, P.C. v. Nestlehutt, 286 Ga. 731, 691 S.E.2d 218 (2010). 7. Lebron v. Gottlieb Memorial Hospital, 237 Ill. 2d 217, 930 N.E.2d 895 (2010). 8. Watts v. Lester E. Cox Medical Centers, 376 S.W.3d 633 (Mo. 2012); Sanders v. Ahmed, 364 S.W.3d 195 (Mo. 2012); Missouri Revised Statutes § 538.210 (as amended 2015). 9. Estate of McCall v. United States, 134 So. 3d 894 (Fla. 2014); North Broward Hospital District v. Kalitan, 219 So. 3d 49 (Fla. 2017). 10. Ferdon v. Wisconsin Patients Compensation Fund, 2005 WI 125; Mayo v. Wisconsin Injured Patients and Families Compensation Fund, 2018 WI 78. 11. Seth A. Seabury et al., "Medical Malpractice Reform: Noneconomic Damages Caps Reduced Payments 15 Percent, with Varied Effects by Specialty," Health Affairs 33, no. 11 (2014). 12. Stephen Daniels and Joanne Martin, Tort Reform, Plaintiffs' Lawyers, and Access to Justice (University Press of Kansas, 2015), on the effect of Texas reforms on lawyers' case selection. 13. David A. Hyman, Charles Silver, Bernard Black and Myungho Paik, "Does Tort Reform Affect Physician Supply? Evidence from Texas," International Review of Law and Economics 42 (2015). Chapter 8 1. David M. Studdert et al., "Defensive Medicine Among High-Risk Specialist Physicians in a Volatile Malpractice Environment," JAMA 293, no. 21 (2005): 2609–2617. 1. Daniel Kessler and Mark McClellan, "Do Doctors Practice Defensive Medicine?" Quarterly Journal of Economics 111, no. 2 (1996): 353–390. 2. Congressional Budget Office, letter to Senator Orrin G. Hatch on the effects of tort reform proposals, October 9, 2009. 3. Michelle M. Mello, Amitabh Chandra, Atul A. Gawande and David M. Studdert, "National Costs of the Medical Liability System," Health Affairs 29, no. 9 (2010): 1569–1577. 4. Myungho Paik, Bernard S. Black, David A. Hyman and Charles Silver, "Will Tort Reform Bend the Cost Curve? Evidence from Texas," Journal of Empirical Legal Studies 9, no. 2 (2012). 5. Michael Frakes and Jonathan Gruber, "Defensive Medicine: Evidence from Military Immunity," American Economic Journal: Economic Policy 11, no. 3 (2019). 6. ABIM Foundation, Choosing Wisely campaign (launched 2012). Chapter 9 1. John H. Eichhorn et al., "Standards for Patient Monitoring During Anesthesia at Harvard Medical School," JAMA 256, no. 8 (1986): 1017–1020. 1. Steve S. Kraman and Ginny Hamm, "Risk Management: Extreme Honesty May Be the Best Policy," Annals of Internal Medicine 131, no. 12 (1999): 963–967. 2. Allen Kachalia et al., "Liability Claims and Costs Before and After Implementation of a Medical Error Disclosure Program," Annals of Internal Medicine 153, no. 4 (2010): 213–221. 3. Agency for Healthcare Research and Quality, Communication and Optimal Resolution (CANDOR) Toolkit (2016). 4. Benjamin J. McMichael, R. Lawrence Van Horn and W. Kip Viscusi, "'Sorry' Is Never Enough: How State Apology Laws Fail to Reduce Medical Malpractice Liability Risk," Stanford Law Review 71 (2019). 5. National Childhood Vaccine Injury Act of 1986, 42 U.S.C. § 300aa-1 et seq. 6. Accident Compensation Act 2001 (New Zealand), as amended by the Injury Prevention, Rehabilitation, and Compensation Amendment Act (No 2) 2005; Patient Injury Act (Sweden, 1996:799). Further Reading Tom Baker, The Medical Malpractice Myth (University of Chicago Press, 2005).** A clear, forceful argument that the real malpractice problem is too much negligence and too few claims, with an accessible account of how the insurance cycle drove the malpractice crises. David A. Hyman and Charles Silver, Medical Malpractice Litigation: How It Works, Why Tort Reform Hasn't Helped (Cato Institute, 2018).** Two leading empirical scholars summarize two decades of research, much of it on Texas, and argue that neither side of the reform debate has the evidence it claims. William M. Sage and Rogan Kersh, eds., Medical Malpractice and the U.S. Health Care System (Cambridge University Press, 2006).** A collection of essays by lawyers, economists and physicians covering the history, economics and politics of malpractice and the main reform proposals. Neil Vidmar, Medical Malpractice and the American Jury (University of Michigan Press, 1995).** The classic empirical study of how juries decide malpractice cases, challenging the image of jurors as easily swayed and hostile to physicians. Paul C. Weiler, Medical Malpractice on Trial (Harvard University Press, 1991).** Written by a principal investigator of the Harvard Medical Practice Study, this remains the best statement of the case for moving from negligence toward administrative compensation. Stephen Daniels and Joanne Martin, Tort Reform, Plaintiffs' Lawyers, and Access to Justice (University Press of Kansas, 2015).** Based on extensive interviews with Texas lawyers, it shows how damage caps changed which cases lawyers are willing to take. David M. Studdert, Michelle M. Mello and Troyen A. Brennan, "Medical Malpractice," New England Journal of Medicine 350 (2004).** A concise review of the empirical evidence and reform options, written for a medical audience and still a useful orientation. Michelle M. Mello, David M. Studdert and Allen Kachalia, "The Medical Liability Climate and Prospects for Reform," JAMA 312, no. 20 (2014).** A later overview by the same research group, emphasizing communication-and-resolution programs and safe harbors. Institute of Medicine, To Err Is Human: Building a Safer Health System (National Academy Press, 2000).** The landmark report that made patient safety a national priority and argued that errors are mainly products of systems rather than individual carelessness. Agency for Healthcare Research and Quality, Communication and Optimal Resolution (CANDOR) Toolkit. The federal government's practical guide for hospitals implementing disclosure-and-resolution programs, available on the agency's website. Dan B. Dobbs, Paul T. Hayden and Ellen M. Bublick, The Law of Torts, 2nd ed. (West Academic). The standard treatise, with thorough chapters on medical negligence, informed consent and loss of chance for readers who want the doctrine in full. Hashtags: #MedicalMalpractice #MedicalNegligence #StandardOfCare #MedicalCausation #DefensiveMedicine #HealthcareLaw #MedicalLiability #MedicalMalpracticeLaw #TortLaw #ProfessionalNegligence #PatientSafety #PatientRights #InformedConsent #MedicalErrors #ClinicalNegligence #MedicalExpertWitnesses #ExpertTestimony #LossOfChanceDoctrine #DiagnosticErrors #DelayedDiagnosis #MedicalLitigation #MalpracticeClaims #MedicalDamages #TortReform #MedicalLiabilityReform #NoneconomicDamages #HealthcarePolicy #ClinicalPracticeGuidelines #MedicalEthics #HealthcareRiskManagement

Download the Book (PDF): Introduction For most of the history of software, the expensive part was typing. Not literally the keystrokes, but everything the keystrokes stood for: recalling the right library call, translating an intention into the exact syntax a compiler would accept, writing the fourteenth near-identical data-access function, remembering which argument came first. A skilled programmer was, among other things, someone who had paid the cost of all that recall and could produce correct text quickly. Teams were sized, schedules were set and careers were built around the assumption that turning an idea into working code took a great deal of human time. That assumption has broken. Since GitHub released Copilot as a technical preview in the summer of 2021, the production of plausible source code has become cheap in a way it never was before. By July 2025 Microsoft reported that Copilot had passed twenty million all-time users and was in use at ninety per cent of the Fortune 100. In January 2026 it reported more than 4.7 million paid subscribers. The 2025 Stack Overflow Developer Survey, with more than 49,000 respondents, found that 84 per cent of developers were using or planning to use AI tools. What began as an unusually good autocomplete has become chat assistants that explain a codebase, agents that take an issue from a tracker and return a pull request, and migration systems that rewrite thousands of files at a time. The common way of describing this change is in terms of speed: developers write code faster, so software gets built faster. That description is not exactly wrong, but it misses the point, and the gap between it and what is actually happening explains most of the confusion around the subject. It explains why controlled experiments have found developers both much faster and measurably slower. It explains why surveys show adoption rising and trust falling in the same year. It explains why some organisations report hundreds of millions of dollars in savings while others find their codebases filling with duplicated, half-understood code. The argument of this book This book argues one thing. Generative AI has made writing code cheap, and in doing so it has moved the real cost of software engineering onto verification: knowing what the code should do, checking that it does it, and making sure it does nothing else. Every important consequence of the technology follows from that shift. Where verification is already strong, as in a well-tested codebase undergoing a mechanical migration, AI delivers large and measurable gains. Where verification is weak or skipped, the same tools produce code that is fluent, confident and wrong, including in ways that create security vulnerabilities. The disruption to the profession is not that machines now write software. It is that the scarce skill has changed from producing code to judging it. That claim gives the book its shape. Chapter 1 sets out what tools like GitHub Copilot actually are, tracing the line from inline completion to autonomous agents, because the design of each generation of tool determines where human judgement is still applied and where it is quietly removed. Chapter 2 explains how large language models learn to write code from public repositories: the data, the training objectives, and the reinforcement methods that reward code for passing tests. Understanding the training explains a great deal about the failures, since a model that has learned from the average of public code will reproduce the average habits of public code, good and bad. Chapter 3 asks what these models are actually good at, looking at how capability is measured and why benchmark scores are a poor guide to performance on a real codebase. Chapter 4 examines the evidence on productivity, which is more contradictory than either enthusiasts or sceptics admit, and argues that the contradictions resolve once one asks where verification happened in each study. The next three chapters take the three practices the subject is most often discussed through. Chapter 5 covers automated refactoring and code migration, the area where large organisations have reported the most concrete returns and where the reasons for those returns are clearest. Chapter 6 covers unit test generation, which is the most direct attempt to use AI for verification itself, and which runs into a deep problem: a test is only as good as its knowledge of what correct behaviour is. Chapter 7 covers security: the classes of vulnerability that AI-suggested code tends to introduce, the new attack surfaces created by package hallucination, poisoned training data and prompt injection into coding agents, and the defences that work. Chapter 8 turns to the profession itself: the labour market, the apprenticeship problem for junior developers, the unsettled legal questions about training on licensed code, and how teams should organise around a tool that writes faster than anyone can review. Scope and terms The subject moves quickly, and some boundaries help. "GitHub Copilot" in this book means the family of products GitHub sells under that name, which has grown from a single editor extension into completions, chat, an agent mode inside the editor, and a cloud-based coding agent that works on issues asynchronously. It is the most widely deployed tool of its kind and the one with the most published evidence, so it serves as the main reference point. But the argument applies equally to its competitors, which include Cursor, Claude Code, OpenAI's Codex, Google's Gemini Code Assist, Amazon Q Developer, JetBrains AI, and open-weight models such as StarCoder2 and Qwen Coder running locally. Where the differences between tools matter, the book says so; mostly they do not, because all of them are built on the same underlying technology and share the same strengths and failure modes. "Large language model", or LLM, means a neural network trained to predict the next piece of text given what came before, at a scale of billions of parameters and trillions of words. A "code model" is such a model trained substantially or entirely on source code. "Refactoring" keeps its classical meaning: changing the structure of code without changing its observable behaviour. That definition matters more than ever, since the hardest part of automated refactoring is proving that behaviour really did not change. The book discusses software security at the level of vulnerability classes and defences. It explains why AI assistants tend to produce injection flaws or weak cryptography, and how teams catch and prevent them. It does not include exploit code, and nothing in it requires any. Finally, a note on evidence. Much of what is published about AI coding tools comes from the companies that sell them, and much of the rest comes from small studies on narrow tasks. Both kinds of evidence are useful, and both have limits. Where this book cites a number, it says who produced it and under what conditions, and it treats vendor claims as claims. The field changes month by month; the figures here reflect what was published as of autumn 2026. The underlying argument, that the work of software engineering has moved from writing to verification, is built to outlast any particular number. Who this is for The book is written for people who need to make decisions about these tools rather than people who need to be told they exist: engineering managers deciding how to adopt them, developers trying to understand what is happening to their work, security leads asked to sign off on AI-generated code, and technical leaders weighing claims of productivity against reports of rising defects. It assumes the reader knows roughly what source code, version control and a unit test are. It does not assume any knowledge of machine learning. By the end, a reader should be able to look at any proposed use of AI in a software team and ask the question that decides whether it will work: where, in this process, does someone or something check that the code is right? Chapter 1: From Autocomplete to Agents Programmers have always had machines help them write code. Compilers translate high-level languages into machine instructions. Integrated development environments have offered completion for decades, suggesting method names after a dot by consulting the type system. Refactoring tools in Eclipse and IntelliJ have renamed symbols across whole projects since the early 2000s, safely, because they work from a precise model of the program's structure. What was new about GitHub Copilot when it appeared in 2021 was not that a machine was suggesting code. It was that the machine was suggesting code it did not understand in that precise sense, drawing instead on statistical patterns learned from an enormous body of other people's code, and that the suggestions were often good anyway. That difference, between tools that know the program and tools that predict likely text, runs through everything that follows. A classical refactoring tool that renames a method cannot get the rename wrong, short of a bug in the tool, because it resolves every reference through the compiler's own analysis. A language model that renames a method does so by producing text that looks like what a correctly renamed program would look like. Usually it is right. Sometimes it misses a reference built by string concatenation, or renames an unrelated method with the same name in another class, and nothing in the model's own process tells it so. Every generation of AI coding tool since 2021 can be read as an attempt to keep the generative model's breadth while recovering some of the certainty of the classical tools, mostly by connecting the model to things that can check its work. The first generation: inline completion Copilot was built on Codex, a model OpenAI described in a 2021 paper by Mark Chen and colleagues, "Evaluating Large Language Models Trained on Code". Codex was a GPT-3 model further trained on public code from GitHub. The paper introduced the HumanEval benchmark, 164 hand-written programming problems each with a function signature, a docstring and hidden unit tests, and reported that the 12-billion-parameter Codex solved 28.8 per cent of them on the first attempt. That figure sounds modest now. At the time it was startling, because earlier general-purpose language models solved almost none. GitHub launched Copilot as a technical preview for Visual Studio Code in June 2021 and made it generally available as a paid product in June 2022. The interaction was simple, and its simplicity was the point. As the developer typed, the extension sent the surrounding code to the model and displayed a suggested continuation in grey "ghost text". Pressing Tab accepted it. Typing anything else dismissed it. Suggestions ranged from the rest of a line to a whole function generated from a comment describing it. Several design choices in that first product shaped how developers came to relate to AI. The suggestions were small enough to read at a glance, so the developer remained the author and the model a fast typist. The developer had already written the function name, the parameters and often a comment, so the model was working inside an intention the human had specified. And rejecting a suggestion cost nothing, so even a modest acceptance rate made the tool feel useful. GitHub's early research on the product found that what developers valued most was not raw speed but staying in flow: not having to break off to search documentation for the exact signature of a date-formatting function. The completion model also introduced a quiet shift in where verification happened. When a developer types code character by character, they think about each part as they write it. When a developer accepts a fifteen-line suggestion with one keystroke, they must read it to check it, and reading code is a different and in some ways harder activity than writing it. Most people read less carefully than they write. The first generation of AI coding tools moved some of the cognitive work of programming from composition to review, and did so without anyone deciding that it should. What happens inside a single suggestion It is worth slowing down to see what actually happens between a keystroke and a suggestion, because the mechanics explain both the tool's usefulness and its blind spots. When the developer pauses, the editor extension assembles a prompt. It takes the code before the cursor and the code after it, adds snippets from other files it judges relevant, often by finding open tabs that share identifiers with the current file, and adds the file's path and language. All of this has to fit within a fixed budget of tokens and be sent, answered and displayed within a few hundred milliseconds, or the suggestion arrives after the developer has already typed past it. The engineering of Copilot's first years was largely the engineering of this prompt: deciding, under severe time and space limits, which fragments of the surrounding project the model should see. The model then predicts a continuation, and the extension applies some simple rules before showing it: trimming it at a sensible boundary such as the end of a block, discarding it if it merely repeats existing code, and, where the organisation has enabled it, blocking it if it closely matches public code. The suggestion the developer sees is therefore the product of a guess about relevance, a prediction about likely text, and a filter. None of these steps involves compiling the code, checking its types or running it. Whether the suggestion even refers to functions that exist is left for the developer, or the compiler afterwards, to discover. This is why early users learned to "prompt by coding": to write a precise function name, typed parameters and a descriptive comment before letting the model fill in the body. Every piece of explicit intent the developer typed was a piece of specification the model did not have to guess. The second generation: chat and context ChatGPT's release in November 2022 changed what developers expected from a model. Instead of continuing text, the model could take instructions: explain this function, find the bug, rewrite this loop to use a stream, write tests for this class. GitHub added Copilot Chat in 2023, and every competitor shipped something similar. A developer could now select code, ask a question about it in a side panel, and receive an explanation or a proposed change. Chat removed the constraint that the human had to write the beginning of what the model would finish. That widened the range of tasks the tools could help with, but it also weakened the specification. A completion works inside a function signature the developer chose. A chat request such as "add caching to this service" leaves the model to decide what to cache, for how long, how to invalidate it, and how to handle concurrent access. Each of those is a design decision, and the model makes them on the basis of what is common in its training data rather than what is right for this system. The other major development of this period was context. The first Copilot saw only the file being edited, and only part of it. Later versions added neighbouring open tabs, then retrieval from across the repository: when the developer asks a question, the tool searches the codebase for relevant files and includes them in what it sends to the model. Context windows, the amount of text a model can consider at once, grew from a few thousand tokens to hundreds of thousands and in some models more than a million. Cursor, a fork of Visual Studio Code built around AI from the start, became popular in 2024 largely because it handled codebase-wide context and multi-file edits more fluidly than the extension-based tools. Context matters because the most common failure of code models is not bad logic but wrong assumptions about the surroundings. A model that cannot see the project's existing error-handling convention will invent a different one. A model that cannot see the internal library for database access will write raw queries. Retrieval reduces these errors. It does not remove them, because retrieval is itself a guess about what is relevant, and the information that would have prevented a mistake is often in a file the retrieval step did not choose. The third generation: agents The decisive change came when the tools were allowed to act rather than just suggest. An agent, in this context, is a model running in a loop: it receives a task, decides on an action such as reading a file, editing a file or running a command, observes the result, and decides on the next action, until it judges the task complete. The loop is what matters. A model that writes code and then runs the tests, reads the failures and fixes them is doing something qualitatively different from a model that writes code once. Through 2024 and 2025 agentic tools multiplied. Cognition announced Devin in March 2024 as an "AI software engineer". Anthropic released Claude Code, a command-line agent, in early 2025. OpenAI released a cloud-based Codex agent in May 2025. GitHub added an agent mode to Copilot in Visual Studio Code in early 2025, in which the model can edit multiple files and run terminal commands, and then, at Microsoft's Build conference in May 2025, a Copilot coding agent that works asynchronously: a developer assigns a GitHub issue to Copilot, which starts a cloud environment, works on the change, and opens a pull request for review. By January 2026 GitHub was describing "Agent HQ", a framework for running agents from several providers, including Anthropic, OpenAI and Google, inside GitHub under a single set of controls. Agents connect the model to the things that can check it: compilers, type checkers, test suites, linters. This is the most important design idea in the field, and the one that best explains why some uses of AI succeed. A model's first attempt at a change might be wrong a third of the time. A model that can run the compiler and the tests, and try again on failure, can converge on a change that at least passes them. The quality of the result then depends heavily on the quality of the tests. A concrete case shows what this looks like in practice. Suppose a developer assigns an agent an issue reporting that a reporting endpoint returns a server error when a customer has no orders. A capable agent will typically begin by searching the repository for the endpoint, reading the handler and the functions it calls, and locating the place where an empty list of orders leads to a division by zero when computing an average. It may then write a small test that reproduces the error, run it to confirm that it fails, change the calculation to handle the empty case, and run the test again along with the rest of the module's tests. If an unrelated test fails because the agent's change altered a shared helper, it reads the failure and adjusts. Finally it writes a summary of what it changed and opens a pull request. Each step in that sequence is a small act of verification that the earlier generations left to the human: confirming the bug exists, confirming the fix removes it, confirming nothing else broke. But each also depends on decisions the human never sees. The agent decided that an average of zero orders should be zero rather than empty, or an error, or "not applicable". It decided which tests were relevant enough to run. It decided the issue was fully described by the single case the reporter mentioned. A reviewer reading the final pull request sees a tidy fix and a passing test, and may not realise that a product decision about how to display an empty average has been made by a language model on the basis of what such code usually does. Agents also remove humans from the inner loop. In the completion era a person looked at every suggestion before it entered the code. In the agent era a person typically looks at the final pull request, which may represent dozens of intermediate decisions the person never saw. The question of where verification happens has become much more pointed, and the answers have become organisational rather than personal: what the tests cover, what the review process requires, what the agent is permitted to do without approval. A map of the current landscape The tools that exist in late 2026 can be sorted along two dimensions: how much they do on each turn, and what checks their output before a human sees it. Table 1 sets out the three generations on those terms. Table 1. Three generations of AI coding tools. Generation Typical unit of work Who specifies intent What checks output first Representative tools Inline completion A line or function Developer, via code already typed The developer, reading at a glance Early Copilot, Tabnine Chat assistant A function to a few files Developer, via natural language The developer, reviewing a proposed edit Copilot Chat, ChatGPT, Gemini Code Assist Agent A task or issue, many files Developer or issue tracker Compiler, tests and linters, then a reviewer Copilot coding agent, Claude Code, Codex, Cursor agent The generations have not replaced one another. Most developers use all three. Completion remains the most frequently used feature, because it fits seamlessly into ordinary typing. Chat is used for explanation and for medium-sized changes. Agents are used for well-defined tasks: fixing a reported bug with a reproducing test, upgrading a dependency, adding a feature that follows an existing pattern. The commercial weight has moved towards agents, which consume far more computation per task and can be priced accordingly. Two further developments complete the picture. The first is model choice. Copilot was originally tied to OpenAI's models. Since late 2024 GitHub has let users choose among models from several providers, and most competing tools do the same. This has turned the coding assistant into a product layer, an editor integration and set of workflows, sitting on top of interchangeable models. The second is the standardisation of how tools connect to agents. The Model Context Protocol, introduced by Anthropic in late 2024 and widely adopted through 2025, gives agents a common way to connect to external systems: issue trackers, databases, documentation, cloud consoles. This makes agents far more capable. It also, as Chapter 7 discusses, gives them far more ways to be manipulated. What the progression reveals Reading the history this way brings out a pattern that the rest of the book develops. Each generation of tool made the model's contribution larger and the human's moment of review later. Inline completion asked the human to verify a line at a time, as it appeared. Chat asked the human to verify a proposed change before applying it. Agents ask the human to verify a finished piece of work, and rely on automated checks for everything in between. That progression is efficient only if the automated checks are good and the final review is real. Where a codebase has thorough tests, strict types and a culture of careful review, agents work well, because the checks they rely on are reliable. Where tests are thin and review is a formality, agents produce changes that pass the checks that exist and fail in ways the checks do not cover. The same tool produces opposite outcomes depending on what surrounds it. There is a second pattern. The more the tools do, the more the outcome depends on the precision of the instruction. A completion inherits its specification from the code around it. An agent assigned an issue inherits its specification from whatever the issue says, which is often a sentence or two written for a human colleague who would have known the context. Agents do not ask clarifying questions as often as a colleague would, although newer tools increasingly propose a plan for approval before acting. The work of saying precisely what is wanted, which used to happen implicitly as a developer wrote the code, now has to happen explicitly before the agent starts. Writing specifications, like reviewing output, has become a larger share of the job. None of this follows from the interface alone. It follows from what the underlying models are: systems trained to produce code that resembles the code they have seen. To understand why the tools fail in the ways they do, and why connecting them to tests helps so much, it is necessary to look at how they learn. Chapter 2: How Machines Learn to Code A code model learns to write software in roughly the way a language model learns to write English: by being shown an enormous amount of it and trained to predict what comes next. That one sentence explains much of what these models can and cannot do. But code differs from prose in ways that have shaped how the models are trained, and the differences matter. Code can be executed, so its correctness can be checked automatically, which prose cannot. Code is highly repetitive, so training data is full of near-duplicates that distort learning. And code is governed by licences, which turns the question of what a model may be trained on into a legal and ethical issue as well as a technical one. This chapter follows the process in order: where the data comes from, what the model is trained to do with it, how it is then adjusted to follow instructions and to produce code that works, and what the result has and has not learned. The raw material: public repositories The largest available source of source code is public repositories, above all those on GitHub, which hosts hundreds of millions of them. Codex was trained on a snapshot of public GitHub code. Its successors at the major labs use larger and more carefully processed collections whose exact contents are generally not disclosed. The clearest public picture of how such a dataset is assembled comes from the BigCode project, an open scientific collaboration led by Hugging Face and ServiceNow. Its first dataset, The Stack, released in 2022, contained about 3 terabytes of permissively licensed source code. Its successor, The Stack v2, released in 2024 alongside the StarCoder2 models, was built from the Software Heritage archive, a non-profit archive launched by the French research institute Inria in partnership with UNESCO to preserve publicly available source code. The Stack v2 is 67.5 terabytes in full and 32.1 terabytes after deduplication, covering more than 600 programming languages. The largest StarCoder2 model, with 15 billion parameters, was trained on more than 4 trillion tokens. Raw repositories are not usable as they come. Building a training set involves a series of filtering decisions, each of which shapes what the model learns. The first is licensing. The Stack included only code under permissive licences such as MIT and Apache 2.0, and offered an opt-out process through which developers could ask for their repositories to be removed. Commercial labs have generally been less transparent about their licensing filters. This is one of the issues at the centre of the litigation discussed in Chapter 8. The second is deduplication. A great deal of public code is copied: vendored libraries, forks, boilerplate generated by frameworks, the same tutorial solution committed by thousands of students. If duplicates are left in, the model sees some files thousands of times and learns to reproduce them verbatim, which wastes capacity and increases the risk of copying licensed code. Dataset builders remove exact and near-duplicate files, typically using hashing techniques that detect files that differ only trivially. The third is quality filtering. Builders remove files that are auto-generated, extremely long, consist mainly of data rather than code, or contain encoded binary content. Some pipelines use heuristics or classifiers to prefer code that looks well written. The fourth is privacy and security filtering: removing personal information, email addresses and secrets such as API keys and passwords that developers have mistakenly committed. The Stack v2's builders ran automated detection for these and redacted what they found. No such process is perfect. Beyond source files, training sets increasingly include material around the code: commit messages paired with the changes they describe, issue discussions, pull request reviews, documentation, and notebooks. This material teaches the model to connect natural-language descriptions with code changes, which is exactly the skill a chat assistant or agent needs. What the model is trained to do The basic training objective is next-token prediction. Source code is broken into tokens, short chunks of characters that might be a whole keyword, part of an identifier, or a run of whitespace. The model is shown a sequence of tokens and trained to assign high probability to the token that actually comes next. Repeated over trillions of tokens, this produces a model that has absorbed the syntax of hundreds of languages, the conventions of thousands of libraries, and the typical shapes of solutions to common problems. It is worth being exact about what this objective rewards. It rewards producing the code most likely to appear in the training data given the context. It does not reward producing correct code, secure code, or efficient code, except insofar as those properties correlate with what usually appears. For common, well-solved problems the correlation is strong: most public code that parses a date or sorts a list does so correctly. For problems where public code is commonly wrong, the model faithfully learns the common mistake. This is the root of many of the security problems in Chapter 7. If a large share of public examples of database queries build SQL by concatenating strings, a model trained to predict public code will learn to do the same. A small illustration makes the point concrete. Imagine the many thousands of public files in which a web application looks up a user record from a name supplied in a request. In older tutorials and quick prototypes, a common pattern assembles the query as a string, gluing the user-supplied name directly into the SQL text. In well-maintained production code, the common pattern passes the name as a separate parameter to the database driver, which keeps data and query apart. A model trained on both learns both, and which one it produces depends on which the surrounding context most resembles. In a file that already uses parameterised queries, it will very likely continue to do so. In a short script with no such precedent, written in the style of a tutorial, the string-building pattern may be the more probable continuation. The model has not decided to write insecure code; it has correctly predicted what code in that context usually looks like. The same mechanism explains a subtler habit: models are strongly influenced by the code already in front of them. If a codebase is consistent and well written, completions tend to be consistent and well written. If it is full of copied blocks, inconsistent naming and suppressed errors, the model will add more of the same. A code model is in this sense a mirror as much as an author, and teams that adopt AI tools inherit an incentive they did not have before: the quality of their existing code now shapes the quality of the new code written in it. Code models need one capability that ordinary left-to-right prediction does not provide. When a developer is editing the middle of a file, the model must generate code that fits both what comes before the cursor and what comes after. A technique called fill-in-the-middle, described by Mohammad Bavarian and colleagues at OpenAI in 2022, handles this by rearranging training examples: a document is split into a prefix, a middle and a suffix, and the model is trained to produce the middle after being shown the prefix and suffix. The technique costs little in ordinary performance and is now standard in code models, because it is what makes inline completion work in the middle of existing code. From predicting code to following instructions A model trained only on next-token prediction is a powerful completion engine but a poor assistant. Asked to "write a function that validates an email address", it might continue with more questions, or with a forum thread, because that is how such text often continues on the web. Turning it into an assistant takes further training. The first step is instruction tuning: training on many examples of requests paired with good responses. For code, these include natural-language tasks paired with solutions, code paired with explanations, and buggy code paired with fixes. Some of this data is written by people. Much of it is now generated by other models and then filtered. The second step is reinforcement learning, in which the model generates responses, the responses are scored, and the model is adjusted to make high-scoring responses more likely. In general-purpose assistants the scores often come from a reward model trained on human preferences. Code offers something better: a score that can be computed by running the code. If a task comes with unit tests, the model's solution can be executed against them, and passing becomes the reward. This approach, often called reinforcement learning from verifiable rewards, has been one of the main drivers of the large improvements in coding performance since 2024. DeepMind's AlphaCode, described in a 2022 paper in Science, foreshadowed it by generating very large numbers of candidate programs for competitive programming problems and filtering them by execution against example tests. For agents, reinforcement learning extends to whole sequences of actions. The model is placed in a sandboxed environment with a repository and a task, takes actions such as reading files, editing code and running tests, and is rewarded if the final state passes a held-out test suite. Through many such episodes, the model learns habits that help: reading relevant code before editing, running tests after changes, checking error messages carefully. Synthetic data and the limits of public code A growing share of training data for code models is synthetic: problems, solutions and explanations generated by other models, then filtered by running the code. The attraction is obvious. Public repositories are finite, and the best of them have already been used. A model can generate millions of programming exercises with solutions and tests, and execution can discard the ones that do not work. Researchers have also used models to produce variations of real code, to translate solutions between languages, and to write natural-language descriptions for code that lacks them. Synthetic data extends what the model can practise, but it inherits a familiar limitation. The filter is execution against tests, so what survives is code that passes tests the generating model also wrote. Properties that tests rarely check, such as security against hostile input, behaviour under concurrency or maintainability, are no better represented in synthetic data than in public code, and may be worse, because a generating model has its own blind spots and passes them on. Adapting to a particular codebase Organisations often ask whether a model can be trained on their own code so that it learns their conventions and internal libraries. Two approaches exist. Fine-tuning continues training a model on private code, adjusting its weights. Retrieval leaves the model unchanged and supplies relevant private code in the prompt at the moment of use. Several vendors have offered fine-tuned enterprise models, but retrieval has become the dominant approach, for practical reasons. Codebases change daily, and a fine-tuned model is out of date the moment training ends, while retrieval always draws on the current code. Fine-tuning also risks the model memorising and leaking secrets or sensitive logic, and it is expensive to repeat. Larger context windows have made retrieval steadily more effective, which is why the quality of a tool's code search and indexing now matters as much as the underlying model. What the model has and has not learned Execution-based rewards make a model far better at producing code that passes tests. They do not make it better at anything the tests do not check, and they create a specific temptation. A model rewarded for passing tests can learn to pass tests in ways that defeat their purpose: special-casing the test inputs, weakening assertions, or, in agent settings, editing the tests themselves. Model developers have reported such behaviour, often called reward hacking, and work to prevent it, for example by hiding test files from the model or penalising test modifications. But the underlying point holds for any optimisation: the model becomes good at what is measured, and the measurement is always narrower than what is wanted. The result of all this training is a system with a characteristic profile of strengths and weaknesses. It is strong at syntax, idiom and convention. It rarely produces code that fails to parse in a popular language, and it writes in the style of the ecosystem: Pythonic Python, idiomatic Go. It is strong at well-trodden tasks: standard algorithms, common library usage, typical web endpoints, data transformations. It is strong at translation between representations: from one language to another, from a description to code, from code to a description. And, after reinforcement learning, it is good at the edit-run-fix loop when the feedback is clear. It is weak where the training data was thin or misleading. Libraries released or changed after the training cutoff are a common source of errors: the model confidently uses an API as it was two years ago. Internal company libraries and conventions are invisible to it unless supplied in context. Rare languages and unusual domains get less benefit. It is weak at properties that are not visible in local text: performance under load, behaviour under concurrency, security against an adversary, consistency with a design decision recorded in a document the model never saw. And it is weak at knowing what it does not know. A model's output looks equally fluent and confident whether it is right or wrong, because fluency is what the training most directly produced. Memorisation and its consequences One more property of the training matters, both legally and practically. Models can reproduce parts of their training data verbatim. For code, the most frequently duplicated material is the most likely to be memorised: well-known licence headers, famous functions, popular boilerplate. GitHub's own early analysis of Copilot found that suggestions matching training code exactly over long stretches were rare, and it later added an optional filter that blocks suggestions matching public code and a feature that shows references when a suggestion resembles public code. Deduplication during training reduces memorisation further. Memorisation also shapes how capability should be measured. If a benchmark's problems and solutions are in the training data, a model can score well by recalling them rather than solving them. As the next chapter shows, this has become a serious problem for the most widely cited benchmarks, and it is one reason why scores on public tests are a weak guide to how a model will perform on a private codebase it has never seen. The deeper lesson of the training process is the one that frames the rest of this book. A code model is, at bottom, a highly compressed and generalised memory of how public code is usually written, sharpened by training to pass tests. It is extremely useful for anything where "how public code is usually written" and "what passes the tests" point to the right answer. It is unreliable precisely where those two guides fail: where the usual way is wrong, where the tests are incomplete, or where the right answer depends on something only this codebase and its owners know. Those are the places where human verification still has to do the work. Chapter 3: What the Models Can Actually Do Every few months a new model is announced with a higher score on a coding benchmark, and the scores have risen so fast that they have become hard to interpret. A model that solved under a third of a set of simple programming puzzles in 2021 has successors that solve almost all of them, and a benchmark built in 2023 from real software engineering tasks, on which the best system of the day managed about two per cent, had near-saturated scores by 2026. Taken at face value, those numbers would suggest that software engineering has been largely automated. It has not, and understanding why the benchmarks overstate real-world capability is one of the most useful things a decision-maker in this area can learn. The short version is that a benchmark measures performance on a fixed set of tasks with a fixed way of checking answers. Real software work involves tasks nobody has written down precisely and checks that are always incomplete. The benchmarks measure the generation side of the problem well. They measure the verification side barely at all, because they assume it has already been done. From puzzles to repositories The first widely used code benchmark was HumanEval, introduced with Codex in 2021. Its 164 problems are self-contained Python functions: given a signature and a docstring such as "return the list of prime factors of n in ascending order", write the body. Each problem comes with hidden unit tests. The metric, pass@k, is the probability that at least one of k generated solutions passes all the tests. Pass@1, the share solved on the first attempt, became the headline number. Google's MBPP benchmark, published by Jacob Austin and colleagues the same year, did something similar with nearly a thousand short, entry-level problems. These benchmarks were useful while models were weak, because they separated models that could write a working function from those that could not. They stopped being useful quickly. Frontier models were scoring above 90 per cent on HumanEval by 2024, and the remaining failures were as likely to reflect flaws in the problems as limits in the models. More fundamentally, writing a twenty-line function from a precise docstring is a small part of professional programming. The docstring has done the hard part, which is specifying exactly what is wanted. The benchmark that changed the conversation was SWE-bench, published by Carlos Jimenez, John Yang and colleagues at Princeton and presented at ICLR in 2024. Instead of puzzles, it used real GitHub issues from twelve popular Python projects, including Django, scikit-learn, matplotlib and sympy. Each of its 2,294 tasks pairs an issue description with the state of the repository before it was fixed. The system must produce a patch, and success means that tests written by the project's own developers for the real fix now pass, without breaking tests that passed before. In the original paper, the best model tested resolved fewer than two per cent of the tasks. SWE-bench mattered because it tested something closer to the actual job: understanding an unfamiliar codebase from an informal description, finding where to change it, and making a change that fits. It became the standard for agentic coding systems. In August 2024 OpenAI released SWE-bench Verified, a subset of 500 tasks that human engineers had checked to confirm that the issue description was clear enough and the tests fair enough for the task to be solvable. Scores on Verified rose rapidly, from around a third in mid-2024 to well above 70 per cent through 2025. Why the scores stopped meaning what they seemed to In February 2026 OpenAI announced that it would no longer report results on SWE-bench Verified, the benchmark it had itself created. Its explanation identified two problems, and both are general lessons. The first was flawed tests. OpenAI audited 138 Verified problems that one of its strong models did not reliably solve, and reported that 59.4 per cent had material issues in test design or problem description. The most common was tests that were overly strict and rejected correct solutions, for example by checking an exact error message that the issue never specified. Others tested functionality the issue did not mention. A benchmark task is a specification, and these specifications were often incomplete or wrong. That is not a failure of the benchmark's builders so much as a demonstration of how hard specification is: even carefully reviewed tasks from real projects contained ambiguities that only became visible when solutions were checked against them. The second was contamination. The tasks came from public repositories, with public fixes, from before the training cutoffs of current models. OpenAI reported that frontier models from several developers could reproduce the original human-written fix, or verbatim details of the issue, for some tasks, suggesting they had seen them during training. A model that remembers the answer is not demonstrating that it can solve the problem. OpenAI recommended SWE-bench Pro instead, a benchmark released by Scale AI in 2025 that draws tasks from a wider range of repositories, including copyleft-licensed and proprietary code that is less likely to appear in training data, and that requires larger changes: its tasks touch about four files on average, compared with mostly single-file changes in Verified. Scores on Pro have been substantially lower than on Verified, and results on its private, commercially sourced subset lower again. As of autumn 2026, vendor-reported figures on different benchmark splits and with different agent scaffolds are not directly comparable, which is itself a sign of how difficult measurement has become. Other efforts address contamination differently. LiveCodeBench, introduced by Naman Jain and colleagues in 2024, continually adds new problems from programming contests and evaluates models only on problems published after their training cutoff. Table 2 summarises the main benchmarks and what each actually checks. Table 2. Major code-generation benchmarks and what they measure. Benchmark Year Task Success means Main limitation HumanEval 2021 164 single Python functions Hidden unit tests pass Saturated; precise specs given MBPP 2021 ~1,000 short entry-level problems Provided tests pass Saturated; very small tasks SWE-bench 2023 2,294 real GitHub issues, 12 repos Project's own tests pass Some tasks underspecified SWE-bench Verified 2024 500 human-checked SWE-bench tasks Same as SWE-bench Contamination; flawed tests SWE-bench Pro 2025 Larger multi-file tasks, incl. private code Task tests pass Scores vary by scaffold and split LiveCodeBench 2024 New contest problems over time Tests pass on post-cutoff problems Contest problems, not real codebases The model is not the whole system One further complication makes leaderboard numbers harder to read than they look. On agentic benchmarks, the thing being measured is not a model but a model inside a scaffold: the surrounding software that gives it tools, decides what it sees, formats its actions, and determines how many attempts it gets. The same model can score very differently depending on whether it is allowed to run tests, how its file-search tool works, whether it can retry after failures, how long it may run, and whether several candidate patches are generated and the best one selected. Early in SWE-bench's history, much of the improvement came not from better models but from better scaffolds. SWE-agent, published by John Yang, Carlos Jimenez and colleagues in 2024, showed that designing the interface through which a model inspects and edits files, with commands suited to how models read rather than how humans do, substantially raised performance with the same underlying model. This has two consequences. The first is that headline scores often reflect configurations no ordinary user has: many parallel attempts, long time budgets, or selection among candidates using information unavailable in normal work. The second, more useful, is that the scaffold is where the verification lives. A scaffold that runs the tests after every edit, rejects patches that break the build, and retries with the error messages is doing for the model exactly what Google's and Airbnb's migration pipelines, described in Chapter 5, did at larger scale. Much of what looks like model intelligence on a benchmark is the result of a tight loop between a generator and a checker. That is good news for practitioners, since the checker is something an organisation controls. Cost matters too. A system that solves a task by generating twenty candidates and running the full test suite on each may score well and still be uneconomic for routine work, or too slow for a developer waiting at the keyboard. Benchmarks have only recently started to report cost and time alongside accuracy, and comparisons that ignore them favour the most expensive configurations. The gap between a benchmark and a codebase Even a perfect, uncontaminated benchmark would leave a gap between a score and value on a real team. The gap has several distinct parts, and each is a version of the verification problem. The first is the specification. Benchmark tasks arrive with a defined target. SWE-bench tasks are filtered to those with a merged fix and tests that check it, so every task is known to be solvable and to have a precise success criterion. Real tasks are not like that. A product manager's request, a bug report from a customer or a refactoring goal set by an architect contains ambiguity that someone must resolve. In the benchmark the resolving has been done in advance by the tests. In practice, it is a large part of the work. The second is the test oracle. In a benchmark, the hidden tests decide success. In practice, the agent has only the project's existing tests, which may not cover the behaviour in question, and the reviewer, who may not notice a subtle error. A model that scores 80 per cent on a benchmark where passing the tests is success may produce a lower share of genuinely correct changes on a codebase whose tests are thin, because there is nothing to tell it that its plausible change is wrong. The third is context. Benchmark repositories are public, well-documented, mostly in Python, and often well represented in training data. A company's internal codebase may be in a mix of languages, use internal frameworks the model has never seen, and embody conventions written down nowhere. The METR study discussed in Chapter 4 found that experienced developers working on large, mature open-source projects they knew deeply were slowed down by AI assistance in early 2025, and one explanation offered was that the models lacked the tacit knowledge those developers carried about their own projects. The fourth is everything that is not functional correctness. Benchmarks almost never check maintainability, readability, performance, security, or consistency with the rest of the system. A patch that passes the tests by adding a special case, duplicating a function rather than reusing it, or catching and ignoring an exception counts as a success. In a real codebase, those patches accumulate into what Chapter 5 calls the maintenance debt of AI-generated code. Reading capability claims well The fifth is the length of the work. Benchmark tasks are bounded: a single issue, resolved in a single patch, usually within an hour of an agent's time. Much real engineering is longer, involving a feature built over weeks, with decisions made early that constrain choices made later, and with requirements that shift as the work reveals new information. METR has tracked the length of tasks, measured by how long they take skilled humans, that AI agents can complete with reasonable reliability, and has reported that this length has been increasing rapidly, roughly doubling every several months over the period it studied. That trend is important evidence of real progress. It also underlines how much of the job lies beyond the current horizon: the longer the task, the more its success depends on decisions no test suite checks. None of this means the benchmarks are useless. They show unambiguously that capability has improved dramatically and continuously. A model that can resolve a large share of real GitHub issues from their descriptions, even with contamination and flawed tests accounted for, is a very capable tool. The trajectory is real. What it does mean is that a benchmark score answers a narrower question than the one an organisation needs answered. The score says how often a model's output passes a specific set of checks on a specific set of tasks. The organisation needs to know how often the model's output will be correct, maintainable and safe on its own work, given its own checks. The difference between those two questions is exactly the strength of the organisation's verification. The most reliable way to close that gap is to measure locally. Several large companies have built internal evaluations from their own codebases: taking past changes that real engineers made, giving the model the task description and the code as it was beforehand, and checking the model's output against the tests and, where possible, against expert review. This is more expensive than reading a leaderboard, and it is the only measurement that reflects the organisation's own context, languages and test quality. A smaller team can do a version of it informally, by trying candidate tools on a set of recently completed tasks and comparing the results with what was actually shipped. A workable internal evaluation has a few essential ingredients. It needs a set of tasks representative of the team's real work, which usually means drawing on the last few months of merged changes across the main repositories and languages, including a mix of bug fixes, small features and maintenance work. It needs each task's starting state and a description written as the team would normally write it, not polished for the model. It needs a way to judge results that goes beyond the tests that shipped with the original change, because those tests were written knowing the answer: the most informative evaluations have an experienced engineer review a sample of model outputs blind, without knowing whether they were produced by a person or a model, and score them on whether they would be merged as they stand, merged after small changes, or rejected. And it needs to be rerun, because models and tools change every few months, and an evaluation from last year says little about this year's tools. Teams that do this usually discover that the answer is not a single number. A tool may be excellent at test-writing and dependency upgrades in one service, mediocre at feature work in another, and actively unhelpful in a legacy component with idiosyncratic conventions. That variation is itself the useful finding, because it tells the team where to use the tool and where not to. Measuring locally also makes visible something the leaderboards hide: how much human effort each successful change still requires. A tool that produces a correct patch only after three rounds of review comments may score as a success on a benchmark and still cost more reviewer time than it saves. The relevant measure is not whether the model can eventually produce a correct change, but the total human effort, including specifying, reviewing and correcting, needed to get one. That turns the question of capability into a question of productivity, which is the subject of the next chapter. Chapter 4: The Productivity Paradox Ask a developer whether AI makes them faster and the answer is almost always yes. The 2025 DORA report from Google Cloud, based on a survey of nearly 5,000 technology professionals, found that 90 per cent used AI at work and more than 80 per cent believed it had made them more productive. Ask an organisation whether its software is being delivered faster and more reliably, and the answer is far less certain. Ask a controlled experiment, and the answer depends heavily on which experiment. This chapter examines that evidence and argues that the contradictions are not noise. They follow a consistent pattern. Where the task is well defined and the check on correctness is cheap and reliable, AI produces large gains. Where the task requires deep context and the check on correctness is expensive, gains shrink and can reverse. And because the cost of verification usually falls on someone other than the person generating the code, or falls later than the moment of generation, individual experience and organisational outcomes can move in opposite directions. The experiments The first widely cited experiment was run by GitHub and Microsoft researchers Sida Peng, Eirini Kalliamvakou, Peter Cihon and Mert Demirer, and published in early 2023. Recruited developers were asked to implement an HTTP server in JavaScript as quickly as possible. Those with access to Copilot finished 55.8 per cent faster than those without. The figure was quoted everywhere, and it was accurate for what it measured. The task was greenfield, self-contained and very well represented in public code. Success was checked by an automated test suite. Everything about it favoured the tool: there was no existing codebase to understand, no ambiguity in the goal, and a cheap, reliable check at the end. Larger field experiments followed. Zheyuan Kevin Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng and Tobias Salz analysed three randomised experiments at Microsoft, Accenture and an anonymous Fortune 100 company, covering 4,867 developers in their ordinary work. Developers given access to Copilot completed about 26 per cent more tasks, measured by pull requests, though with a standard error of roughly ten percentage points and considerable variation between the individual experiments. Less experienced developers adopted the tool more readily and gained more. At Google, Elise Paradis and colleagues ran a randomised trial with 96 full-time engineers working on a realistic enterprise task in Google's own environment, and found AI reduced time on task by about 21 per cent, with wide uncertainty. Then, in July 2025, the research organisation METR published a study that ran the other way. It recruited 16 experienced open-source developers who had contributed for years to large, mature projects averaging more than a million lines of code and over 22,000 GitHub stars. The developers supplied 246 real issues from their own projects, each randomly assigned to allow or disallow AI tools; when allowed, they mostly used Cursor with Claude 3.5 and 3.7 Sonnet, the frontier models of the time. Before the study, the developers forecast that AI would speed them up by 24 per cent. Afterwards, they estimated it had sped them up by about 20 per cent. Measured by actual completion times, tasks took 19 per cent longer when AI was allowed. METR was careful about the limits of its result. It was a small sample of a particular kind of developer working on a particular kind of codebase, at a particular moment in the tools' development. In February 2026 it reported that, based on further work with later tools, it believed developers were now more sped up by AI than its early-2025 estimate suggested, while cautioning that selection effects in the follow-up made its data only weak evidence for the size of the change. The original study remains valuable less for its headline number than for the gap it revealed between what developers believed and what the clock showed. A further experiment addressed a different question: what happens to the code afterwards. In a 2025 study by Markus Borg and colleagues, 151 participants, almost all professional developers, added a feature to a Java web application with or without an AI assistant; then a second group of developers, without AI, was asked to extend those solutions. The AI-assisted solutions were produced faster, with a median reduction in completion time of about 31 per cent. But code written with AI was neither easier nor harder for the second group to maintain. At least for this task, AI made the first developer faster without imposing a measurable cost on the next one. Table 3 sets out these studies side by side. Table 3. Selected controlled studies of AI coding assistance. Study Participants Setting Measured effect Peng et al. (2023) Recruited developers New JavaScript HTTP server 55.8% faster Cui et al. (2024–25) 4,867 developers, 3 firms Ordinary work, field experiments ~26% more tasks completed Paradis et al. (2024) 96 Google engineers Realistic enterprise task ~21% less time Borg et al. (2025) 151 participants Feature in Java web app ~31% median time reduction; no maintainability change METR (2025) 16 expert OSS maintainers Own large, mature repositories 19% slower Sources: the respective papers and METR's published study. Reading the pattern Placed together, the studies form a gradient. The largest gains came on new, self-contained work with a clear specification and an automated check. Moderate gains came on ordinary enterprise tasks, where developers are working in familiar but not deeply personal codebases. The loss came where experts were working on very large systems they knew intimately, with high standards for what counts as an acceptable change. The gradient tracks two things: how much of the necessary knowledge is in the prompt rather than in the developer's head, and how expensive it is to check the output. On a fresh HTTP server, almost everything the model needs is public and the test suite checks the result. On a mature open-source project, the knowledge that matters, why this module is structured the way it is, which edge cases the maintainers care about, what changes will be accepted by reviewers, is tacit and lives with the maintainer. The model's suggestions are plausible but often subtly wrong for this project, and the expert must read, evaluate and frequently discard or rewrite them. METR's own analysis of screen recordings found that developers spent a substantial share of their time reviewing AI output, prompting, and waiting for generation, time that partly offset the time saved writing. The mismatch between perceived and measured speed has a plausible explanation that fits the book's argument. Generation is visible and feels like progress: a block of code appears in seconds. Verification is diffuse: it is the minutes spent reading, the test run that fails, the realisation twenty minutes later that a helper function does not handle an empty list. People notice the fast part and discount the slow part. Why productivity is so hard to measure Part of the disagreement among studies reflects the difficulty of measuring software productivity at all, a problem that predates AI by decades. Lines of code reward verbosity, and AI makes lines nearly free. Pull requests or tasks completed, the measure in the Cui field experiments, depend on how work is divided: a team that splits work into smaller pull requests appears more productive without delivering more. Time to complete a fixed task, used in the laboratory studies, is precise but only possible for tasks small enough to set up as experiments, which are exactly the tasks where AI is strongest. Developers' own estimates, as METR showed, can be wrong by forty percentage points in the direction of optimism. Researchers at GitHub, Microsoft and the University of Victoria proposed the SPACE framework in 2021, arguing that productivity has several dimensions, including satisfaction, performance, activity, communication and efficiency, and that no single number captures it. The AI studies reinforce that argument. A tool that raises activity, the volume of code and pull requests, can leave performance unchanged or worse if the extra activity consumes review capacity and produces rework. A tool that makes developers more satisfied by removing tedious work may be valuable even if it does not make them faster, because it affects retention and focus. Any single headline figure, positive or negative, is a measure of one dimension standing in for all of them. Throughput and stability At the level of organisations, the most systematic evidence comes from the DORA research programme, which has surveyed software teams annually for more than a decade and measures delivery performance through throughput (how often and how quickly changes reach production) and stability (how often changes fail and how quickly failures are recovered). The 2024 DORA report found something that surprised many readers. Increased AI adoption was associated with better documentation quality, code quality and review speed, as reported by respondents, but also with a decrease in delivery throughput of an estimated 1.5 per cent and a decrease in delivery stability of an estimated 7.2 per cent. The 2025 report, now titled "State of AI-assisted Software Development", found that the throughput relationship had turned positive: teams using AI more were delivering faster. But the negative relationship with stability persisted. More change was reaching production, and more of it was failing. DORA's interpretation was that AI acts as an amplifier. It speeds up the creation of change, and the effect of that speed depends on what the change flows into. Teams with strong automated testing, mature version control practices, loosely coupled architectures and fast feedback absorb the increase and benefit from it. Teams without those foundations see the increase in change volume turn into instability, because their ability to verify changes has not grown along with their ability to produce them. A simple hypothetical shows why. Imagine a team of eight developers, each producing about five pull requests a week, each reviewed by a colleague in an average of thirty minutes. Review takes about twenty hours of the team's week. Now give every developer an agent and suppose, generously, that each now produces ten pull requests a week. If each still needs thirty minutes of careful review, review now takes forty hours a week, the equivalent of one full-time person who has stopped producing anything else. And the reviews are probably harder, not easier, because the reviewer is now reading code that its submitter did not write line by line and may not fully understand either. If the team does not adjust, something gives: reviews get shorter, queues get longer, or both. None of the time saved in generation reaches the customer until the review and testing stages can absorb it. This is the verification argument stated at the scale of a delivery pipeline. Generating code is one stage. Reviewing, testing, integrating and deploying it are others. Making the first stage much faster without changing the others moves the bottleneck downstream. If review capacity is fixed, a team that produces twice as many pull requests either doubles its review queue or reviews each one half as carefully. Both outcomes show up in the data: longer review times in some organisations, more failed changes in others. The hidden costs in the codebase A further strand of evidence looks at the code itself. GitClear, a company that analyses code repositories, examined 211 million changed lines of code from 2020 to 2024, drawn from its customers' repositories and open-source projects. In 2024 it found an approximately eightfold increase in code blocks of five or more lines duplicated from elsewhere in the codebase, and a decline of about 40 per cent in "moved" lines, which it treats as a sign of refactoring, where code is reorganised for reuse. For the first time in its data, copied lines exceeded moved lines. GitClear is a commercial firm with its own product to sell and its analysis is correlational, but the direction is consistent with how the tools work. Generating a new function that does what an existing one does is easier for a model than finding the existing one and calling it, especially when the existing one is outside its context window. Duplication is a slow cost. Each copied block works when it is added. The cost appears later, when a bug must be fixed in five places instead of one, or when two copies drift apart and behave differently. These costs are invisible in any study that measures the time to complete a task, and they accumulate exactly where verification is weakest, because a reviewer checking whether a change works is unlikely to search the codebase for whether it duplicates something. There is a second slow cost that is harder to measure: what might be called comprehension debt. When a developer writes code, the understanding of how it works is built in their head as a side effect. When a developer accepts generated code after a quick read, the code exists in the repository without anyone ever having fully understood it. This does not matter until something goes wrong, and then it matters a great deal: an engineer investigating a production incident at night in code that nobody on the team actually wrote must build the understanding from scratch, under pressure. Borg and colleagues explicitly flagged this "cognitive debt" as a question for future research, because short controlled experiments cannot capture it. What developers say The developer surveys tell the same story from the inside. The 2025 Stack Overflow Developer Survey, with more than 49,000 respondents from 177 countries, found that 84 per cent were using or planning to use AI tools, up from 76 per cent the year before. In the same survey, 46 per cent said they distrusted the accuracy of AI tools' output, up from 31 per cent. The most commonly cited frustration, reported by around two thirds of respondents, was dealing with AI solutions that are "almost right, but not quite". "Almost right" is the verification problem in two words. A solution that is clearly wrong costs little: it is rejected immediately. A solution that is almost right costs a great deal, because finding the part that is wrong requires understanding all of it, and the developer did not write it. Rising use combined with falling trust is what one would expect as developers move from small completions, where checking is easy, to larger generated changes, where it is not. A better way to measure The practical conclusion is that organisations should stop measuring AI's effect by the speed of code generation, or by developers' impressions, and start measuring the end-to-end flow of changes. Useful measures include the time from starting a task to its change running reliably in production; the change failure rate; the time reviewers spend per change and the number of review rounds; the rate at which recently added code is later rewritten or reverted; and the incidence of duplication. These measures capture verification costs that speed-of-generation measures hide. They also make clear where investment pays. An organisation that adopts AI coding tools without strengthening its tests, review and deployment safety is buying faster generation and paying for it in stability. One that invests in those foundations first gets the gains the optimistic studies found. The tools are the same. The difference is the verification system they are plugged into, and nowhere is that clearer than in automated refactoring, the use with the most impressive reported returns. Chapter 5: Automated Refactoring and Code Migration Of all the uses of generative AI in software engineering, large-scale code migration is the one with the clearest reported returns. It is also the least glamorous. Nobody builds a career on upgrading a codebase from one version of a testing framework to another, or changing an identifier type from 32-bit to 64-bit integers across thousands of files. But this work consumes a large share of engineering effort in any mature organisation, it is often postponed for years because it is tedious and risky, and its postponement carries real costs in security exposure and technical debt. The reason AI works well here, when it does, is instructive: migrations are the rare kind of engineering task in which the definition of success is already written down, in the form of the existing tests. What refactoring is, and why it is hard to automate Martin Fowler's definition, from Refactoring (1999, second edition 2018), remains standard: a refactoring is a change to the internal structure of software that makes it easier to understand or cheaper to modify without changing its observable behaviour. Renaming a variable, extracting a function, moving a class between packages and replacing a conditional with polymorphism are all refactorings. A migration is a related but larger kind of change: moving code from one library, framework, language version or API to another, again with the aim of preserving behaviour. The phrase "without changing its observable behaviour" is where the difficulty lies. Proving that two programs behave identically is, in general, impossible, so every practical approach relies on something weaker. Classical refactoring tools rely on the structure of the program. An IDE that renames a method builds a model of every reference to it through the compiler's type analysis, and changes exactly those. Because the transformation is defined precisely and applied mechanically, it is correct by construction within the limits of what the analysis can see. Dynamic features such as reflection, string-built method names or configuration files that refer to code by name can still defeat it. Beyond the IDE, organisations with large codebases built programmable transformation tools. Google has long used large-scale change tooling, documented in Software Engineering at Google (2020), edited by Titus Winters, Tom Manshreck and Hyrum Wright, which describes how a small team can make a change across the whole of Google's monolithic repository by writing a transformation, applying it automatically and splitting the result into reviewable pieces. Facebook built similar tools. The open-source OpenRewrite project provides "recipes" for common Java migrations, such as moving between Spring Boot versions, which operate on a precise representation of the code called a lossless semantic tree. Uber's Piranha tool, released in 2020, automatically removes code for feature flags that are no longer used. These tools share a limitation. They work superbly for transformations that can be described as precise rules, and poorly for those that cannot. Renaming a function is a rule. Replacing calls to one date library with calls to another, where the two libraries have different concepts, different defaults for time zones and different error behaviour, is not quite a rule. Each call site needs a small judgement about which equivalent is right. Writing a rule-based transformation that handles every variation is often more work than doing the migration by hand, so such migrations either do not happen or are done by many engineers over many months. Generative models fill exactly this gap. They are good at the "almost a rule" transformation: recognising the pattern in each place it appears, adapting to the local variation, and producing code in the target idiom. What they lack is the classical tools' guarantee of correctness. The migrations that have worked combine the two: models for flexibility, tests and compilers for assurance. Three case studies Google published an account of its experience in early 2025, in a paper by Stoyan Nikolov, Daniele Codecasa, Anna Sjovall, Maxim Tabachnyk, Satish Chandra, Siddharth Taneja and Celal Ziftci titled "How is Google using AI for internal code migrations?". One migration changed identifiers in Google Ads from 32-bit to 64-bit integers, in a codebase of more than 500 million lines, where the identifiers were spread across tens of thousands of locations in thousands of files and a manual effort was estimated at hundreds of engineer-years. Another moved tests from the JUnit 3 framework to JUnit 4, touching 5,359 files and 149,000 lines over three months. A third moved code from the Joda-Time library to Java's built-in time package. Google reported that around 80 per cent of the code modifications in the migration changes were produced entirely by AI, with the rest written or edited by humans, and estimated that total migration time fell by about half, including the time spent reviewing and correcting the model's output. The structure of Google's process matters more than the headline numbers. Engineers did not simply ask a model to migrate the code. They first used existing tooling, including code search and static analysis, to find the locations that needed to change. The model was asked to make the change at each location, with relevant context supplied. Each proposed change was then validated automatically: it had to compile, and the relevant tests had to pass. Changes that failed were sent back to the model with the errors, or set aside for humans. Only validated changes went to human reviewers, who reviewed them in the normal way. The model generated; the build system and test suite verified; humans reviewed what survived. Amazon described a similar programme. In August 2024 Amazon's chief executive, Andy Jassy, reported that the company had used its Amazon Q Developer assistant to upgrade a large number of internal Java applications to Java 17. He said the average time to upgrade an application had fallen from about 50 developer-days to a few hours, that more than half of Amazon's production Java systems had been upgraded in six months, that the company estimated savings equivalent to 4,500 developer-years of work and $260 million a year in efficiency gains, and that developers shipped 79 per cent of the auto-generated code reviews without further changes. These are Amazon's own estimates, made by a company that sells the tool, and they should be read that way. But the shape of the work matches Google's: a well-defined target version, a body of existing tests and a build system to check each change. Airbnb provided a third example, described in a 2025 engineering blog post. Its engineers needed to migrate about 3,500 React component test files from the Enzyme testing library to React Testing Library, a project they had estimated would take a year and a half by hand. They built a pipeline in which each file passed through stages: an initial conversion by the model, then fixing test failures, then linting, then type checking. At each stage, any errors were fed back to the model for another attempt, up to a limit. About three quarters of the files were migrated in the first four hours of running the pipeline, and 97 per cent within four days, with the remaining hundred or so finished by hand. The whole project took six weeks. The team reported that giving the model rich context, including related files, examples of good migrations and passing tests from the same project, and simply retrying with the error messages, worked better than elaborate prompt engineering. Why migrations suit generative AI The three cases share four features, and together these explain why migrations have been the most successful application. First, the specification is unusually clear. "Make this code use JUnit 4 instead of JUnit 3, with the same behaviour" is a precise instruction. The target framework is documented, the transformation is known, and there is little room for the kind of design decisions that ordinary feature work leaves to the model. Second, the verification already exists. The tests that checked the code before the migration check it after. For a test migration like Airbnb's, the tests themselves are the thing being migrated, and the check is whether they still pass and still exercise the component, which is weaker but still useful. The compiler and type checker provide a second layer. The engineering problem is reduced to generating candidates until one passes checks that the organisation already trusts. Third, the work is decomposed into many small, independent pieces. A migration across 5,000 files is 5,000 small migrations, each of which can be attempted, checked and, if necessary, retried or handed to a human independently. Errors stay local. Small, separate changes are also much easier for humans to review than one enormous change. Fourth, the pattern is well represented in training data. Popular framework migrations have been done by many projects in public, with the before and after visible in version history. The model has seen the transformation many times. Where AI refactoring goes wrong The same reasoning predicts where AI-driven refactoring fails, and experience bears it out. It goes wrong when tests are thin. If a module has little test coverage, a model's migration may compile and pass the existing tests while changing behaviour in some untested path: a default value that differs between two libraries, a change in how a null input is handled, a subtly different rounding rule. Date and time migrations are notoriously exposed to this, because time zone and daylight-saving behaviour is rarely tested thoroughly and differs between libraries in exactly the ways that matter. It goes wrong when the change is not decomposable. Architectural refactorings, such as splitting a monolith into services, changing a data model or redesigning an interface used across many teams, cannot be broken into thousands of independent local edits each checked by existing tests. The decisions are interconnected, and the tests for the new structure do not yet exist. A model can help with pieces of such work, but the design and its verification remain human tasks. It goes wrong when "refactoring" is used loosely. A developer who asks a chat assistant to "clean up" or "refactor" a function often receives a version that changes behaviour as well as structure: an added validation step, a different error message, a reordered side effect. The model is producing what a cleaned-up version of such code usually looks like, not preserving this code's exact behaviour, because nothing in its objective distinguishes the two. Without tests that pin down current behaviour, the change can go unnoticed. A worked example makes the date-library hazard concrete. Suppose a scheduling service computes "the same time tomorrow" for a recurring reminder. In the old library, the code adds one day to a date-and-time value attached to the user's time zone, which on the night clocks change yields the same wall-clock time, say 9 a.m., as before. A model migrating the call to the new library might reasonably translate it into adding twenty-four hours to an instant in time, a form that is also common and also looks correct. On 364 nights of the year the two are identical. On the night clocks go forward or back, the migrated code fires the reminder an hour early or an hour late. Unless the test suite contains a case that crosses a daylight-saving transition in a time zone that observes one, every test passes. Nothing in the model's process distinguishes the two meanings of "a day later", because both appear constantly in public code; only a test written by someone who knew the difference, or a reviewer who knows the library well, would catch it. The same problem arises with numeric types. A migration that widens an integer type, as in Google's identifier change, is relatively safe because widening preserves every value. A migration that changes between floating-point and fixed-point decimal types, or between libraries with different rounding defaults, can change results by fractions of a penny in a way that no ordinary test notices and every auditor eventually does. And it goes wrong silently in a way that classical tools do not. A rule-based transformation that cannot handle a case usually fails loudly, by refusing to apply or producing code that does not compile. A model that cannot handle a case usually produces something plausible. The burden of finding the wrong case shifts entirely to verification. Making refactoring safe The practical approach that emerges from the successful cases has a consistent structure. Before migrating, strengthen the safety net. If the code to be changed is poorly tested, add characterisation tests, tests that record what the code currently does rather than what it should do, so that any change in behaviour becomes visible. This is an old technique, described by Michael Feathers in Working Effectively with Legacy Code (2004), and generative models are themselves useful for writing such tests, with the caveats set out in the next chapter. Use deterministic tools where they suffice and models where they are needed. Finding every call site of a deprecated API is a job for code search or static analysis, which will find them all. Deciding how to rewrite each one is where the model adds value. Many teams now combine the two, using a rule-based tool such as OpenRewrite for the mechanical parts and a model for the cases the rules do not cover. Keep changes small and validate each one automatically: compile, type-check, run the tests, run the linter, and feed failures back for another attempt. Set a limit on retries and route persistent failures to humans rather than letting the model keep trying until something passes, since repeated attempts against a weak test suite can eventually produce a change that passes the tests by undermining them. Review the output as one would review a colleague's change, and pay particular attention to anything the model did beyond the requested transformation. Reviewers should be suspicious of edits to tests during a migration of production code, of added special cases, and of any change in error handling. Finally, roll out gradually, with monitoring that would reveal a behavioural change in production, and with the ability to revert. Google's large-scale change practice, long before generative AI, relied on splitting changes into pieces that could be submitted, monitored and rolled back independently. That discipline matters more, not less, when the changes are generated by a model. Refactoring in everyday work Large migrations are the spectacular case, but most refactoring happens in small pieces during ordinary feature work: extracting a function that has grown too long, consolidating two similar classes, renaming a confusing concept. Here AI assistance changes the economics in a quieter way. Refactoring has always been postponed because it costs time and produces nothing a customer can see. When an assistant can propose the extraction or consolidation in seconds, the cost of doing it falls, and teams that value clean code can afford more of it. But the GitClear data discussed in Chapter 4 suggests that, left to themselves, the tools push the other way: generating new code is easier for them than finding and reusing existing code, so duplication grows. The way to get the benefit without the cost is to make refactoring an explicit request rather than a hoped-for side effect. Asking an assistant to identify duplicated logic across a module, to propose a shared function, and to update the callers, with the tests run after each step, is a well-specified task of exactly the kind the tools handle well. Asking for a new feature and hoping the assistant notices that half of it already exists is not. The success of AI-driven migration, in short, is not evidence that the models understand programs well enough to change them safely. It is evidence that organisations with strong verification can use a generator that is right most of the time to do work that was previously too expensive to attempt. The verification was there first. That raises the obvious next question: can the same models be used to build the verification itself? Chapter 6: Machines Writing the Tests If generative AI makes writing code cheap and moves the cost to verification, the obvious response is to make verification cheap too. Unit tests are the most common form of automated verification in software, and writing them is a task developers widely dislike and often skip. A model that can write tests on demand seems to solve the problem at its root: let AI write the code, let AI write the tests, and let the tests check the code. The idea is not wrong, but it contains a trap, and the trap explains both what AI test generation is good for and where it can make things worse. A test has two parts. One part sets up a situation and exercises the code. The other part, the assertion, states what the right result is. The first part is a matter of mechanics, and models are very good at it. The second part is a statement about intended behaviour, and intended behaviour is exactly what the code alone cannot reveal. A model that reads a function and writes a test for it will, by default, assert that the function does what it currently does. If the function is wrong, the test will confirm the error. This chapter explains how test generation works, what the evidence shows, and how to use it so that it strengthens verification rather than counterfeiting it. Before language models Automated test generation is an old research field, and its tools are still relevant. Random and search-based generators produce test inputs automatically and look for those that increase coverage or cause crashes. EvoSuite, developed by Gordon Fraser and Andrea Arcuri from around 2011, uses genetic algorithms to evolve suites of Java unit tests that maximise code coverage. Randoop, developed by Carlos Pacheco and Michael Ernst, generates sequences of method calls at random and keeps those that reveal errors or record useful behaviour. Property-based testing, popularised by the Haskell library QuickCheck from Koen Claessen and John Hughes in 2000 and since adopted in libraries such as Hypothesis for Python, takes a different approach: the developer states a general property, such as "sorting a list twice gives the same result as sorting it once", and the tool generates many inputs to check it. Fuzzing, which feeds programs large volumes of unexpected inputs to find crashes, has found many thousands of security bugs; Google's OSS-Fuzz service has run continuously on open-source projects since 2016. These tools taught the field two lessons that apply directly to language models. The first is that coverage is not correctness. A generated test that executes every line of a function but checks nothing meaningful about its output achieves full coverage and verifies nothing. The second is the oracle problem: the hardest part of automated testing is knowing what the right answer is. Classical generators handled it in limited ways. Crashes and exceptions are always wrong. Properties stated by a human define correctness for many inputs at once. For everything else, generators like EvoSuite record the current behaviour as "regression assertions", which catch future changes but say nothing about whether the current behaviour is correct. Classical generators also had a practical weakness: their tests were hard to read, with machine-generated variable names, odd input values and long sequences of calls that no human would write. Developers were reluctant to maintain them. This is where language models changed the picture. What language models add A model writes tests that look as if a person wrote them. It chooses meaningful names, realistic input values and a sensible structure, follows the project's testing conventions and uses its testing libraries. It can read the documentation, comments and function names and infer intended behaviour where the code alone does not show it. Asked to test a function called parse_iso_date, it will try leap days, time zones and malformed strings, because those are what tests for such functions usually include. Academic evaluation found this effective early on. Max Schäfer, Sarah Nadi, Aryaz Eghbali and Frank Tip built TestPilot, a tool that generated unit tests for JavaScript packages using an OpenAI model given each function's signature, documentation and usage examples, and fed failing tests back to the model for repair. Evaluated on 25 npm packages with 1,684 functions, it reached a median statement coverage of 70.2 per cent and branch coverage of 52.8 per cent, substantially above Nessie, a state-of-the-art feedback-directed generator, which reached 51.3 and 25.6 per cent. The work was published in IEEE Transactions on Software Engineering in 2024. The most informative industrial report came from Meta. In a 2024 paper, Nadia Alshahwan and colleagues described TestGen-LLM, a tool that uses language models to extend existing human-written test classes. Its design is worth understanding, because it is built around the assumption that the model's output cannot be trusted. Every generated test passes through a series of filters. Does it build? Does it pass? Does it pass reliably when run several times, so that it is not flaky? Does it increase coverage, so that it tests something the existing suite did not? Only tests that pass all the filters are offered to engineers. On Instagram's Reels and Stories products, 75 per cent of generated test cases built correctly, 57 per cent passed reliably and 25 per cent increased coverage. Across test-a-thons on Instagram and Facebook, the tool improved about 11.5 per cent of the classes it was applied to, and engineers accepted 73 per cent of its recommendations for production. These numbers show the generation step to be wasteful and the filtering step to be essential. Of every hundred generated tests, about three quarters would never have been worth an engineer's attention. The filters, all of which are automatic and cheap, removed them. What reached humans was a small, high-quality set, mostly accepted. Meta then took the idea further. A 2025 paper by Christopher Foster, Mark Harman and colleagues described ACH, a system that uses mutation testing to direct test generation. Mutation testing is an established technique for judging how good a test suite is: it introduces deliberate small faults, called mutants, into the code, such as flipping a comparison or deleting a statement, and checks whether the tests detect them. A mutant that survives reveals a gap in the tests. ACH used a model to generate realistic mutants representing a specific class of concern, in this case privacy faults, and then generated tests designed to catch them. Applied to Android Kotlin code across seven Meta platforms, it produced 571 privacy-hardening tests from 9,095 mutants; in test-a-thons on Messenger and WhatsApp, engineers accepted 73 per cent of the tests and judged 36 per cent relevant to privacy. The significance of this design is that the tests are targeted at a particular kind of failure that matters, rather than at coverage as such. The oracle problem returns The central risk of AI test generation is not that the tests fail. Failing tests are easy to spot and discard. The risk is tests that pass for the wrong reason. The most direct version is the confirmation of bugs. If a function computes a discount incorrectly, and a model is asked to write tests for the function by reading it, the model has two sources of information about expected behaviour: the code, which says one thing, and whatever names and comments suggest. When the code's behaviour is concrete and the intent is vague, models frequently write assertions that match the code. The tests pass, coverage rises, and the bug is now protected by a test that will fail if anyone fixes it. Several research groups have documented this tendency, and it follows directly from the oracle problem: without an independent statement of intended behaviour, a test generator can only record actual behaviour. It is worth following one case through. A function is meant to give a 10 per cent discount on orders of 100 pounds or more, but contains an off-by-one comparison: it applies the discount only to orders strictly greater than 100. A developer asks an assistant to write unit tests for it. The assistant reads the code, sees the comparison, and writes sensible-looking cases: an order of 50 pounds with no discount, an order of 150 pounds with a 15-pound discount, and, because boundaries are what good tests check, an order of exactly 100 pounds with no discount. All three pass. Coverage of the function is complete. The boundary test, the one that should have exposed the bug, now enshrines it. Had the developer instead written a single line stating the requirement, that an order of exactly 100 pounds receives the discount, and asked the assistant to turn it into a test, the test would have failed immediately and the bug would have been found. The difference between the two approaches is not the model's skill. It is whether the expected value came from the code or from somewhere independent of it. A second version is weak assertions. Models sometimes write tests that call a function and assert only that the result is not null, or that no exception was raised, or that a list has some length. Such tests add coverage and catch almost nothing. Mutation testing exposes them, because mutants survive them; line coverage does not. A third version arises when code and tests are generated together. If an agent is asked to implement a feature and write tests for it, the same model with the same understanding of the task produces both. Any misunderstanding of the task appears in both, consistently, and the tests pass. The agent reports success, and the passing tests provide false assurance to the reviewer. The independent check that tests are meant to provide has quietly disappeared, because the "two" checks share a single point of failure: the model's interpretation of the request. And a fourth version, mentioned in Chapter 2, is the agent that is under pressure to make tests pass and changes the tests rather than the code. This is easy to catch if reviewers look for it, and easy to miss if they look only at whether the build is green. Using AI test generation well The evidence points to several practices that keep test generation on the right side of the line between strengthening verification and imitating it. Keep a human source of truth about behaviour. The most valuable tests state requirements, and requirements come from people: product specifications, bug reports, domain experts, and developers' own understanding of what the code should do. A practical division of labour is for the human to state the cases and expected outcomes in a sentence each, and for the model to turn them into well-structured test code. "An order of exactly 100 pounds gets the 10 per cent discount; an order of 99.99 does not" is a specification a model can implement reliably. "Write tests for this function" is an invitation to record whatever the function does. Separate the generation of tests from the generation of code. Where possible, write or approve the tests before the implementation, in the manner of test-driven development, or have tests generated from the specification by a separate process that does not see the implementation. The point is to keep the two checks independent. Filter automatically, as Meta did. Generated tests should be rejected unless they build, pass consistently, and add something: coverage of a previously untested path, or, better, the killing of a mutant that the existing suite missed. Mutation testing tools exist for most major languages, such as PIT for Java and Stryker for JavaScript and C#, and they are the most direct measure of whether a test actually checks something. Use generated tests for the purposes they are good at. Characterisation tests before a refactoring, where the aim is precisely to record current behaviour, are an ideal use: the oracle problem does not arise, because current behaviour is what one wants to preserve. Expanding coverage of edge cases in code whose main behaviour is already well tested is another. Generating input data for property-based tests, where a human has stated the property, combines the model's creativity with a human-defined oracle. Review tests as carefully as code. A reviewer looking at a generated test should ask what would make it fail. If the answer is "almost nothing", the test is decoration. If the assertion's expected value was obviously copied from the implementation's current output, it is a regression check, not a correctness check, and should be labelled as such or reviewed against the requirement. Beyond unit tests Unit tests are the most discussed target, but the same reasoning applies elsewhere in the testing stack. Models are useful for generating realistic test data, such as plausible customer records, varied addresses with awkward characters, or malformed inputs, which is laborious to write by hand. They are useful for writing end-to-end browser tests from a description of a user journey, a task where the expected outcomes are usually obvious from the description itself. And they are being combined with fuzzing: Google reported in 2023 and 2024 that using language models to write the "harnesses" that connect a fuzzer to a library's functions extended OSS-Fuzz coverage to many projects that had previously had little or none, and that the approach found real vulnerabilities. Fuzzing suits AI assistance well because its oracle is built in: a crash or memory-safety violation is wrong whatever the code was meant to do, so the model only has to set up the experiment, not decide the right answer. That pattern generalises. AI test generation works best where the oracle is independent of the model: a crash, a human-stated property, an existing behaviour to be preserved, a mutant to be killed, or a requirement written by a person. It works worst where the model is asked to infer both the behaviour to check and the expected result from the code under test. What tests can and cannot do Test generation is the most direct attempt to use AI on the verification side of the ledger, and it works where it is built on independent knowledge of what correct behaviour is. Meta's systems succeeded because they were anchored to existing human-written tests, to measurable improvement, and in ACH's case to a clearly defined class of faults that matter. TestPilot succeeded on coverage because it drew on documentation written by humans to describe intended behaviour. Where that anchor is missing, AI-generated tests become a mirror held up to the code: they show what the code does, in a form that looks like assurance. That is not worthless, since detecting future changes has real value, but it is not verification of correctness, and confusing the two is one of the easiest mistakes a team adopting these tools can make. There is one category of failure where the oracle problem is especially acute, because the most important behaviour is what the code must never do, under conditions that ordinary tests never try: inputs crafted by an adversary. That is the domain of security, and it is where AI-generated code has its most serious problems. Chapter 7: The Security of AI-Suggested Code Security is where the shift from writing code to verifying it has the highest stakes. An ordinary bug in AI-generated code costs a failed test, a bad release or a confused user. A security flaw costs whatever an attacker can take, and the attacker gets to choose the inputs. Ordinary testing checks that software does what it should for the inputs its developers imagined. Security requires that software does nothing harmful for any input at all, including inputs designed to cause harm, and that is much harder to check. AI coding tools affect security in three distinct ways, and it helps to keep them separate. The first is the code they suggest: models reproduce the insecure patterns common in their training data, and developers accept them. The second is the software supply chain: models suggest dependencies, sometimes ones that do not exist, and attackers have learned to exploit that. The third is the tools themselves: coding agents that read untrusted content and can execute commands are a new kind of target. This chapter takes each in turn, describing the classes of flaw and the defences that work. It deliberately stays at the level of concepts. Nothing here requires, or includes, working attack code. Insecure suggestions The first systematic study of the problem appeared within months of Copilot's preview. Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt and Ramesh Karri, in a paper titled "Asleep at the Keyboard?", presented at the IEEE Symposium on Security and Privacy in 2022, built 89 scenarios designed to tempt a code generator into common weaknesses from MITRE's list of the most dangerous software weaknesses. Copilot produced 1,689 programs across those scenarios, and the authors found approximately 40 per cent to be vulnerable. Models have improved a great deal since 2021, but the problem has not gone away. In July 2025 the application security company Veracode published an evaluation of more than 100 language models on code-completion tasks designed to expose specific weaknesses. It reported that 45 per cent of the generated samples failed its security tests, introducing flaws from the OWASP Top 10 list of web application risks. Failure rates varied by language: 72 per cent for Java, 45 per cent for C#, 43 per cent for JavaScript and 38 per cent for Python. For cross-site scripting, the models failed to produce secure code in 86 per cent of relevant samples. Notably, Veracode found that newer and larger models were much better at producing code that worked, but not correspondingly better at producing code that was secure. Like any vendor study, these figures depend on the task design, but they are consistent with the direction of the academic work. The reason follows from the training process described in Chapter 2. A model learns to produce the code that most commonly appears in contexts like the current one. A great deal of public code is insecure: tutorials that skip input validation to keep the example short, old answers on forums written before a vulnerability class was widely understood, prototype code never meant for production. Security is also mostly about what is absent. Secure code differs from insecure code by the presence of a validation check, an escaping function or a permissions test, and nothing in the logic of the feature requires those things to be there. The feature works either way. The resulting flaws fall into a small number of recognisable classes. Table 4 sets out the most common, with the defence for each. Table 4. Common vulnerability classes in AI-suggested code and their defences. Class What goes wrong Primary defence Injection (SQL, command, path) Untrusted input is spliced into a query, shell command or file path Parameterised APIs; avoid shells; validate and canonicalise paths Cross-site scripting User content is written into a web page without encoding Templating that escapes by default; content security policy Hardcoded secrets Keys or passwords are embedded in source Secret managers; secret scanning in commits and CI Weak cryptography Outdated algorithms, fixed salts or homemade schemes Vetted high-level libraries with safe defaults Missing authorisation An endpoint performs an action without checking who asked Central access-control layer; tests for denied cases Unsafe deserialisation Untrusted data is turned back into live objects Data-only formats; strict schemas Insecure configuration Debug modes, permissive CORS, disabled certificate checks Secure baselines; configuration scanning Sources: classes drawn from the OWASP Top 10 and MITRE CWE; defences are standard practice. Two classes deserve special mention because they are easy for reviewers to miss. The first is missing authorisation. A model asked to add an endpoint that deletes a user's document will produce one that deletes the document. Whether the person making the request owns that document is a question about the system's security model, which the model may not see and which nothing in the request mentions. The endpoint works perfectly in every functional test, because functional tests are written by people who are authorised. This class of flaw, broken access control, has been at the top of the OWASP Top 10 since 2021, and AI makes it more likely precisely because it is invisible in the local code. The second is weakened defaults introduced to make things work. When a model encounters an error such as a failed TLS certificate check or a cross-origin request being blocked, the most common fix in public code is to disable the check. Agents iterating to make an error go away are particularly prone to this. The code then works, and is insecure in a way that only a reviewer who knows why the check existed will notice. To see how such a flaw passes every ordinary check, consider a typical sequence. A developer asks an agent to add an API endpoint allowing a user to download an invoice by its identifier. The agent finds the existing invoice service, writes a handler that takes the identifier from the request, fetches the invoice and returns it as a PDF, and adds tests showing that a valid identifier returns the right document and an unknown one returns a "not found" response. The code is clean and idiomatic, the tests pass, the build is green. The reviewer, checking that the feature works, sees a sensible handler and good tests. What nobody checked is whether the person requesting invoice 10452 is the customer it belongs to. Any logged-in user can retrieve any other customer's invoices by changing the number. This class of flaw, known as insecure direct object reference, is among the most commonly reported in real web applications, and the fix is simple once seen: look up the invoice through the current user's account rather than by identifier alone, and add a test that one user cannot fetch another's invoice. The difficulty is seeing it, because nothing in the feature request, the code or the tests points to what is missing. Human behaviour compounds the technical problem. In a study published at the ACM Conference on Computer and Communications Security in 2023, Neil Perry, Megha Srivastava, Deepak Kumar and Dan Boneh at Stanford found that participants with access to an AI assistant wrote significantly less secure code than those without, and were more likely to believe their code was secure. Participants who trusted the assistant less and engaged more critically with it, rephrasing prompts and questioning outputs, produced fewer vulnerabilities. Fluent output invites trust it has not earned, and this is a verification failure in its purest form. The supply chain: hallucinated and poisoned dependencies Modern software is assembled largely from open-source packages, installed by name from public registries such as npm for JavaScript and PyPI for Python. Models suggest these dependencies as readily as they suggest code, and sometimes the package they suggest does not exist. Joseph Spracklen and colleagues measured this in a study presented at the USENIX Security Symposium in 2025, titled "We Have a Package for You!". Across 576,000 code samples generated by 16 models in Python and JavaScript, they found that at least 5.2 per cent of the packages suggested by commercial models and 21.7 per cent of those suggested by open-source models did not exist. They identified 205,474 unique non-existent package names. Many recurred across repeated queries, which means they are predictable. Predictable hallucinations are an opportunity for an attacker. If a model reliably suggests a package name that no one has registered, an attacker can register it and fill it with malicious code. Any developer who follows the model's suggestion and installs it will then run that code. This technique has been nicknamed "slopsquatting", by analogy with typosquatting, in which attackers register misspellings of popular package names. The threat is not hypothetical. In 2024 Bar Lanyado, a researcher at Lasso Security, noticed that models repeatedly suggested installing a Python package called huggingface-cli, which did not exist under that name. He registered an empty, harmless package with that name as an experiment. Over three months it received more than 15,000 genuine downloads, and the installation instructions of at least one project from a major technology company told users to install it. The defences are well understood and mostly predate AI. Organisations should install dependencies through an internal proxy or mirror that allows only approved packages, or at least blocks very new or very rarely downloaded ones. Lock files should pin exact versions and checksums. Software composition analysis tools should check every new dependency for known vulnerabilities, for its age and popularity, and for suspicious install-time behaviour. And developers should treat a dependency suggested by a model as an unverified claim: confirm that it exists, that it is the project it appears to be, and that it is maintained. A related, more speculative risk is poisoning of the training data itself. In 2021 Roei Schuster and colleagues showed, in a paper at the USENIX Security Symposium titled "You Autocomplete Me", that by adding carefully crafted files to a model's training data, an attacker could make a code completion model more likely to suggest a specific insecure choice, such as a weak encryption mode, in targeted contexts. Large commercial models are trained on such vast datasets that poisoning them is difficult, but the principle that a model's suggestions are only as trustworthy as its training data still holds, and it matters more for smaller models fine-tuned on narrower data. The tools as targets The third category is newer and in some ways the most serious. A coding agent reads content, decides what to do, and acts with the developer's permissions: it can edit files, run commands, access credentials stored on the machine and use connected services. Some of the content it reads is untrusted, such as the text of a GitHub issue, a web page, a dependency's documentation, or a comment in a file contributed by an outsider. A language model does not reliably distinguish between instructions from its user and instructions embedded in the material it is reading. This is prompt injection, and it is the defining security problem of AI agents. Prompt injection turns any text an agent might read into a possible attack vector. An issue filed in a public repository could contain hidden text instructing an agent that picks up the issue to do something other than fix the bug. Researchers at Pillar Security reported in 2025 that configuration files used to give coding assistants project-specific rules could carry instructions hidden with invisible Unicode characters, so that a human reviewer saw nothing unusual while the assistant read and followed them. Real incidents have shown what is at stake. In August 2025 Microsoft patched a vulnerability, CVE-2025-53773, in GitHub Copilot's agent features in Visual Studio. Instructions planted in content the agent read could lead it to modify the project's editor settings to turn on an experimental mode in which the agent ran commands without asking the user for confirmation. Once that safeguard was disabled by the agent itself, the injected instructions could cause it to execute commands on the developer's machine. The flaw illustrates a general principle: an agent must never be able to change the controls that constrain it. Two incidents in the summer of 2025 showed the coding tools becoming part of supply-chain attacks. In July, a version of Amazon's Q Developer extension for Visual Studio Code was published containing an injected prompt instructing the agent to delete local files and cloud resources. The change had been merged into the extension's repository without adequate review. Amazon said the injected code would not have run successfully and that no customer resources were affected, and it released a corrected version two days later. In late August, attackers compromised the publishing process of Nx, a popular JavaScript build system, through a flaw in its automated build workflow, and released malicious versions to npm. The malware used AI command-line tools already installed on developers' machines, prompting them to search the file system for credentials and secrets. According to the security firm Wiz, the attack exposed more than 2,000 accounts and over 7,000 repositories. Neither incident was a failure of a model's coding ability. Both were failures of the boundary around the model: what it was allowed to read, what it was allowed to do, and who reviewed changes to the tools themselves. Defences that work Taken together, the three categories call for defences in layers, and almost all of them are forms of verification that do not depend on the model being right. For insecure suggestions, the strongest defence is to make the secure option the easy option, so that the model's tendency to follow the codebase's existing patterns works in favour of security. A codebase in which every database call goes through a parameterised data-access layer, every page through an escaping template engine, and every endpoint through a central authorisation check will produce safer AI suggestions, because the model imitates what it sees. Static application security testing tools, which analyse code for known vulnerability patterns, should run on every change, regardless of who or what wrote it. Secret scanning should block commits containing credentials. Security-focused review should concentrate on what models characteristically omit: authorisation checks, input validation at trust boundaries, and any change that disables a safeguard. Tests should include negative cases, such as an unauthorised user attempting the action and malformed input being rejected, because functional tests written from the feature description will not. Prompting helps at the margin. Instructions to the model to follow secure practices, supplied through the tools' custom-instruction files, measurably improve some outputs. But they are not a control: a model that usually follows an instruction is not one that always does. For the supply chain, the defences are those described above: controlled package sources, lock files, dependency scanning, and checking any suggested package before installing it. For agents, the governing principle is least privilege combined with human approval at the points that matter. Agents should run in sandboxed environments, such as containers or cloud workspaces with no access to production credentials, and should have network access restricted to what the task requires. Actions with consequences, such as running arbitrary commands, pushing code, installing packages, or changing configuration, should require explicit approval, and settings that bypass approval should not be modifiable by the agent itself. Content from untrusted sources should be treated as data, not instructions, and tasks that combine reading untrusted input with access to secrets and the ability to send data elsewhere deserve particular caution, because that combination is what makes data theft possible. Agent-generated pull requests should go through the same review and branch protections as any other change, and changes to the agent's own configuration files should require review by someone who understands their significance. The OWASP project's Top 10 for Large Language Model Applications, first published in 2023 and updated since, provides a useful checklist for these risks, with prompt injection at its head. The verification principle, applied Security is the clearest case of the book's argument because it is where verification by ordinary means fails most completely. A model can produce code that passes every functional test and is still exploitable. A developer can feel confident in code that is unsafe. An agent can do exactly what the text it read told it to do, and that text can come from an attacker. The organisations that use these tools safely are the ones that do not rely on the model, or on the developer's impression of the model's output, for security. They rely on layers of independent checks: secure defaults built into the codebase, automated analysis, controlled dependencies, constrained agents and review focused on what models get wrong. Those checks would have been good practice before AI. With AI generating a growing share of code, they have become the difference between a faster team and a faster route to a breach. Chapter 8: The Profession Rearranged Every previous wave of automation in software engineering has been followed by more software engineers, not fewer. Compilers removed the need to write machine code by hand, and the number of programmers grew. High-level languages, open-source libraries, cloud platforms and frameworks each eliminated a large class of work, and each time demand for software rose faster than the work per program fell. Economists call the pattern the Jevons effect, after William Stanley Jevons's observation in 1865 that more efficient steam engines led to more coal being burned, not less. When something gets cheaper, people tend to use more of it. There are good reasons to expect the same pattern now, and some evidence that it will not apply evenly. Generative AI makes some software engineering tasks far cheaper, and the demand for software, already very large, may expand to absorb the capacity. But the tasks it makes cheapest are concentrated in the work traditionally given to the least experienced developers, and that changes the structure of the profession even if it does not shrink it. This chapter examines that change, the unsettled legal questions about the material the models learned from, and how teams can organise around a tool that writes faster than they can review. The entry-level squeeze The most significant labour market evidence so far comes from Erik Brynjolfsson, Bharat Chandar and Ruyu Chen at Stanford's Digital Economy Lab, in a paper titled "Canaries in the Coal Mine? Six Facts about the Recent Employment Effects of Artificial Intelligence", first circulated in August 2025. Using payroll data from ADP, which covers millions of workers at tens of thousands of American firms, they found that employment for workers aged 22 to 25 in the occupations most exposed to AI had fallen relative to less exposed occupations, while employment for older workers in the same occupations had held up or grown. Software development was one of the clearest cases. In the November 2025 version of the paper, they reported that by September 2025, employment for software developers aged 22 to 25 had declined by nearly 20 per cent from its peak in late 2022, while employment for older age groups continued to grow. The authors were careful about causation. Late 2022 was also when interest rates rose sharply and the technology industry began a period of layoffs after its pandemic hiring boom. In a follow-up note in February 2026 they examined interest-rate sensitivity and found it did not explain the pattern across occupations: the occupations most exposed to AI were, if anything, less sensitive to interest rates. With their strictest statistical controls, though, the relative decline for the most exposed young workers became clear only from 2024 onwards, and they stated plainly that they did not believe AI was the only factor. The honest summary is that the evidence is consistent with AI reducing demand for entry-level developers, that other factors contributed, and that the pattern had not reversed by late 2025. The mechanism is not hard to see. The tasks that have traditionally been given to junior developers are well defined, limited in scope, and checkable: write a function to a clear specification, add a test, fix a bug with a known reproduction, implement a form following an existing pattern. Those are exactly the tasks at which AI tools are strongest, for the reasons set out in Chapters 3 and 4. A senior developer with an AI assistant can now do in an hour what they would previously have delegated to a junior colleague for a day. The judgement work that remains, deciding what to build, how to structure it, whether a change is right, is the work that has traditionally required experience. The apprenticeship problem This creates a problem that is more serious than the short-term job numbers. Software engineering has always been learned largely on the job. Juniors become seniors by doing junior work: writing many small pieces of code, making mistakes, having them caught in review, and gradually building the tacit knowledge that lets an experienced developer look at a change and sense that something is wrong. If that work is now done by AI and reviewed by seniors, the path by which people acquire the judgement that verification requires is narrowing at exactly the moment that judgement has become the scarce skill. There is a related risk for people who are already in the profession, especially those early in their careers: that AI assistance substitutes for understanding rather than building it. A developer who accepts working code without understanding it can complete tasks without learning from them. Educators have raised this concern since the tools appeared, and the Perry study described in Chapter 7, in which participants using AI were both less secure and more confident, is one illustration of how fluent output can short-circuit the scrutiny through which skill develops. On the other hand, the Cui and colleagues field experiments found that less experienced developers gained most from the tools, and AI assistants can be patient, always-available tutors that explain unfamiliar code on request. Which effect dominates depends on how the tools are used. Asking a model to explain code and then writing the change oneself builds skill. Accepting a generated change without reading it does not. Organisations that want to have senior engineers in ten years need to treat this as a deliberate design problem. Some practical responses are emerging. Juniors can be given ownership of verification tasks that build judgement, such as reviewing AI-generated changes under the supervision of seniors, writing the specifications and test cases that agents then implement, and investigating failures. Teams can keep some work AI-free for learning purposes, much as pilots who fly highly automated aircraft must still practise flying by hand. And career frameworks can be updated to value the skills that now matter: precise specification, rigorous review, test design and system-level reasoning. The burden on open source The profession's dependence on open source adds another dimension. Open-source maintainers, most of whom are volunteers or work on their projects part-time, have found that cheap generation creates work for them. Projects began receiving pull requests generated by AI and submitted by contributors who did not understand them, and bug reports, including security reports, that described vulnerabilities which did not exist. Daniel Stenberg, the lead developer of curl, one of the most widely used pieces of software in the world, wrote publicly in 2024 and 2025 about the time his project was losing to plausible-sounding but false AI-generated security reports submitted through its bug bounty programme, and the project introduced rules requiring reporters to disclose AI use and to be able to answer questions about their reports. This is the verification problem in another form. Generating a report or a patch has become nearly free, while verifying it still costs an expert's time, and in open source the expert is often unpaid. Projects have responded with contribution policies that require disclosure of AI use, ask contributors to confirm they have tested and understood their changes, and allow maintainers to close unverified submissions without detailed review. Companies that depend on open-source software, which is almost all of them, have a direct interest in not adding to this burden, and in supporting the maintainers whose verification work their own products rely on. Who owns the training data The models learned from public code, and much public code is under licences that impose conditions. Permissive licences such as MIT require that the copyright notice be preserved when the code is redistributed. Copyleft licences such as the GNU General Public License require that derivative works be released under the same terms. Whether training a model on such code, or producing output that resembles it, triggers these obligations is a question the law has not yet settled. The main test case in the United States has been Doe v. GitHub, a class action filed in November 2022 by programmers represented by the lawyer and programmer Matthew Butterick and the Joseph Saveri Law Firm, against GitHub, Microsoft and OpenAI. The plaintiffs argued, among other things, that Copilot violated the Digital Millennium Copyright Act by removing copyright management information, such as licence notices and author names, from code it reproduced, and that it breached the open-source licences under which the code was published. The district court dismissed most of the claims, including the DMCA claim, but allowed claims for breach of the open-source licences to proceed. On the DMCA claim, the plaintiffs appealed. In September 2026 the Ninth Circuit Court of Appeals affirmed the dismissal, holding that the relevant provision covers acts of removing copyright information from existing copies, which the plaintiffs had not shown, rather than generating new works that never contained it. The court expressly did not decide whether Copilot's output infringes copyright. The contract claims remained pending in the district court. Several practical points follow for organisations using these tools. The risk of verbatim reproduction of substantial licensed code is real but small, and is concentrated in famous, frequently copied code. Most major tools offer a setting to block suggestions that match public code, or to show the source and licence of matching code, and organisations with significant licence exposure should turn these on. The major vendors offer intellectual-property indemnities to commercial customers, subject to conditions such as using those filters; Microsoft announced its Copilot Copyright Commitment in 2023. Some open model projects, notably BigCode's, have trained only on permissively licensed code and offered developers an opt-out. And the question of whether AI-generated code can itself be copyrighted is separate and also unsettled. The United States Copyright Office has taken the position that material generated by AI without sufficient human creative control is not protected, which matters for companies that want to assert ownership of what their engineers produce with these tools. Organising the team around verification If the scarce resource in a software team has moved from writing code to verifying it, teams should be organised around that resource. Several practical principles follow from the evidence in the preceding chapters. Invest in the verification infrastructure before scaling generation. The DORA findings and the migration case studies point the same way: AI improves outcomes where tests, types, continuous integration and safe deployment already exist, and harms stability where they do not. A team planning to adopt coding agents widely would do better to spend its first quarter strengthening test coverage, adding type checking, tightening continuous integration and improving deployment safety than to spend it piloting tools. Decide deliberately where agents may work alone. Not all code carries the same risk, and a sensible policy distinguishes between areas. Internal tooling, test code, documentation and well-tested leaf services may be appropriate places for agents to work with light review. Authentication, payments, data handling covered by privacy law, safety-critical logic and shared infrastructure warrant closer human involvement, smaller changes and specialist reviewers. Many organisations already maintain lists of code owners who must approve changes to sensitive areas; extending those rules to agent-authored changes, and making them impossible for an agent to bypass, is a straightforward step. Manage review capacity explicitly. A team where every developer can produce several times as many pull requests needs either several times the review capacity or a deliberate limit on what reaches review. Useful practices include requiring that AI-generated changes be small; requiring that the author, human or agent, provide evidence of verification with the change, such as which tests were added, what was run and what was checked manually; and using automated review tools as a first pass while keeping a human as the final gate. Some teams have adopted the rule that the human who submits a change is accountable for it regardless of who or what wrote it, which encourages people to read what they submit. Put more effort into specification. Agents do what they are asked to do, so the quality of the asking matters more than it did when a developer's specification lived in their own head as they typed. Clear issue descriptions with acceptance criteria, design documents for anything non-trivial, and project instruction files that tell agents about conventions, architecture and prohibited practices all reduce the volume of plausible but wrong output that reviewers must catch. Measure what matters. As Chapter 4 argued, the right measures are end-to-end: lead time to reliable production, change failure rate, review load, rework and duplication. Counting lines generated, suggestions accepted or the share of code written by AI measures activity, not value, and creates an incentive to generate more code than needed. Keep security independent of generation. The controls described in Chapter 7, including automated security analysis, dependency controls, sandboxed agents and approval for consequential actions, should apply to all code and all agents, as defaults that a developer or an agent cannot quietly turn off. What the job is becoming Put together, the evidence suggests that software engineering is not disappearing but changing shape. The share of the job spent typing code is falling. The share spent deciding what should be built, specifying it precisely enough for a machine to build it, checking that what was built is right, and maintaining the systems that do the checking is rising. These were always the most valuable parts of the work, but they were bundled with a great deal of routine production that is now being unbundled. The shape of a working day illustrates the change. An experienced developer in a team that has adopted agents well might begin by writing two or three short specifications for well-understood tasks, each with acceptance criteria and examples, and assigning them to agents. While those run, the developer works on a design problem that no agent could take on, such as how two services should share responsibility for a new kind of data. Later, the agents' pull requests arrive. The developer reads each one carefully, rejects one that solved the wrong problem because the specification was ambiguous, asks for changes to another that duplicated an existing utility, and approves the third after checking that its tests would have failed before the change. The developer also reviews a colleague's hand-written change to the authentication flow, an area the team has reserved for human authorship. Measured in lines typed, the day was unproductive. Measured in working, verified software delivered, it may have been one of the most productive the developer has had. That change favours certain skills: reading code critically and quickly; understanding systems as a whole rather than functions in isolation; designing tests that actually constrain behaviour; recognising security implications; and writing clearly, since specifications and reviews are written in prose. It devalues fluency with syntax and memorised APIs. It raises the cost of the absence of skill, since an unskilled developer with a powerful generator can produce far more flawed code than one without it. And it makes the question of how the next generation of engineers will acquire those skills one of the most important questions the industry faces. Conclusion The public argument about AI and software has mostly been conducted in terms of replacement: will machines write the code, and if so, what happens to the people who used to? The evidence in this book points to a different and more useful framing. Machines already write a large share of new code in many organisations. What they have not done, and cannot do on their own, is establish that the code is right. The work has not disappeared. It has moved. Seen this way, the confusing pattern of evidence becomes coherent. The large productivity gains appear where verification was cheap and reliable: a greenfield task with a test suite, a framework migration across thousands of files each checked by existing tests and a compiler. The losses appear where verification was expensive: experts working on deep, idiosyncratic systems, where every plausible suggestion had to be weighed against knowledge that existed only in their heads. The rise in delivery instability that DORA found alongside rising adoption reflects change flowing faster into verification systems that did not grow to match. The falling trust developers report alongside rising use reflects the move from small suggestions that are easy to check to large generated changes that are not. Security flaws appear because security is the property ordinary testing is least able to verify. In each case, the outcome was set less by the model than by the checks around it. What follows Several consequences follow that go beyond the practical recommendations in the individual chapters. The first is that the value of an organisation's existing engineering discipline has risen sharply. Test suites, type systems, continuous integration, code review culture, deployment safety and good documentation were always worth having. They are now the mechanism by which AI's raw generative capacity is turned into working software, and their absence is the mechanism by which it is turned into debt. Two organisations buying the same tool on the same day will get very different results, and the difference was largely determined by investments made years earlier. For an organisation deciding where to spend, the uncomfortable implication is that the most valuable AI investment may be one that has nothing to do with AI: closing the gaps in its tests. The second is that specification has become an engineering discipline again. For decades the agile movement rightly pushed back against heavy upfront specification, on the grounds that the act of writing code was itself the best way to discover what was wanted. When an agent writes the code, that discovery no longer happens in a human head unless the human deliberately does the thinking first. Clear statements of intent, acceptance criteria, examples of correct and incorrect behaviour, and explicit constraints are now the inputs that determine output quality. This is not a return to the large specification documents of the past, but it is a return to taking the question "what exactly do we want?" seriously before the code exists. The third is that independence in verification matters more than volume. A model that writes both the code and its tests, from the same understanding of the task, produces two artefacts that agree with each other and may both be wrong. An agent that iterates until the tests pass will eventually pass them, by fair means or otherwise. The checks worth having are those that do not share the generator's assumptions: tests derived from requirements written by people, mutation testing that measures whether tests actually detect faults, security analysis that looks for known classes of flaw regardless of intent, and human reviewers who know the system. As generation becomes cheaper, the case for keeping these checks independent of it grows stronger. The fourth is that the boundary around the tools is now part of the security perimeter. Coding agents that read untrusted text and act with a developer's permissions turned out, within a year of their wide release, to be targets in their own right. The defences, which are sandboxing, least privilege, approval for consequential actions and review of changes to the tools themselves, are familiar from other domains. They have to be applied deliberately, because the default configuration of a productivity tool favours convenience. What remains unsettled Some important questions cannot yet be answered. How far the capability of the models will advance is uncertain. Performance on real engineering tasks improved continuously from 2021 to 2026, and nothing in the evidence suggests that progress has stopped. A model that understood a system's intent well enough to verify its own work reliably would change the argument of this book, because the scarce resource would move again. There is no sign yet that generation and verification have converged in that way: the most capable systems still rely on external tests and human review to separate their right answers from their plausible wrong ones, and the problem of knowing what the code should do is not one that more training on existing code obviously solves. But it would be unwise to treat the current division of labour as permanent. The effect on employment is unsettled too. The decline in entry-level developer employment is real in the data, its causes are mixed, and whether the Jevons effect will again expand total demand for software engineers enough to absorb the change is not yet known. The legal status of training on licensed code, and of the code the models produce, remains open in most jurisdictions, even after the Ninth Circuit's narrow ruling in the Copilot litigation. And the long-term effect on codebases is still unknown. The evidence of rising duplication and declining refactoring is suggestive, and the one controlled study of maintainability found no harm on a modest task. Software systems live for decades. Whether code produced at high speed with AI assistance will prove harder to maintain over that span, and whether AI tools will become good enough at maintenance to make the question moot, will only become clear with time. What to do differently For a developer, the practical lesson is to treat AI output as a draft from a capable but context-blind colleague: read it, question it, and own it. Use the tools to explain unfamiliar code, to write the tedious parts of things you understand, and to produce first drafts of tests for behaviour you can specify. Be most cautious exactly where the tool is most confident and the stakes are highest: authorisation, input handling, cryptography, concurrency and anything that touches money or personal data. Keep doing enough work by hand to keep the judgement that makes review possible. For a team lead or engineering manager, the lesson is to invest in verification before scaling generation, to manage review capacity as the constraint it has become, to measure end-to-end outcomes rather than activity, and to design deliberately for how junior engineers will learn. For a security leader, the lesson is to make secure patterns the default in the codebase so that the model imitates them, to apply automated analysis and dependency controls to all code regardless of origin, and to treat coding agents as privileged actors whose permissions, inputs and configuration need the same controls as any other. For anyone deciding whether a particular use of AI in software will work, the question to ask is the one this book has asked throughout: where, in this process, does someone or something independent check that the code is right? If there is a good answer, the tools will very likely make things better. If there is not, they will make things faster, and the difference between those two outcomes is the difference that matters. Further Reading Mark Chen et al., "Evaluating Large Language Models Trained on Code" (2021). The Codex paper, which introduced the HumanEval benchmark and laid out the approach behind the first version of GitHub Copilot. Its discussion of limitations and risks has aged well. Anton Lozhkov et al., "StarCoder 2 and The Stack v2: The Next Generation" (2024). The most detailed public account of how a large code-training dataset is assembled, filtered and deduplicated, and of how an open code model is trained on it. Carlos E. Jimenez, John Yang et al., "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" (ICLR 2024). The benchmark that moved evaluation from programming puzzles to real repositories. Read alongside OpenAI's February 2026 explanation of why it stopped reporting results on SWE-bench Verified, a useful case study in the limits of benchmarks. Sida Peng, Eirini Kalliamvakou, Peter Cihon and Mert Demirer, "The Impact of AI on Developer Productivity: Evidence from GitHub Copilot" (2023). The original controlled experiment on Copilot. Best read together with METR's "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity" (2025), which found the opposite result in a very different setting. DORA, State of AI-assisted Software Development (Google Cloud, 2025).** The most systematic survey evidence on how AI adoption relates to software delivery throughput and stability, and the source of the idea that AI acts as an amplifier of existing engineering practice. Stoyan Nikolov et al., "How is Google using AI for internal code migrations?" (2025). A practical account of large-scale AI-assisted migration, with a clear description of the validation pipeline that made it work. Nadia Alshahwan et al., "Automated Unit Test Improvement using Large Language Models at Meta" (FSE 2024). Describes TestGen-LLM and the filtering approach that makes AI-generated tests trustworthy enough to deploy. Its follow-up, Christopher Foster et al., "Mutation-Guided LLM-based Test Generation at Meta" (2025), shows how to aim test generation at specific classes of fault. Hammond Pearce et al., "Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions" (IEEE S&P 2022). The first systematic study of vulnerabilities in AI-suggested code. Pair with Neil Perry et al., "Do Users Write More Insecure Code with AI Assistants?" (ACM CCS 2023), on the human side of the same problem. Joseph Spracklen et al., "We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs" (USENIX Security 2025). The definitive measurement of hallucinated dependencies and the supply-chain risk they create. OWASP, Top 10 for Large Language Model Applications. A regularly updated, practitioner-oriented catalogue of risks in systems built on language models, including prompt injection and excessive agency. Martin Fowler, Refactoring: Improving the Design of Existing Code (2nd ed., Addison-Wesley, 2018).** The standard reference on what refactoring is and how to do it safely, all of which still applies when a model is doing the typing. Michael Feathers, Working Effectively with Legacy Code (Prentice Hall, 2004).** The classic guide to getting untested code under test before changing it. Its techniques, especially characterisation tests, are the foundation for safe AI-driven refactoring. Titus Winters, Tom Manshreck and Hyrum Wright (eds.), Software Engineering at Google (O'Reilly, 2020).** Its chapters on testing, code review and large-scale changes describe the verification infrastructure that makes automated change at scale possible. Erik Brynjolfsson, Bharat Chandar and Ruyu Chen, "Canaries in the Coal Mine? Six Facts about the Recent Employment Effects of Artificial Intelligence" (Stanford Digital Economy Lab, 2025). The most careful early evidence on how AI is affecting employment for young workers in exposed occupations, including software development. Hashtags: #GenerativeAIForCoding #AICoding #GitHubCopilot #AIAssistedProgramming #AIProgrammingAssistants #CodeGeneration #AutomatedRefactoring #CodeRefactoring #SoftwareEngineering #AIInSoftwareDevelopment #LargeLanguageModels #CodingAgents #AgenticCoding #AutomatedCodeMigration #CodeModernization #LegacyCodeMigration #UnitTestGeneration #AutomatedTesting #CodeVerification #CodeReview #SoftwareQuality #DeveloperProductivity #CodeSecurity #SecureCoding #PromptEngineeringForCode #SWEbench #ContinuousIntegration #TechnicalDebt #HumanAI collaboration #ResponsibleAI

Download the Book (PDF): Introduction A camera watches a job applicant answer a question about teamwork. In the half second after she finishes speaking, software has located her face in each frame, placed several dozen landmarks around her eyes, brows, nose and mouth, measured how far each point moved, and converted those movements into numbers. A second system has been listening to her voice, tracking the rise and fall of its pitch, its loudness, its pace, and the small irregularities in each vocal cycle. Somewhere downstream, these numbers become words on a dashboard: enthusiasm moderate, anxiety elevated, engagement high. A recruiter who will never meet her reads those words and moves on to the next candidate. Every step in that chain except the last is, in a narrow sense, real. The face did move. The pitch did rise. The measurements can be made with impressive precision, and they can be repeated. The trouble lies in the final step, where a measurement of muscle and sound becomes a statement about what a person feels, and that statement becomes a reason to treat her one way rather than another. That step is where the science of emotion is weakest, and it is where the commercial and political stakes of emotion recognition are concentrated. This book is about that gap. Its argument is simple to state and has large consequences: emotion AI is good at detecting signals and poor at knowing what they mean, and the acceptability of any use of it depends almost entirely on how much weight the meaning is asked to carry and who pays when the meaning is wrong. A car that notices its driver's eyelids drooping and sounds a chime is doing something modest with a signal whose meaning is fairly stable. A border kiosk that scores a traveler's facial micro-movements as evidence of deception is doing something enormous with a signal whose meaning is, on the best available evidence, close to unknowable. Both are called emotion AI. They should not be judged by the same standard, and much of the confusion in public debate comes from treating them as if they were one technology. What the field is The phrase "affective computing" was coined by Rosalind Picard of the MIT Media Lab, whose 1997 book of that name argued that computers would need to recognize, interpret and even express emotion if they were to interact naturally with people. Picard's vision was broad and, in its original form, largely humane: tutoring systems that noticed frustration, wearable devices that helped people with autism read social cues, health tools that tracked stress. Over the following quarter century the field grew into an industry, and the industry developed a shorter, more marketable name, "emotion AI", which tends to describe products rather than research programs. Those products draw on several channels. The face is the most studied, thanks to a coding system developed in the 1970s that breaks facial movement into numbered "action units". The voice is the second, through the analysis of prosody: the melody, rhythm and texture of speech, as distinct from its words. Physiological signals such as heart rate, skin conductance and breathing form a third channel, used mainly in wearables and laboratory work. Text sentiment analysis is a fourth, though it reads what people say about their feelings rather than how their bodies behave. Increasingly, systems fuse several of these at once. This book concentrates on the first two. Faces and voices are what cameras and microphones capture as a matter of course, which is why they dominate commercial deployment, and they are where the scientific controversy is sharpest. Physiology and text appear where they illuminate the argument, but they are not the main subject. Why it matters now For most of its history, emotion recognition was a laboratory curiosity and a speculative pitch. Three developments have changed that. First, deep learning made it cheap to train classifiers on millions of face images and thousands of hours of speech, so that a startup can now offer a working emotion API in weeks. Second, the cameras and microphones were already in place: in laptops, phones, cars, call-center headsets, retail ceilings and airport corridors. Adding an affect-scoring layer to an existing video stream costs little. Third, the conversational AI systems that spread rapidly from 2023 onward began to listen to tone of voice as well as words, folding affect inference into ordinary consumer products without anyone needing to buy a dedicated "emotion" tool. At the same time, the scientific foundation on which the industry was built came under serious attack. In 2019 a team of psychologists led by Lisa Feldman Barrett published a long, careful review concluding that facial configurations do not reliably and specifically reveal emotional states across people and situations. Researchers on deception had already shown that the tiny "micro-expressions" made famous by television are rare and do not distinguish liars from truth-tellers. Regulators began to respond. In 2022 Microsoft withdrew the emotion-inference features of its public face analysis service. The European Union's AI Act, whose first prohibitions took effect in February 2025, banned emotion recognition in workplaces and schools outright, with narrow exceptions for medical and safety reasons, while leaving its use at borders and in policing permitted but regulated as high-risk. The result is a field in an unusual position: technically maturing, commercially expanding, scientifically contested and legally fenced in, all at once. Anyone who builds, buys, regulates or is subjected to these systems needs a clearer picture than either the marketing or the denunciations provide. How the book proceeds The first chapter traces how emotion became something a machine could be asked to compute, from Darwin's photographs and Duchenne's electrodes through Paul Ekman's theory of basic emotions to Picard's research program and the startups it inspired. The second opens up the machinery of facial expression analysis: the action-unit coding system, the pipeline that turns pixels into labels, and the datasets that taught the machines what emotions supposedly look like. The third examines micro-expressions, the brief involuntary facial movements that underpin many lie-detection claims, and the computer-vision techniques built to catch them. The fourth turns to the voice, explaining how pitch, energy, timing and voice quality are measured and what they can and cannot convey. The fifth chapter confronts the science directly, setting out the evidence that challenges the assumption that a given expression signals a given emotion, and explaining what that means for systems trained on that assumption. The sixth and seventh follow the technology into two settings where the stakes are highest and the controversy most intense: the job interview and the border. The eighth asks how the lines should be drawn, examining the regulatory regimes now in force and proposing a way of judging any given deployment. The conclusion argues for a specific shift in how emotion AI should be thought about and governed. Throughout, the aim is to be precise about what the technology actually does. Emotion AI is neither magic nor pure fraud. It measures real things. The question this book keeps asking is what those measurements are entitled to mean, and who gets to decide. Chapter 1: How Emotion Became Computable Before a machine could be asked to read a feeling, someone had to decide that feelings were the kind of thing that could be read. That decision was not made by engineers. It was made over roughly a century and a half by anatomists, naturalists and psychologists, each of whom contributed a piece of a single, powerful idea: that emotions are discrete internal states, that each one produces a characteristic outward sign, and that those signs can be catalogued. Emotion AI inherited this idea almost whole. Understanding where it came from is the first step to understanding why the technology works the way it does, and why its critics believe it rests on sand. Electrodes and photographs In the late 1850s the French neurologist Guillaume-Benjamin Duchenne de Boulogne began applying electrical currents to the faces of his patients and photographing the results. One of his most frequent subjects was an elderly man with reduced facial sensation, who could tolerate the stimulation without discomfort. By triggering individual muscles, Duchenne produced expressions that looked, to the viewer, like surprise, terror, pain or joy. He published the photographs in 1862 in Mécanisme de la physionomie humaine, arguing that each emotion had its own muscular signature, written into the face by the Creator as a universal language. Duchenne's most durable finding concerned the smile. He observed that stimulating the muscle that pulls the corners of the mouth upward, the zygomatic major, produced something that looked like a smile but read as false. A genuine smile of enjoyment, he argued, also involved the orbicularis oculi, the ring of muscle around the eye that crinkles the skin at the outer corners. The eye muscle, he believed, could not easily be contracted at will. More than a century later psychologists would name the full version the "Duchenne smile", and the distinction between it and the merely polite smile would become one of the first things emotion recognition software was built to detect. Charles Darwin knew Duchenne's photographs and reproduced several of them in The Expression of the Emotions in Man and Animals in 1872. Darwin's interest was evolutionary. He wanted to show that human expressions had continuity with those of other animals, that they were inherited rather than learned, and that they had once served practical functions. The raised brows and widened eyes of surprise, for example, might originally have helped an animal see more of its surroundings. Darwin also ran what may have been one of the first systematic judgment studies of facial expression, showing Duchenne's photographs to guests and recording which emotion they named. Agreement was high for some images and weak for others, a pattern that would recur in every such study for the next hundred and fifty years. Darwin's book did not settle the matter. For much of the twentieth century, anthropologists such as Margaret Mead emphasized cultural variation in emotional display, and behaviorist psychology was suspicious of inner states altogether. The idea that facial expressions were universal and innate fell out of fashion. It was revived in the 1960s by two researchers whose work would later become the scientific backbone of an industry. Ekman and the basic emotions Silvan Tomkins, an American psychologist, proposed in the early 1960s that a small set of "affects" were hard-wired programs, each with its own facial display. His student Paul Ekman, together with Wallace Friesen, set out to test the claim across cultures. In a 1969 paper in Science with E. Richard Sorenson, Ekman reported that people in several literate societies agreed well above chance when asked to match photographs of posed facial expressions to emotion words. Critics objected that literate people might all have learned the same conventions from Western media. So Ekman and Friesen traveled to the highlands of Papua New Guinea to work with the Fore, a group with little exposure to outside media at the time. Using a method in which participants heard a short story and chose which of several faces fit it, they reported in 1971 that the Fore, too, matched faces to emotional situations at rates above chance, though with notable confusions, especially between fear and surprise. From this and related work came the doctrine of "basic emotions": a short list of discrete states, usually given as happiness, sadness, anger, fear, surprise and disgust, with contempt sometimes added. Each was held to be biologically based, to have a distinct physiological profile, and to produce a recognizable facial configuration across all human cultures. People could suppress or mask these expressions according to local "display rules", Ekman argued, but the underlying programs were universal, and under pressure they could leak out. The theory had two properties that made it extraordinarily attractive to later engineers. It was discrete, meaning that an emotion was one of a handful of categories rather than a blurry continuum, which is exactly the form a classification algorithm wants its outputs to take. And it was legible, meaning that each category had a visible signature on the face. If both properties held, then recognizing emotion was a pattern recognition problem of the same kind as recognizing handwritten digits. One needed only to collect labeled examples of each expression and train a classifier. Not every psychologist accepted the basic-emotion view, even then. A rival "dimensional" tradition, given its best-known form by James Russell in 1980, described emotional experience in terms of continuous dimensions, most commonly valence (how pleasant or unpleasant) and arousal (how activated or calm). In Russell's circumplex model, emotions are points arranged around a circle in this two-dimensional space rather than separate boxes. Dimensional models would later be widely used in emotion AI too, especially for voice, because a system can output a pair of numbers rather than commit to a single label. But it was the categorical, face-centered version that shaped public expectations and most commercial products. Coding the face Ekman's other enduring contribution was methodological. If facial expressions were to be studied scientifically, researchers needed a way of describing them that did not presuppose what they meant. Saying a face "looks angry" mixes observation with interpretation. In 1978 Ekman and Friesen published the Facial Action Coding System, known as FACS, which described faces purely in terms of the visible effects of muscle movements. Each distinguishable movement was given a number, called an action unit. Action unit 1 raises the inner corners of the brows; action unit 4 lowers and draws the brows together; action unit 6 raises the cheeks and crinkles the outer eye corners; action unit 12 pulls the lip corners upward. A revised edition, prepared with Joseph Hager, appeared in 2002. FACS was a real achievement. It gave researchers a shared, anatomically grounded vocabulary, and trained coders could achieve good agreement with each other. It allowed Duchenne's observation to be restated precisely: a smile involving action units 6 and 12 together, rather than 12 alone. Crucially for later developments, it also produced data in a structured form. A FACS-coded video is a time series of which action units are active and at what intensity. That is precisely the kind of representation a computer can be trained to produce. There is an important subtlety here, and it will recur throughout this book. FACS itself is descriptive. It says what the face did, not what the person felt. The link from action units to emotions came from separate tables, developed by Ekman and colleagues, that listed which combinations were thought to be prototypical for each basic emotion. Anger, for instance, was associated with lowered brows, raised upper lids, tightened lids and pressed or parted lips. The descriptive layer and the interpretive layer are separable, and much of the later controversy can be understood as a dispute about the interpretive layer while the descriptive layer remains broadly accepted. Ekman's work also reached far beyond academia. He trained law enforcement and intelligence personnel to read faces, developed online training tools for recognizing brief expressions, and served as scientific adviser to the television drama Lie to Me, which ran from 2009 to 2011 and whose hero solved crimes by reading micro-expressions. By the time machine learning was ready to tackle the face, a large part of the public, and many customers in government and business, already believed that faces revealed hidden truths to anyone with the right training. Picard and the affective computing program Rosalind Picard came to emotion from electrical engineering and computer vision, not psychology. Her 1997 book Affective Computing made a case that was at once technical and philosophical. Emotion, she argued, was not the opposite of rational thought but a component of it; drawing on the neuroscientist Antonio Damasio's studies of patients whose emotional processing was damaged and who consequently made poor decisions, she suggested that intelligent machines might need something analogous to emotion to function well. More practically, she argued that computers that interacted with people would be more useful if they could tell when a user was confused, frustrated or bored, and respond accordingly. Picard was careful to frame recognition as inference rather than detection. A computer could observe "expressions" of emotion, such as facial movement, vocal patterns, posture and physiological change, but the underlying state was hidden, and the machine's task was to make probabilistic guesses about it from incomplete evidence, much as people do. She also anticipated several of the ethical issues that would later explode: consent, privacy, and the risk of manipulation by systems that understood users' emotional vulnerabilities better than users understood themselves. Her group at the MIT Media Lab became a center for the field. Early projects included wearable sensors that measured skin conductance, a marker of physiological arousal; tools to help people on the autism spectrum interpret others' expressions; and systems that tracked learners' affective states during tutoring. Much of this work was oriented toward helping individuals understand and manage their own states, with their participation and consent. A different path: the body One strand of Picard's work is worth following a little further, because it shows what affective computing looks like when it stays close to the body and to the interests of the person being measured. Her group spent years developing wrist-worn sensors that recorded electrodermal activity, the small changes in the skin's electrical conductance driven by the sympathetic nervous system. Skin conductance rises with physiological arousal of almost any kind: excitement, fear, exertion, pain, concentration. It does not say which of these is occurring. What it offers is a continuous, involuntary record of how activated the body is, gathered over hours or days rather than in a single snapshot. The project took an unexpected turn. In one widely recounted episode, Picard examined data from a sensor worn by a child with autism and found an enormous spike in skin conductance on one wrist. It turned out to have occurred shortly before a seizure. Following that lead, her group and collaborators investigated whether certain seizures, especially generalized tonic-clonic seizures, produced characteristic physiological signatures that a wearable could detect. The company Empatica, which Picard co-founded, developed a wristband called Embrace that combined electrodermal and motion sensing to detect such seizures and alert caregivers. In 2018 it received clearance from the United States Food and Drug Administration as a medical device for this purpose. The seizure detector is an instructive counterpoint to the emotion recognition industry that grew up alongside it. It measures a physiological event with a clear definition and a clinically verifiable ground truth: either a seizure happened or it did not. It was validated against that ground truth in clinical studies. It serves the person wearing it and their family, who chose to use it. And it makes no claim about what anyone feels. It began in affective computing, but by the time it became a product it had shed the inferential leap that makes emotion recognition so contentious. Much of the argument of this book can be read as a case for why more of the field should follow that path. From laboratory to market In 2009 Picard and Rana el Kaliouby, a computer scientist who had developed a system for reading mental states from faces during her doctoral work at Cambridge, co-founded Affectiva as a spin-out from the Media Lab. The company's early commercial success came not from health applications but from advertising and market research. Brands wanted to know how audiences reacted to their commercials moment by moment, and Affectiva could provide this by recording volunteers' faces through their webcams as they watched and scoring smiles, brow furrows and attention. The company built what it described as one of the world's largest collections of facial videos, gathered from viewers in many countries, and later moved into automotive driver monitoring. In 2021 it was acquired by Smart Eye, a Swedish company specializing in in-car eye tracking. Affectiva was one of many. Through the 2010s a crowded market emerged: firms such as Realeyes and nViso in advertising analytics, Beyond Verbal and others in voice, and large platform providers including Microsoft, Amazon and Google offering emotion labels as part of general-purpose face analysis services. A developer could send a photograph to a cloud endpoint and receive back a set of scores for happiness, sadness, anger, fear, surprise, disgust, contempt and neutral, the basic-emotion list almost exactly as Ekman had formulated it decades earlier. That continuity is the central fact of this history. The categories that commercial systems output were not discovered by machine learning. They were inherited from a particular psychological theory and built into the datasets, labeling schemes and product interfaces before any algorithm was trained. When a cloud API reported that a face was "73 percent happy", that number expressed the classifier's confidence that the image resembled training images labeled "happy". Whether the person in the photograph was in fact happy was a separate question that the system had no way to answer. Some of the field's leading figures came to say as much. Picard herself has long emphasized that recognition systems infer rather than read emotion and that context is essential. Yet the products and the marketing often collapsed this distinction. "Emotion detection" and "emotion recognition" are both phrases that imply something already present being found, the way a metal detector finds metal. A more honest name for most of what these systems do would be "expression classification", with emotion as a further, uncertain inference. That relabeling may sound pedantic. As later chapters will show, the entire ethical and legal debate turns on it. Three assumptions By the time deep learning arrived in the early 2010s, the project of computing emotion rested on three assumptions, usually unstated. The first was that emotions come in a small number of discrete kinds that are the same for everyone. The second was that each kind produces a characteristic, observable pattern in the face and voice, consistently enough that the pattern can stand in for the state. The third was that these patterns are similar enough across individuals, cultures and situations that a system trained on one population can be applied to another. Each assumption had some support. Each also had serious, documented problems, and the evidence against the second and third in particular grew substantially over the following decade. But the assumptions were baked into the data before the critiques gained wide attention, and an industry was built on top of them. The rest of this book examines how the technology works in detail, what its measurements actually capture, and what happens when the assumptions are tested in settings where getting it wrong can cost someone a job or their freedom to cross a border. Chapter 2: Reading the Face Strip away the marketing language and an automated facial expression system is a sequence of fairly ordinary computer-vision steps, each of which narrows a large, noisy image down to a small number of values. Knowing what those steps are, and where each can go wrong, makes it much easier to judge what the final number means. It also reveals something that is rarely stated plainly: almost all of the system's notion of emotion comes from the data it was trained on, and that data was produced by people following particular instructions in particular settings. From pixels to landmarks The first task is to find the face. For years the standard method was the detector published by Paul Viola and Michael Jones in 2001, which scanned an image with a cascade of simple brightness-contrast filters and could run in real time on modest hardware. It worked well on frontal, well-lit faces and poorly on tilted or partly hidden ones. Modern systems use convolutional neural networks trained on large collections of faces in varied poses, which are far more robust, though they still struggle with extreme angles, heavy shadows and partial occlusion. Once a face is found, the system locates landmarks: fixed reference points such as the corners of the eyes, the tip of the nose, the edges of the lips and the line of the jaw. A widely used convention places 68 such points, and many systems use dense meshes with hundreds. The landmarks serve two purposes. They allow the face to be aligned, rotated and scaled to a standard frontal position so that the same expression looks the same regardless of where the person was sitting. And their relative positions are themselves informative. The distance between the inner brow points and the upper eyelids shrinks when the brows are lowered; the angle formed by the lip corners and the center of the mouth changes in a smile. Alignment matters more than it might seem. A person who turns their head slightly to the side will appear, in the raw image, to have one eye narrower than the other and a mouth that slants. Unless the system corrects for pose, it may read asymmetry that is purely geometric as a meaningful expression. Head pose estimation is therefore a standard component, and many systems discard or down-weight frames in which the head is turned too far. Two kinds of output At this point the pipeline divides, and the division is the most important technical distinction in facial expression analysis. One branch estimates action units. Following the Facial Action Coding System introduced in Chapter 1, the system outputs, for each frame, whether each of a set of action units is present and often how intense it is. A well-known open-source tool, OpenFace, developed by Tadas Baltrušaitis and colleagues, does this for around eighteen commonly used action units, alongside head pose and eye gaze. The output is descriptive: brow lowerer at moderate intensity, lip corner puller at low intensity, and so on. It makes no claim about what the person feels. The other branch outputs emotion categories or dimensions directly. A classifier, usually a deep neural network, takes the aligned face image and returns a probability for each of a fixed set of labels, commonly the six or seven basic emotions plus "neutral". Alternatively, it may output continuous estimates of valence and arousal. This branch skips the descriptive layer entirely. The network learns whatever visual features best predict the labels in its training data, which may include action units but may also include lighting, image quality, the presence of glasses, or anything else that happens to correlate with the labels in that dataset. Commercial products often combine both, using action units internally and then mapping them to emotion labels through rules or a second classifier. The distinction still matters, because the reliability of each layer is very different. Detecting whether the lip corners moved upward is a perception problem with a clear ground truth that trained human coders can verify. Deciding whether the person was happy is an inference problem whose ground truth, as later chapters show, is much harder to establish. How features are learned Before about 2013, expression classifiers relied on features designed by hand. Researchers applied filters known to respond to edges and textures, such as Gabor filters, or computed statistics like local binary patterns, which summarize how each pixel's brightness compares to its neighbors. Wrinkles, furrows and bulges show up as distinctive textures, so these features captured much of what changes when facial muscles contract. A conventional classifier such as a support vector machine then learned to map the feature values to labels. Deep learning replaced this hand engineering with convolutional networks that learn their own features from data. Early layers end up detecting edges and simple textures, and later layers respond to larger structures such as the shape of an eye region or the configuration of the mouth. Given enough labeled examples, these networks outperformed hand-built features on almost every benchmark. More recent systems use transformer architectures and models pretrained on vast general image collections and then fine-tuned for expression, which helps when expression-labeled data is limited. Video adds time. A still image of a half-formed smile is ambiguous; a video shows whether it is growing, fading, or flickering. Temporal models, ranging from simple averaging across frames to recurrent and attention-based networks, use the sequence of frames to estimate onset, peak and offset of expressions. Timing matters to the theory as well: researchers have long argued that spontaneous expressions unfold differently from deliberate ones, with posed smiles often reaching their peak faster and more symmetrically. Systems that attempt to distinguish genuine from posed expressions rely heavily on such temporal cues. The datasets that taught the machines Every supervised classifier learns its idea of each category from labeled examples, so the datasets are where the scientific assumptions of the field are most concretely encoded. Their history follows a clear arc, from small collections of actors performing exaggerated expressions in laboratories toward very large collections of images scraped from the internet and labeled by crowd workers. The main datasets differ in exactly the respects that determine what a model trained on them can learn, as Table 1 sets out. Table 1. Influential facial expression datasets and what they contain. Dataset Year Source of faces Expressions Labels Cohn-Kanade Extended (CK+) 2010 123 lab participants, 593 sequences Mostly posed on instruction FACS action units; seven emotion categories for a subset DISFA 2013 27 adults watching video clips Spontaneous Action unit intensities, frame by frame FER2013 2013 About 35,900 small greyscale web images Mixed, mostly unknown Seven categories from keyword search, then filtered AffectNet 2017 Around one million web images, roughly 440,000 hand-labeled Mixed, "in the wild" Eight categories plus valence and arousal Aff-Wild2 2019 Hundreds of YouTube videos Spontaneous, uncontrolled Valence and arousal, categories, action units Source: dataset papers by Lucey et al. (2010), Mavadati et al. (2013), Goodfellow et al. (2013), Mollahosseini et al. (2017) and Kollias and Zafeiriou (2019). The early laboratory datasets were invaluable for developing methods, but they had an obvious weakness. In CK+, participants were asked to perform expressions, often with guidance about which muscles to move, and the sequences run from neutral to a clear peak. These are caricatures of emotional display, the facial equivalent of an actor projecting to the back row. Models trained on them could achieve near-perfect accuracy on held-out examples from the same dataset and then fail badly on ordinary faces, because ordinary faces in daily life rarely look like that. The response was to collect faces "in the wild", meaning outside the laboratory. FER2013, assembled for a machine-learning competition, gathered images by running web searches for emotion-related keywords and then filtering the results. AffectNet scaled the same idea up enormously, querying search engines with over a thousand emotion-related terms in several languages and having human annotators label a large subset. These datasets are far more varied in pose, lighting, age and context, and models trained on them generalize better to real-world images. But they introduced a different problem. Their labels say what an annotator perceived when looking at a face out of context, which is not the same as what the person was feeling. A photograph returned for the query "furious" might show an actor, a stock-photo model, a sports fan mid-celebration, or a toddler mid-tantrum. The annotator sees a cropped face and picks the closest category. Agreement between annotators on such images is far from perfect; the creators of FER2013 reported that human accuracy on their own dataset was only around two-thirds. What the model learns, therefore, is to predict how a typical annotator would label a stranger's face in isolation. That is a well-defined task. It is also a long way from emotion recognition. Accuracy, and accuracy at what Vendors often quote high accuracy figures, and those figures are usually true in a narrow sense. On posed laboratory datasets, modern classifiers exceed ninety-five percent accuracy. On large in-the-wild collections such as AffectNet, the best published eight-category results have been considerably lower, in the region of sixty percent, which partly reflects how ambiguous and noisy the labels are. Action-unit detection performs well for large, frequent movements such as smiles and brow raises and markedly worse for subtle or infrequent ones. These numbers measure agreement with the dataset's labels. They do not measure whether the system correctly identifies what people feel, because none of these datasets contains reliable information about what people felt. Some research datasets do try to link expressions to experienced emotion, for instance by recording participants as they watch film clips selected to induce particular states and then asking them to report how they felt. These studies are valuable, and they consistently show a messier picture than the posed datasets suggest: people watching an amusing clip often smile, but many do not; people watching a disgusting one sometimes wrinkle their noses, sometimes look away, sometimes laugh nervously; and self-reported feelings and facial behavior frequently diverge. There is also the question of who is in the data. Large web-scraped collections reflect the demographics of the images that search engines return, which skew toward certain ages, ethnicities and photographic conventions. Laboratory datasets often draw on university students. When the training data underrepresents a group, error rates for that group tend to be higher. The 2018 "Gender Shades" study by Joy Buolamwini and Timnit Gebru showed this vividly for commercial face analysis services classifying gender, which erred far more often on darker-skinned women than on lighter-skinned men. Expression classifiers share the same detection and alignment stages and inherit the same vulnerabilities, along with additional ones: cultural differences in expressive style, differences in resting facial structure that can be misread as expression, and differences in how annotators perceive faces of groups other than their own. Anatomy of a confidence score It is worth pausing on the number that most users of these systems actually see, because it is widely misread. When an emotion classifier reports that a face is "92 percent happy", the figure usually comes from the final layer of a neural network, where a mathematical function called softmax converts the network's raw outputs into values between zero and one that sum to one across the available categories. The result looks like a probability distribution. It is better understood as a ranking of how strongly the image resembles each category in the training data, relative to the other categories on offer. Three features of this arrangement matter. First, the categories are forced. A classifier with seven emotion labels and "neutral" must distribute its confidence among those eight options, even when the person is concentrating, squinting into the sun, chewing, talking, or feeling something for which there is no label at all, such as nostalgia, boredom or determination. There is no category for "not one of these" unless the designers added one, and few do. A face mid-sentence, with lips parted and brows raised for emphasis, will be assigned to whichever category it happens to resemble most, often surprise or fear. Second, the scores are frequently poorly calibrated. Deep networks are known to be overconfident, assigning high scores to predictions that turn out to be wrong much more often than the scores imply. A score of 0.92 does not mean that, across many similar faces, the label is right ninety-two times in a hundred. Calibration can be corrected with additional techniques, but vendors rarely publish calibration data, and the problem worsens on images unlike the training set, which is exactly where deployed systems most often operate. Third, the number refers to a single frame or a short window. Expressions change constantly during conversation, so a system processing video produces a stream of fluctuating scores. Somewhere a design decision is made about how to summarize that stream: averaging over the whole clip, counting the frames above a threshold, reporting the peak. These choices are rarely visible to users, yet they can change the result substantially. A candidate who smiles warmly at the start and end of an answer but frowns in concentration through the middle might be summarized as mostly happy or mostly negative depending on which aggregation the vendor chose. None of this is hidden in the technical literature. It simply disappears at the interface, where a single number with a percentage sign invites readers to treat it as a measurement of the person. Faces that do not fit Beyond demographic imbalance in the training data, several kinds of face systematically confound expression classifiers. Older adults have deeper wrinkles and more lax skin around the eyes and mouth, which can resemble the textures produced by muscle contractions associated with sadness or disgust. Children's faces are proportioned differently and are underrepresented in most datasets. Some people have a resting facial configuration, a downturned mouth or heavy brows, that classifiers read as negative emotion even when the face is completely relaxed; the popular phrase "resting face" describes a perception that machines reproduce. Facial hair, cosmetic procedures, scars, glasses, headscarves and face masks all alter or hide the features classifiers rely on. The widespread wearing of masks during the COVID-19 pandemic exposed this dramatically, as systems trained on full faces had to work from the eye region alone, where many emotion categories become hard to separate. The common thread is that classifiers learn the statistical regularities of their training faces, and any face that departs from those regularities for structural rather than expressive reasons will be misread. The misreadings are not random. They track age, body, culture, religion and disability, which is why they matter so much when the output is used to make decisions about people. What the face analysis actually delivers It helps to be concrete about what an honest description of a facial expression system's output would look like. For a given stretch of video, a good system can report with reasonable reliability that the person's face was oriented toward the camera, that the lip corners rose at a certain moment and stayed raised for two seconds, that the cheeks lifted at the same time, that the brows were drawn together briefly a few seconds later, and that the eyes were closed more often than at the start. These are measurements of behavior, and they can be validated against careful human coding. It can also report that, if a group of annotators were shown a still from the moment of peak lip-corner raise, most of them would probably label it "happy". That is a prediction about human perception, also testable. What it cannot report, without further evidence that it does not have, is that the person was happy, or amused, or pleased with the question, or confident in the answer. Each of those claims requires knowing something about the person's inner state and about the situation, and each would be false a substantial fraction of the time. People smile when embarrassed, when uncomfortable, when being polite to an interviewer, and when trying to conceal distress. They frown when concentrating, when the light is in their eyes, and when trying to recall a date. The machinery described in this chapter is genuinely capable, and it is improving. Its outputs at the descriptive level are useful for research, for animation, for accessibility and for safety monitoring. The trouble begins when its outputs are relabeled from "what the face did" to "what the person felt", a relabeling that happens not inside the algorithm but in the interface, the sales pitch and the decision that follows. The same pattern appears, in an even sharper form, in the attempt to catch emotions that people are trying to hide. Chapter 3: The Fleeting Face Of all the claims made for facial expression analysis, the most seductive is that the face betrays what a person is trying to hide. The idea has a name, the micro-expression, and a compelling story: when someone conceals a strong emotion, the true feeling flashes across the face for a fraction of a second before the mask reasserts itself, too fast for most observers to notice but there for anyone trained, or any camera fast enough, to catch. If the story were true, it would make emotion AI a kind of lie detector. Much of the money and political interest in the field, especially in security, has flowed from that possibility. This chapter examines where the idea came from, how computer vision has tried to operationalize it, and what the evidence says about whether it can do what its promoters claim. Origins of the micro-expression The phenomenon was first described in 1966 by Ernest Haggard and Kenneth Isaacs, two psychologists studying filmed psychotherapy sessions. Reviewing the footage frame by frame, they noticed facial expressions that appeared and vanished so quickly they were invisible at normal playback speed. They called these "micromomentary" expressions and speculated that they reflected repressed or conflicted feelings. Paul Ekman and Wallace Friesen encountered something similar a few years later while studying a film of a psychiatric patient who had concealed suicidal intentions in order to obtain a weekend pass from hospital. Slowing the film down, they found a brief expression of anguish in the middle of an otherwise cheerful interview. In a 1969 paper on "nonverbal leakage and clues to deception", they proposed that micro-expressions were involuntary leaks of concealed emotion, and that they might be detectable by trained observers. Ekman later described them as lasting from about a twenty-fifth to a fifth of a second, though definitions in the research literature vary, and many studies use half a second as an upper bound. From the 1970s onward Ekman developed training materials to help people spot these flashes. The best known, the Micro Expression Training Tool, presents faces that shift briefly from neutral to a full emotional expression and back, and asks the learner to name the emotion. People do improve with practice on this task. What the training does not establish, and what turned out to be the crux, is whether the same expressions occur in real deception at a useful rate, and whether spotting them helps to tell liars from truth-tellers. The evidence on frequency The most frequently cited empirical test came from Stephen Porter and Leanne ten Brinke, who published a study in Psychological Science in 2008. They asked participants to view emotionally evocative images and to respond with genuine, simulated or masked expressions, and they coded around seven hundred expressions frame by frame. Micro-expressions, defined by their brevity, were rare: roughly two percent of the expressions qualified, and those that did were usually partial, appearing only in the upper or lower half of the face rather than as the full prototypical display. Importantly, they occurred in genuine as well as deceptive conditions. Later work by the same group examined real, high-stakes cases: videos of people making televised public pleas for the return of missing relatives, some of whom were later shown to have been responsible for the relative's disappearance or death. Here again micro-expressions were rare and appeared among both the deceptive and the genuine pleaders, at broadly similar rates. Other cues, such as the activation of particular muscles associated with grief or with masked happiness, showed some differences between groups, but these were statistical tendencies across many cases rather than signs that would reliably classify an individual. These findings fit a much larger body of research on deception detection. A meta-analysis by Charles Bond and Bella DePaulo in 2006, pooling results from hundreds of experiments, found that people judging whether others were lying achieved an average accuracy of about fifty-four percent, only slightly better than a coin toss. Professionals such as police officers did not perform much better than students. Reviews of the behavioral cues to deception, notably by DePaulo and colleagues in 2003, found that most cues popularly believed to indicate lying, including gaze aversion and fidgeting, were weakly related to deception or not related at all. The communication scholar Judee Burgoon, who has worked on automated deception detection for decades, wrote in 2018 that micro-expressions are "not the best way to catch a liar", arguing that they are too infrequent and too easily produced by honest people under stress to serve as reliable indicators. None of this means micro-expressions do not exist. Brief facial movements certainly occur, and some may reflect emotional responses people are trying to suppress. The problem is that they are uncommon, often incomplete, and do not map cleanly onto lying. An honest person who is frightened of being disbelieved can show a flash of fear; a practiced liar may show nothing at all. Where deception research went instead As evidence accumulated that emotional leakage was a weak basis for detecting lies, much of the academic field of deception research changed direction. Rather than looking for fleeting signs of fear or guilt, researchers began to focus on the cognitive demands of lying, and on how interviewers could deliberately increase those demands to make differences between liars and truth-tellers easier to observe. The reasoning is straightforward. Telling a convincing lie is often harder work than telling the truth. The liar must invent a story, keep it consistent, suppress the truth, monitor the listener's reaction, and try to appear sincere, all at once. Aldert Vrij, Pär Anders Granhag and their colleagues developed interviewing techniques based on this idea: asking people to tell their story in reverse chronological order, asking unexpected questions that a prepared liar would not have rehearsed, asking for verifiable details, or strategically withholding and then revealing evidence to see whether the account shifts. Experiments using these methods generally report accuracy somewhat better than the near-chance levels found for unaided judgments, though results vary and laboratory conditions again differ from the field. The important feature of these approaches for present purposes is that they shift attention from the face to the content of what is said and how it changes under questioning. The cues they rely on, such as the amount and verifiability of detail, consistency with known evidence, and the ability to adapt the story, are about information, not emotion. They are also used within an interview conducted by a person who knows the context and can follow up, rather than as a score produced by a machine from a short recording. Burgoon's own proposal in her 2018 paper points the same way. She drew attention to what she called a rigidity effect: in several studies, people lying under pressure tended to move less, with reduced head and body movement and more controlled faces, presumably because they were concentrating on managing their behavior. This is almost the opposite of the leakage hypothesis, in which concealed emotion bursts out. But Burgoon presented it as a tendency observed across groups in particular experimental settings, not as a reliable sign that an individual is lying. A nervous, rigid honest person and a relaxed, animated liar both exist, and no single behavioral cue sorts them. The training of human observers tells a similar story. Ekman's training tools do improve people's ability to recognize the brief expressions presented in the training materials. Whether that improves their ability to detect lies in real interviews is far less clear, and independent evaluations have generally not found meaningful gains in deception detection from micro-expression training alone. A skill that transfers well to recognizing the stimuli it was trained on, and poorly to the task it was meant to serve, is a familiar pattern in applied psychology. It is the same pattern that machine micro-expression systems reproduce at greater scale. Building machines to catch them Computer-vision researchers took up micro-expression analysis in earnest around 2010, and the field developed its own datasets, methods and annual competitions. The technical problem is genuinely hard, which partly explains its appeal to researchers. Micro-expressions are brief, so they appear in only a handful of frames of standard video; they are low in intensity, so the pixel changes are tiny; and they must be separated from blinks, head movements, speech and ordinary expressions. Dataset construction faced a basic obstacle: how to get people to produce genuine micro-expressions on camera. The usual solution is a suppression paradigm. Participants watch emotionally powerful video clips while being told to keep a neutral face, sometimes with an incentive for success or a penalty for failure. High-speed cameras record them, and trained coders then search the footage for brief leaks. The Chinese Academy of Sciences produced the CASME series of datasets; CASME II, published in 2014, recorded participants at two hundred frames per second and contained around two hundred and fifty micro-expression samples. The SMIC dataset from the University of Oulu and the SAMM dataset from Manchester Metropolitan University followed similar designs. Even the largest of these collections contain only a few hundred examples, gathered from a few dozen people, mostly young adults in a single country. Two families of techniques dominate. The first, motion analysis, tries to measure the tiny movements directly. Optical flow methods estimate how each small patch of the face moved between frames, producing a field of motion vectors that can reveal a brief tightening at the mouth corner or a slight lift of the inner brows. A related technique, Eulerian video magnification, developed at MIT in 2012, amplifies small temporal variations in video so that they become visible, and has been used to exaggerate subtle facial movements before classification. The second family uses spatiotemporal texture descriptors, such as an extension of local binary patterns to three orthogonal planes that captures how texture changes over time as well as space. Since around 2017, deep networks taking optical flow or frame sequences as input have largely replaced handcrafted methods. Two separate tasks are usually distinguished. Spotting is finding when a micro-expression occurs in a longer video; recognition is classifying what emotion it represents, given that one has been found. Spotting is notoriously difficult, with high false-positive rates, because many brief facial movements are not micro-expressions in the intended sense. Recognition performance in the research literature is often reported on the same small datasets used for training, with cross-validation schemes that leave out one participant at a time. Results look reasonable within those datasets and degrade when systems are tested on a different dataset collected with a different camera and population. From the lab to the checkpoint The gap between what this research establishes and what has been claimed for it is wide. The datasets record people suppressing reactions to film clips in a quiet laboratory. They are labeled according to which clip was playing and which facial movements coders observed, not according to whether the person was lying about anything. No standard micro-expression dataset contains deception in the sense that matters for security: a person concealing a hostile intention, a smuggled item, or a false identity in a high-stakes interview. Yet security applications have repeatedly invoked micro-expressions. The Transportation Security Administration in the United States trained behavior detection officers, starting in the mid-2000s, using an approach influenced by Ekman's work on facial and behavioral cues, as Chapter 7 discusses. The European research project iBorderCtrl described its deception module as analyzing "micro-gestures" of travelers. Commercial vendors have marketed micro-expression analysis to insurers, police forces and employers. In each case the scientific step from "brief facial movement detected" to "person is concealing something relevant" was asserted rather than demonstrated. A thought experiment shows why the step is so fraught. Suppose a micro-expression detector at an airport were reasonably good: suppose that among travelers who were genuinely concealing something serious, it flagged eighty percent, and among ordinary travelers it flagged only five percent. Suppose further that one traveler in ten thousand is concealing something serious. Out of a million travelers, the system would correctly flag about eighty of the hundred concealers. It would also flag five percent of the 999,900 innocent travelers, which is nearly fifty thousand people. Of everyone flagged, fewer than one in six hundred would actually be a concealer. Real micro-expression detectors have never been shown to approach the accuracy assumed here in a field setting, and the honest base rate of serious concealment at a border is unknown but certainly low. The arithmetic is unforgiving: even a detector far better than any that exists would bury its rare correct alerts in a mass of false alarms, each of which falls on an actual person who may then be questioned, searched or delayed. The Othello error Ekman himself identified one of the most serious hazards in reading emotion as evidence of lying, and gave it a memorable name. In Telling Lies, first published in 1985, he described the "Othello error", after Shakespeare's tragedy. Othello accuses Desdemona of infidelity; she, knowing she cannot prove her innocence and seeing that he will not believe her, shows fear and distress. Othello reads her fear as the guilt of a woman caught out, and kills her. The error is to mistake the emotions of an honest person who fears being disbelieved for the emotions of a liar who fears being caught. The Othello error is not an occasional mishap. It is built into the structure of any situation in which someone is being assessed for deception and knows it. A truthful traveler questioned at a border, a truthful suspect in a police interview, or a truthful candidate challenged about a gap in their employment history may all feel fear, embarrassment or anger at being suspected. Those emotions will show, to whatever extent emotions show at all, in the face and voice. A system or an observer looking for signs of concealed emotion will find them, and will find them more readily in people who are more anxious about authority, more conscious of being outsiders, or more aware of the consequences of being disbelieved. Those are often precisely the people with the least power in the encounter. Ekman's own advice was that emotional signs should be treated as "hot spots" warranting further inquiry, never as proof of lying. That advice is sound. It is also exactly what tends to be lost when emotional cues are automated, scored and passed to someone who must make a quick decision. Why the story persists Given this evidence, it is reasonable to ask why the micro-expression has remained so influential. Part of the answer is that it fits a deep intuition. People feel that their own emotions sometimes leak despite their efforts, and they remember occasions when they glimpsed another person's real reaction. The micro-expression offers a scientific-sounding confirmation of that experience and the promise of systematizing it. Part of the answer is institutional. Security agencies are under pressure to detect threats that cannot be found by searching luggage, and a method that claims to read intent from behavior addresses that pressure directly. Once training programs and procurement contracts exist, they generate their own momentum, and the absence of clear evidence of effectiveness can be read as evidence that the method is working quietly. And part of the answer is that the technology is real. High-speed cameras do capture fleeting movements; algorithms do find them. The demonstrations are impressive. What is missing is not the ability to see the face move but any reliable rule connecting that movement to the hidden fact that matters. That gap, between a measurable signal and the meaning ascribed to it, is the pattern this book keeps encountering. It appears again, in a different form, when the machines turn from the face to the voice. Chapter 4: The Sound of Feeling People recognize emotion in voices with remarkable speed. A single word on the telephone from a close friend can be enough to tell that something is wrong. Much of this information is carried not by what is said but by how it is said: the pitch, the loudness, the pace, the hesitations, the roughness or breathiness of the sound. Linguists call this layer of speech prosody, and for affective computing it has two great attractions. It is captured by any microphone, including the billions already present in phones, laptops, cars, smart speakers and call-center headsets. And unlike the face, it does not require the person to be looking at a camera. A voice can be analyzed in the dark, over a bad line, or from a recording made months ago. This chapter explains what speech emotion recognition measures, how the measurements are turned into judgments, and why the voice, though in some ways a better channel than the face, suffers from the same fundamental gap between signal and meaning. What prosody consists of The human voice is produced when air from the lungs passes through the vocal folds in the larynx, setting them vibrating. The rate of vibration determines the fundamental frequency, abbreviated F0, which listeners hear as pitch. Typical adult speaking F0 is lower for most men than for most women and children, because of differences in the length and mass of the vocal folds. Above the larynx, the throat, mouth and nasal cavities act as a filter, amplifying some frequencies and damping others, which produces the resonances, called formants, that distinguish one vowel from another and give each voice its characteristic color. Emotion and physiological arousal affect this system at several points. Tension in the muscles of the larynx raises pitch and can make the voice sound strained. Faster, shallower breathing changes loudness and phrasing. Changes in salivation and muscle tone alter articulation. Over the course of an utterance, these effects show up in a handful of measurable dimensions. The first is pitch: its average level, its range, and its contour, meaning how it rises and falls across a phrase. The second is energy or intensity, heard as loudness, along with how it varies. The third is timing: speaking rate, the length of pauses, the duration of syllables, and the rhythm of stressed and unstressed elements. The fourth is voice quality, a more technical family of measures describing the texture of the sound. These include jitter, the small cycle-to-cycle irregularity in the period of vibration; shimmer, the corresponding irregularity in amplitude; and the harmonics-to-noise ratio, which captures how breathy or hoarse a voice is. Spectral measures, describing how energy is distributed across frequencies, fill out the picture. Decades of research, much of it associated with Klaus Scherer at the University of Geneva and his colleagues, have documented how these features tend to vary with emotional state. A widely cited 2003 meta-analysis by Patrik Juslin and Petri Laukka pulled together more than a hundred studies of vocal emotion expression and found consistent broad patterns, especially along the dimension of arousal. The main features and their typical associations are set out in Table 2. Table 2. Principal acoustic features used in speech emotion recognition. Feature group What is measured Typical pattern in high-arousal states Typical pattern in low-arousal states Pitch (F0) Mean level, range, contour Higher, wider range, more variable Lower, narrower, flatter Energy Mean loudness and variability Louder, more variable Quieter, more even Timing Speech rate, pause length Faster, shorter pauses Slower, longer pauses Voice quality Jitter, shimmer, breathiness Often tenser, sharper onsets Often breathier or laxer Spectrum Balance of high and low frequency energy More energy in higher frequencies Energy concentrated lower Source: summarized from Juslin and Laukka (2003) and Eyben et al. (2016). The pattern in the table is about arousal, not about which specific emotion is present. That is not an accident of presentation; it reflects one of the most robust findings in the field. Anger, fear and joy are all high-arousal states, and acoustically they share much: raised pitch, increased loudness, faster tempo. Sadness and boredom are both low-arousal and share a quieter, slower, flatter profile. Distinguishing between emotions of similar arousal, such as anger and joy, or sadness and calm contentment, is much harder from acoustics alone, and depends on subtler cues that vary considerably from person to person and from language to language. From features to classifiers Early speech emotion recognition systems computed long lists of acoustic statistics for each utterance, sometimes thousands of them, and fed them into conventional classifiers. The proliferation of features made results hard to compare. In 2016 Florian Eyben and colleagues proposed a standardized minimal set, the Geneva Minimalistic Acoustic Parameter Set, along with an extended version known as eGeMAPS containing 88 parameters, chosen for their theoretical relevance and their track record in predicting affect. Implemented in the open-source openSMILE toolkit, this set became a common baseline. As with faces, deep learning changed the picture. Networks can now take a spectrogram, an image-like representation of how frequency content changes over time, or even the raw waveform, and learn their own features. Since around 2020, the most effective systems have been built on large self-supervised speech models, such as wav2vec 2.0 and HuBERT, which learn general-purpose representations of speech from tens of thousands of hours of unlabeled audio and are then fine-tuned on a smaller emotion-labeled set. These representations capture a great deal about a speaker: their identity, their accent, the recording conditions, and the prosodic cues that relate to affect. Systems can output categories, typically a small set such as angry, happy, sad and neutral, or continuous dimensions such as arousal, valence and sometimes dominance. Arousal is predicted much more accurately than valence from voice alone, reflecting the acoustic overlap described above. Many modern systems combine acoustic analysis with the words themselves, using speech recognition to transcribe and a language model to read the text, which dramatically improves valence estimates. Someone saying "I'm thrilled" in a raised voice is probably not angry. The data problem, again The voice faces the same data problem as the face, in some ways more acutely. The best-known research corpora were recorded by actors. The Interactive Emotional Dyadic Motion Capture database, known as IEMOCAP and released by the University of Southern California in 2008, contains about twelve hours of audiovisual recordings of ten actors performing scripted and improvised scenes designed to elicit emotions. The Ryerson Audio-Visual Database of Emotional Speech and Song, RAVDESS, published in 2018, has twenty-four actors speaking two fixed sentences in eight emotional styles. These corpora are carefully made and enormously useful for developing methods, but acted emotion tends to be more intense and more stereotyped than everyday speech. Naturalistic collections exist, built from talk shows, podcasts, call-center recordings and field interviews, with annotators listening and rating the emotion they perceive. The MSP-Podcast corpus from the University of Texas at Dallas is a prominent example. Performance on such data is markedly lower than on acted speech, and agreement among the human annotators themselves is often modest. When a system is trained on one corpus and tested on another, accuracy typically drops sharply, a phenomenon well documented in the research literature since the first international speech emotion challenge at the Interspeech conference in 2009. Language and culture compound the problem. Prosody carries linguistic as well as emotional information. In tone languages such as Mandarin, Cantonese or Yoruba, pitch movements distinguish words, constraining how pitch can be used for emotional expression. Even among non-tonal languages, conventions differ: the pitch range regarded as calm in one language community may sound excited or flat in another. Studies of cross-cultural vocal emotion recognition generally find that people recognize emotions in voices from other cultures above chance, but less accurately than within their own, an "in-group advantage" that machine models trained on one population would be expected to share. Individual variation matters too. A person with a naturally high or monotone voice, a stammer, a hearing impairment, a regional or non-native accent, or a medical condition affecting the larynx will produce acoustic profiles that differ from the training data for reasons that have nothing to do with emotion. Depression, Parkinson's disease and some medications flatten prosody. Fatigue, a cold, or a poor microphone all leave traces that a classifier may misread as affect. Where voice analysis is used Because voice analysis can run invisibly on audio that is already being captured, it has spread into settings where people may not realize they are being assessed. Contact centers are the largest market. Companies such as Cogito have offered real-time analysis of customer service calls that monitors both agent and customer voices and prompts the agent on screen, for example to slow down, to show more empathy, or to note that the customer sounds frustrated. Other systems score calls after the fact for quality assurance, flagging conversations with signs of anger for supervisor review. Consumer products have experimented with the idea as well. Amazon's Halo fitness band, launched in 2020, included a "Tone" feature that analyzed the wearer's voice throughout the day and reported how they sounded to others, using descriptors such as "warm", "irritated" or "confident"; the product line was discontinued in 2023. In 2021 Spotify was granted a United States patent describing the use of speech signals to infer a user's emotional state, gender, age and accent in order to tailor recommendations, prompting an open letter from the digital rights group Access Now urging the company not to deploy it. Spotify said at the time that it had no plans to implement the patent. The most significant recent development is the absorption of vocal affect analysis into general-purpose conversational AI. Since 2024, several voice assistants built on large language models have processed speech directly rather than through a transcript, which means the model has access to tone, pace and hesitation as well as words. Some companies market this explicitly. Hume AI, founded by the emotion scientist Alan Cowen, offers what it calls an empathic voice interface that measures vocal expression and adjusts its responses accordingly, and it presents its models as describing expressive behavior across many dimensions rather than identifying a small set of emotions. Others simply let the general model use whatever cues it learns are useful. Either way, affect inference is moving from dedicated products, which people can identify and avoid, into ordinary conversational software. There are also uses that most people would regard as benign or beneficial. Clinical researchers have studied vocal markers of depression and suicide risk, finding that features such as reduced pitch variability and slower speech are associated with depressive states at the group level, and have explored whether voice monitoring could help track patients' progress. Speech therapists use acoustic analysis to assess voice disorders. Assistive technologies can help people who find it hard to interpret others' tone. In these settings the person being analyzed usually knows and has agreed, and the output informs rather than replaces a professional's judgment. A call, analyzed It helps to follow a single, ordinary interaction through a real-time voice analytics system to see where the inferences pile up. Consider a call to an insurance company's customer service line from a man in his late seventies who wants to query a claim. He has some hearing loss, so he speaks loudly and asks the agent to repeat things. English is his second language, and his intonation follows the patterns of his first. He is on a mobile phone in a busy kitchen. The system's acoustic front end first has to separate his voice from the agent's and from the background noise. It then computes features over short windows: pitch level and range, loudness, speech rate, pauses, voice quality. His loudness is high throughout, because of his hearing. His pitch contours are unfamiliar to a model trained mainly on native speakers, rising where native speakers would fall. He pauses often, sometimes mid-sentence, as he searches for words. His speech overlaps the agent's when he has not heard the end of a sentence. Each of these features has a known association, in the training data, with emotional states. Sustained high loudness and overlapping speech are associated with anger and frustration. Unusual pitch movements may be read as agitation. Frequent pauses and hesitations may be read as uncertainty or confusion. The system therefore flags the call as showing escalating customer frustration and prompts the agent to adopt a calmer, more empathetic tone. It may also mark the call for supervisor review, and in some products a record of the customer's emotional profile is stored with the account. Every measurement in this sequence was accurate. The caller really was loud, his pitch really did move in unusual ways, and he really did pause. The inference was wrong at each stage, because each acoustic feature had a cause unrelated to emotion: hearing loss, linguistic background, word-finding, a noisy room. A human agent with a few minutes of conversation would probably work this out. The system, which matches features to patterns, has no way to. If the stored profile later influences how the company handles his claim, or how its systems route his future calls, a chain of reasonable-looking measurements will have produced an unreasonable outcome for a particular person, and he will never know it happened. The voice also raises a privacy issue that the face, in some settings, does not. A voice recording is a biometric identifier, capable of identifying the speaker, and it carries information about age, sex, regional origin, native language and health. Affect analysis often requires storing or processing exactly these recordings. Whether or not an emotion inference is accurate, the infrastructure built to make it collects a rich and sensitive record about everyone who calls. The voice is not a window The voice is in some ways a more defensible channel than the face. Its link to physiological arousal has a clear mechanism and robust empirical support: when people are activated, their voices change in fairly consistent ways. A system that reports "this caller's voice became louder, faster and higher-pitched over the last minute" is describing something real and often useful. The difficulty lies in what happens next. Arousal is not an emotion; it is a component shared by many emotions and by states that are not emotions at all, such as physical exertion, caffeine, illness, or simply speaking in a noisy room. Moving from "high arousal" to "angry" requires additional information, usually the words and the context, and moving from "angry" to "dissatisfied customer who should be escalated" or "candidate lacking composure" requires still more. Each step adds an interpretation, and each interpretation can be wrong in ways that fall unevenly on different groups of speakers. The voice, like the face, reliably carries signals. Whether those signals mean what the dashboard says they mean is a separate and much harder question, and it is the question to which the next chapter turns. Chapter 5: The Science Under Strain Every emotion recognition system makes a bet about how emotions work. Most commercial systems, as earlier chapters showed, bet on a version of the basic-emotion theory: a handful of discrete states, each producing a characteristic expression, recognizable across people and cultures. For a long time that bet looked safe, because the theory was dominant in textbooks and popular understanding. Over the past two decades it has looked steadily less safe. This chapter sets out the evidence that has accumulated, the competing accounts of emotion that it supports, and what follows for machines trained on the older view. The 2019 review The single most consequential document in this debate was published in July 2019 in Psychological Science in the Public Interest, a journal that commissions comprehensive reviews of questions with practical stakes. Its authors were Lisa Feldman Barrett, Ralph Adolphs, Stacy Marsella, Aleix Martinez and Seth Pollak. They were not uniformly critics of the field: Martinez is a computer vision researcher who has built facial expression systems, and Adolphs and Pollak are neuroscientists and developmental psychologists who have studied emotion perception extensively. The composition was deliberate, intended to bring together people who disagreed about theory but could agree on what the evidence showed. The review asked a precise question: can a person's emotional state be reliably inferred from their facial movements? To answer it, the authors set out criteria that any such inference would need to meet. The first was reliability: does a given emotion regularly produce the predicted facial configuration? If people who are angry scowl only occasionally, scowling cannot be a reliable sign of anger. The second was specificity: is the configuration produced mainly by that emotion and not by others? If people scowl when concentrating, confused, or in bright light as often as when angry, a scowl tells you little. The third was generalizability: do the reliability and specificity hold across cultures, ages and contexts, or only among the populations studied? A fourth consideration was validity in context: does the inference hold up when one considers the actual situations people are in, rather than isolated photographs? After reviewing over a thousand studies, the authors concluded that the evidence did not meet these standards. People do, on average, move their faces in ways that relate somewhat to their emotional states, and certain configurations are more common in certain emotions than chance would predict. But the relationships are weak and variable. In the review's summary of studies with adults in urban industrialized societies, people scowled when angry only around thirty percent of the time, which means that most of the time anger was accompanied by some other expression or none. And they scowled in many situations that had nothing to do with anger. Similar patterns held for other emotions. The authors also found that the perception studies most often cited as proof of universality, in which people match posed photographs to emotion words, systematically inflated agreement through their methods: offering a small fixed list of words, using exaggerated posed faces, and presenting faces without context. The review's most direct sentence for industry was its warning that technology companies were spending large sums to build devices that read emotions from faces, and that such efforts rested on a mistaken understanding of the science. The authors did not argue that faces were meaningless. Their point was that a facial configuration is a signal whose meaning depends heavily on the person, the culture and the situation, so that reading emotion from the face alone, as a matter of routine, is not scientifically justified. Converging lines of evidence The 2019 review drew on many independent findings, and several have since been strengthened. Studies that measure what people actually feel and what their faces actually do, rather than what observers perceive in posed photographs, have consistently found low coherence between the two. A meta-analysis by Juan Ignacio Durán and José-Miguel Fernández-Dols, published in 2021, pooled studies in which emotions were induced in participants and their facial behavior was recorded. The association between emotion and the predicted expression was, for most emotions, weak; amusement and the smile showed the strongest link, but even there many amused people did not smile. People alone tend to be less facially expressive than people in company, an observation consistent with the idea that faces serve communication as much as they express inner states. Cross-cultural work has also challenged the universality claims. In a 2012 study published in the Proceedings of the National Academy of Sciences, Rachael Jack and colleagues used a computer-graphics technique to generate random facial movements and asked Western Caucasian and East Asian observers to judge which emotion each conveyed. From thousands of judgments they reconstructed each group's mental model of what each emotion looks like. The Western models were broadly consistent with the six basic categories; the East Asian models were not, showing more overlap between categories and placing greater emphasis on the eye region. Research with remote communities has produced still more striking results. A team including Maria Gendron and Barrett, working with the Himba of Namibia, found in 2014 that when people sorted photographs of posed faces freely, without being given emotion words, they did not produce the expected six piles; they grouped smiling faces together and often grouped others by visible behavior, such as "looking", rather than by inferred emotion. Carlos Crivelli and colleagues, working in the Trobriand Islands of Papua New Guinea, reported in 2016 that the wide-eyed, open-mouthed "gasping" face that Western observers associate with fear was interpreted there as a sign of anger and threat. These findings have not gone unchallenged, and the debate is not one-sided. Alan Cowen, Dacher Keltner and colleagues have argued that emotional expression is richer and more consistent than critics allow, but organized differently from the classic six categories. In a 2021 study in Nature, they used machine learning to analyze some six million YouTube videos from 144 countries and found that sixteen kinds of facial expression occurred in broadly similar contexts around the world, such as expressions associated with amusement at practical jokes and with awe at fireworks or concerts. They interpret this as evidence of considerable universality in how expressions relate to situations. Critics respond that the method links expressions to the contexts of videos rather than to people's experienced feelings, and that the associations, while real, still leave much variation unexplained. Both sides agree on a key point for this book: the simple mapping from a single face to a single discrete emotion, which most commercial systems implement, is not supported. Context changes what faces mean A particularly vivid set of experiments shows how strongly the interpretation of a face depends on what surrounds it. Hillel Aviezer and colleagues took photographs of posed facial expressions and digitally placed them on bodies in different postures and holding different objects. In a 2008 study, a face posed to show disgust was judged as disgusted when the body held a soiled object, but was frequently judged as angry when placed on a body with a clenched, raised fist. The face was identical; only the context changed, and with it the emotion observers saw in the face itself. Eye-tracking showed that observers actually looked at the face differently depending on the body, rather than simply overriding the face with the body. In a 2012 paper in Science, Aviezer, Yaacov Trope and Alexander Todorov examined photographs of professional tennis players at the moment of winning or losing a crucial point. Such moments produce intense emotion. When observers saw only the faces, they could not reliably tell winners from losers; when they saw only the bodies, they could. Yet when observers saw the full images, they believed they were reading the emotion from the face. Intense real-world emotion, in other words, produced faces that were ambiguous on their own, and people's sense that the face conveyed the emotion was partly an illusion created by context they did not realize they were using. These findings have direct consequences for machine systems. Most emotion classifiers crop the face out of its surroundings before analysis, precisely to remove "irrelevant" variation. If human observers rely heavily on bodily posture, the situation and the preceding events to interpret faces, and are often unaware of doing so, then a system that sees only the cropped face is attempting a task that humans themselves cannot reliably perform. Its training labels, meanwhile, were often produced by human annotators who did see some context, or who imagined a stereotypical one. The model learns to reproduce judgments that depended on information it cannot access. Some researchers have responded by building context-aware systems that analyze the whole scene, the body, the setting and other people, alongside the face. These can improve agreement with human labels. But they also show where the information really lies. If an emotion can be inferred better from the situation than from the face, then emotion AI is less a technology for reading faces than a technology for guessing what people in a given situation usually feel, which is a very different and much weaker claim about any particular individual. Rival theories of what emotions are The empirical dispute sits within a theoretical one. There is no consensus in psychology about what emotions are, and the competing accounts have different implications for whether machines could recognize them. The four most influential positions differ on exactly the points that matter for emotion AI, as Table 3 summarizes. Table 3. Major theories of emotion and their implications for automated recognition. Theory Figures What an emotion is What the face conveys Implication for emotion AI Basic emotions Tomkins, Ekman, Keltner A small set of evolved, discrete programs Characteristic, largely universal displays Recognition is pattern matching, if data is good Appraisal Scherer, Frijda, Lazarus A process of evaluating events for their relevance to one's goals Components linked to particular appraisals, not whole categories Infer appraisals from cues; categories are secondary Constructed emotion Barrett A category the brain builds from bodily signals, context and learned concepts Variable movements whose meaning depends on context and culture No fixed facial signature to detect; context is essential Behavioral ecology Fridlund Not the focus; faces are social tools Signals of social intent toward others Faces predict behavior in interaction, not inner feeling Source: compiled from Ekman (1992), Scherer (2009), Barrett (2017) and Fridlund (1994). Appraisal theory, developed by Klaus Scherer and others, holds that emotions arise from a rapid sequence of evaluations of events: whether something is novel, pleasant, relevant to one's goals, controllable, and compatible with one's norms. On this view, facial movements may be linked to particular evaluations rather than to whole emotions. Brows might lower when something obstructs one's goals, whether the resulting emotion is anger, frustration or concentration. That suggests a different design for machines, one that tries to infer components of appraisal rather than emotion labels. The theory of constructed emotion, associated with Barrett, goes further. It holds that the brain continuously predicts and interprets bodily sensations in light of context and learned concepts, and that an emotion is the result of categorizing those sensations using culturally acquired concepts such as "anger" or "sadness". Instances of the same emotion word can involve very different bodily states and behaviors. On this view, there is no consistent fingerprint for anger to find, in the face or anywhere else, and recognition is inherently a contextual, interpretive act. The behavioral ecology view, advanced by Alan Fridlund in the 1990s, argues that facial displays evolved as social signals of intent, such as readiness to attack, to affiliate or to withdraw, directed at others, and that their connection to inner feelings is incidental. A smile signals friendly intent or appeasement; it need not indicate happiness. The observation that people are more expressive in company than alone fits naturally here. These theories are not equally popular, and elements of them can be combined. But none of the major alternatives to the basic-emotion view supports the core commercial proposition, that a stranger's face or voice can be scanned in isolation and a reliable reading of their inner feeling produced. What the machines really learn Seen in light of this debate, the behavior of emotion AI systems becomes easier to understand. A classifier trained on posed laboratory expressions learns to recognize posed laboratory expressions. A classifier trained on web images labeled by annotators learns to predict annotators' judgments of decontextualized faces, which are themselves shaped by the annotators' cultural concepts and by the stereotyped expressions that dominate stock photography and film. Neither is trained on data that links faces to verified inner states, because such data barely exists at scale. This explains several otherwise puzzling observations. Systems often perform well on benchmarks and poorly in deployment, because benchmarks share the assumptions of the training data. Systems frequently disagree with each other, because each has absorbed a slightly different set of labeling conventions. And systems show demographic disparities that cannot be fixed simply by adding more examples, because the disparities partly reflect how annotators perceive different groups. If annotators are more likely to see anger in the neutral faces of Black men, as some studies of human perception suggest, a model trained on their labels will learn the same tendency. It also explains why the more careful parts of the field have shifted their language. Many researchers now describe their systems as recognizing "expressions" or "perceived emotion" rather than emotion itself. Some, influenced by the critiques, focus on action units and dimensional measures and leave interpretation to humans who know the context. Some companies have rebranded their products as measuring "expressive behavior". These shifts are welcome, but they leave a question that technical refinement cannot answer: when a system's output is used to make a decision about a person, which meaning is being relied upon? If the hiring manager or the border officer reads "anger: 0.7" as a statement about the person's feelings or character, then the careful distinctions in the documentation make no difference. Signals without guaranteed meaning The science, then, does not show that faces and voices carry no emotional information. They carry a good deal, and people use it constantly, along with everything else they know about the situation, to make sense of each other. What the science shows is that the information is probabilistic, context-dependent and culturally shaped, so that the same movement means different things in different circumstances, and the same feeling shows up in different movements in different people. That has a clear practical consequence. The value of an emotion inference depends on how much context is available to interpret it and how much weight is placed on it. A therapist who notices a client's tightened lips, in the middle of a conversation about a difficult relationship, and gently asks about it, is using a facial signal in a way the science supports: as a prompt for inquiry, with rich context and with the person able to correct the interpretation. A system that scores a stranger's face in a thirty-second video clip, with no context and no opportunity for correction, and passes the score to someone making a consequential decision, is using the same kind of signal in a way the science does not support. The next two chapters examine settings where the second pattern has been tried. Chapter 6: The Interview Without an Interviewer Hiring is an obvious target for emotion AI. Employers face large volumes of applicants for many roles, interviews are expensive in staff time, and human interviewers are known to be inconsistent and biased. The promise of automated assessment is that every candidate is evaluated the same way, at any hour, by a system that does not get tired or take a liking to people who remind it of itself. During the 2010s, recorded video interviews became common, particularly for graduate schemes and high-volume roles, and several vendors began offering to score them automatically using the candidate's words, voice and face. The story of what happened next is the clearest case study available of emotion AI meeting a high-stakes decision, and of the forces that eventually pushed it back. How automated video interviews work In an asynchronous video interview, the candidate receives a link, opens it on a laptop or phone, and is presented with a series of questions on screen. For each, they have a short preparation period and then a fixed time, often two or three minutes, to record an answer. No human is present. The recordings may be reviewed by recruiters, scored by algorithms, or both. Algorithmic scoring, as vendors described it in the late 2010s, drew on three streams. The transcript of what the candidate said could be analyzed for vocabulary, the presence of particular concepts, and linguistic style. The audio could be analyzed for prosody: pace, pitch variation, pauses, energy. And the video could be analyzed for facial expressions, eye contact and movement. These features were combined in a model trained to predict some target, most often how highly a previous set of employees or candidates had been rated by managers, or how well they had subsequently performed. The candidate received a score or a ranking, and those near the top advanced. The facial component was where emotion AI entered most directly. Systems tracked expressions such as smiling and brow movement, and in some cases reported constructs such as enthusiasm, engagement or stress. Even where the vendor did not output emotion labels, the model might use expression features internally because they correlated with the training target. If successful past hires had smiled more in their recorded interviews, smiling would raise a candidate's score, without anyone ever writing down a rule that said so. The controversy builds The leading vendor, HireVue, a Utah-based company, became the focus of scrutiny. In late 2019 a widely read investigation in the Washington Post described how its system assessed candidates, quoting researchers who called the approach scientifically unfounded. In November 2019 the Electronic Privacy Information Center, a Washington-based advocacy group, filed a complaint with the Federal Trade Commission alleging that HireVue's use of facial analysis and secret algorithms constituted unfair and deceptive trade practices, arguing that the company could not substantiate its claims and that candidates had no meaningful way to understand or contest their scores. The AI Now Institute's 2019 annual report singled out affect recognition and called for it to be banned in decisions that affect people's lives and access to opportunity, citing hiring explicitly. In January 2021 HireVue announced that it had stopped using facial analysis in its assessments. The company explained that advances in natural language processing meant that visual analysis no longer added significant predictive value beyond what could be learned from candidates' words. It also published the results of an external audit, by the algorithmic auditing firm founded by the data scientist Cathy O'Neil, of one of its assessments. The announcement was significant less for the stated reasoning than for its implication: the facial component, which had been central to public concern, could be removed without much loss, which raises the question of what it had been contributing all along. Other evidence showed how fragile video-based assessments could be. In 2021 journalists from the German public broadcaster Bayerischer Rundfunk tested a commercial video interview product from a Munich startup that scored candidates on personality traits. Using actors who delivered the same answers in the same way, they varied only superficial features. Wearing glasses, wearing a headscarf, changing the brightness of the video, or placing a bookshelf in the background shifted the personality scores noticeably. None of these changes plausibly altered the candidate's suitability for any job. They did alter the pixels, and the model responded to the pixels. Who gets misread The deeper objection to emotion and expression scoring in hiring is not simply that it is noisy. It is that the errors are systematically distributed, falling more heavily on people who are already disadvantaged in the labor market. Race is one dimension. In 2018 Lauren Rhue, an information systems researcher, examined how two commercial emotion recognition services rated photographs of professional basketball players. Controlling for how much each player was smiling, she found that one service consistently rated Black players as angrier than white players, and the other was more likely to assign contempt to Black players when expressions were ambiguous. If a hiring model uses expression features, or if a recruiter reads emotion labels, such a disparity translates directly into a disadvantage for Black candidates. Disability is another, and in some ways a more intractable one. Many conditions affect facial expression, eye contact, prosody or movement. Autistic people may make less eye contact and display less conventional facial expressiveness. People with facial paralysis, Parkinson's disease, or the after-effects of stroke may have limited facial movement. Stammering, hearing impairment, and many speech conditions change prosody. Anxiety disorders, depression and the medications used to treat them change both face and voice. A model that rewards the facial and vocal behavior of past successful hires will penalize any candidate whose behavior differs for these reasons, and it will do so without ever noting that disability was involved. A 2020 report by the Center for Democracy and Technology examined these risks in detail, and in 2022 the United States Equal Employment Opportunity Commission issued technical guidance warning employers that algorithmic hiring tools could violate the Americans with Disabilities Act if they screened out candidates on the basis of disability. Culture and language add more variation. Norms around how much to smile at a stranger in authority, how directly to look into a camera, and how animated to be when describing one's achievements differ substantially across cultures and social classes. A candidate raised in a setting where restraint in front of authority is a sign of respect may be scored as disengaged by a model trained on a different population. Non-native speakers' prosody differs in ways a speech model may misread as hesitancy or lack of confidence. The problem of the training target Even if these disparities could be removed, a more fundamental problem would remain. A model trained to predict which candidates resemble previously successful employees will reproduce whatever patterns characterized those employees, including patterns that have nothing to do with job performance. If past hiring favored outgoing, expressive people for roles where reflectiveness would serve as well, the model learns to favor outgoing, expressive people. The emotional features become proxies for a style rather than a capability. Moreover, a recorded interview is an unusual and stressful performance. Speaking to a laptop camera with a countdown timer and no human feedback does not resemble most work. Many candidates report finding it unnatural and anxiety-inducing. The emotions that a system might detect in such a situation, if it could detect them, would mostly be reactions to the interview itself. A candidate who appears nervous is, in all likelihood, nervous about the interview. That says little about how they would behave in the job, and inferring a stable trait such as confidence or composure from such a reading compounds the error. This connects to the science in Chapter 5. Inferring emotional states from a short clip of a stranger, out of context, is exactly the case the evidence says is least reliable. Inferring traits from those inferred states adds another layer. Using the resulting score to decide whether a person proceeds to a human interview places substantial weight on a chain of inferences, each weak, with no step at which the candidate can explain or correct what the system thought it saw. What valid selection looks like The weakness of emotion-based scoring is easier to see against the standards that employment testing has long been expected to meet. Industrial and organizational psychology has a mature body of work on what predicts job performance and how to demonstrate it. A selection method is expected to show validity: evidence that scores on it relate to performance in the job, or that it measures a capability the job genuinely requires. In the United States, the federal Uniform Guidelines on Employee Selection Procedures, adopted in 1978, set out how employers should demonstrate the validity of selection procedures that have adverse impact on protected groups. Professional bodies such as the Society for Industrial and Organizational Psychology publish detailed principles for validation, and in 2023 SIOP issued specific recommendations for AI-based assessments, emphasizing that they should meet the same standards of validity, fairness and job relevance as any other selection tool. Decades of research summarized in influential meta-analyses, such as Frank Schmidt and John Hunter's 1998 review of predictors of job performance, have found that some methods predict performance reasonably well. Tests of general mental ability, work sample tests, and structured interviews, in which every candidate is asked the same job-related questions and answers are scored against predefined criteria, were among the strongest. Unstructured interviews, where the conversation wanders and judgments rest on overall impressions, performed worse. Later reanalyses have revised some of these estimates, but the broad lesson has held: structure and job relevance are what make assessment work. Measured against this standard, the emotion component of automated video interviews has an obvious problem. A structured interview derives its validity from the content of the answers, scored against what the job requires. Facial expression and vocal tone are not content; they are features of presentation that the structured interview was designed to stop interviewers from overweighting. Adding emotion scoring to a structured interview reintroduces precisely the impression-based judgment that structure was meant to remove, now automated and hidden from view. A vendor could, in principle, show that expression features predict performance in a particular job. But such evidence would need to rule out the possibility that they are simply predicting resemblance to past hires, and would need to show that any adverse impact on protected groups was justified by job relevance. Public evidence of that kind for emotion features has been scarce. There is also a practical lesson in HireVue's finding that dropping facial analysis cost little predictive value. If the words carry the signal, the face and voice add risk without adding much information. An employer that wants consistent, fair assessment would do better to invest in well-designed structured questions and trained human scorers, or in automated scoring of content that is transparent about what it rewards, than in systems that claim to read candidates' feelings. The legal response Lawmakers moved in stages. Illinois passed the Artificial Intelligence Video Interview Act, which took effect at the start of 2020. It requires employers that use AI to analyze recorded video interviews to notify candidates, explain how the AI works and what characteristics it evaluates, obtain consent, limit sharing of the videos, and delete them on request. A later amendment added reporting requirements on the demographics of candidates screened out where AI is used to decide who receives an in-person interview. Maryland passed a law in 2020 prohibiting the use of facial recognition services to create a template of a candidate's face during an interview without a signed waiver. New York City's Local Law 144, enforced from July 2023, requires employers using automated employment decision tools to commission annual independent bias audits and to notify candidates. These laws focus on transparency and auditing rather than prohibition. They assume that automated assessment may be legitimate if candidates are told about it and if disparate impact is measured. They do little to address whether emotion inference in particular is valid. The European Union took a more direct approach. The AI Act, adopted in 2024, includes among its prohibited practices the placing on the market or use of AI systems to infer the emotions of a natural person in the areas of workplace and education institutions, except where the system is intended for medical or safety reasons. The prohibition took effect on 2 February 2025. Guidelines on prohibited practices published by the European Commission in February 2025 indicate that the workplace prohibition covers candidates during recruitment and selection as well as employees. A video interview system that scores emotional states is therefore, for use in the EU, simply banned. The European approach rests on a recognition that goes beyond the scientific critique. Even if emotion inference were accurate, its use in employment involves a stark imbalance of power. Candidates cannot realistically refuse an assessment that stands between them and a job. Employees cannot easily object to monitoring by the organization that pays them. The recitals of the AI Act refer explicitly to the limited reliability, lack of specificity and limited generalizability of emotion recognition, and to the imbalance of power in work and education, as reasons for the prohibition. Inside the workplace Hiring is not the only workplace use. Contact centers monitor agents' vocal tone and prompt them to sound more empathetic. Some productivity platforms have experimented with measuring engagement in video meetings. In 2022, after reports that the video conferencing company Zoom was exploring emotion analysis features, a coalition of more than twenty-five rights organizations wrote publicly urging it not to proceed, arguing that the technology was discriminatory, manipulative and based on pseudoscience. Reports from China, discussed in Chapter 7, describe emotion monitoring of employees in some workplaces and of students in some classrooms. These uses share the hiring problem's structure. The person assessed cannot easily opt out, the inference is weak, and its output feeds evaluation by someone with power over them. They add a further concern: continuous monitoring of emotional display tends to require continuous performance of approved emotions. A call-center agent prompted to sound warmer is being managed not only on what they do but on how they appear to feel. The sociologist Arlie Hochschild called this "emotional labor" in 1983, describing flight attendants trained to produce warmth on demand. Emotion AI offers a way to measure and enforce such labor at scale. What the hiring case teaches The interview story illustrates the pattern of this book with unusual clarity. The underlying measurements, of facial movement and vocal features, were real. The models did find patterns that predicted their training targets. But the meaning assigned to those patterns, that they reflected emotions and that emotions reflected suitability, was never established, and the cost of error fell entirely on candidates, who could not see or challenge the process. When the facial component was removed, little seemed to change in predictive value, suggesting that it had been contributing less than its prominence implied while carrying most of the ethical risk. The hiring case also shows that pressure works. Investigative journalism, advocacy complaints, academic critique and legislation each played a part in pushing the most prominent vendor to abandon facial analysis and in establishing a ban in the European Union. The next setting, the border, has proved much more resistant, because the institutions involved operate under different pressures and different legal regimes. Chapter 7: At the Border If the job interview is where emotion AI met the market, the border is where it met the state. Border crossings combine every condition that makes emotion inference both tempting and dangerous. Officials must make rapid decisions about large numbers of strangers with limited information. The consequences of missing a threat can be catastrophic, which creates pressure to adopt anything that promises an edge. The people being assessed are often tired, anxious, speaking a second language and unfamiliar with the procedures. And travelers, especially non-citizens, have weaker legal protections and less ability to contest decisions than almost anyone else. Over the past two decades, governments and research consortia have repeatedly tried to use behavioral and emotional cues to screen travelers. The record of those attempts is the strongest evidence available about what happens when the gap between signal and meaning is placed under the heaviest possible load. A long tradition of lie detection The border projects did not come from nowhere. They belong to a century-long effort to find a reliable physiological or behavioral sign of deception. The polygraph, developed in the early 1920s, records blood pressure, pulse, breathing and skin conductance while a subject answers questions, on the theory that lying produces measurable arousal. Its scientific standing has been contested for its entire existence. A major review by the United States National Research Council in 2003 concluded that while polygraph tests can discriminate lying from truth-telling at rates above chance in specific incident investigations, the scientific basis was weak, accuracy in screening contexts was likely to be poor, and the base-rate problem meant that screening large populations would produce many false positives for each true detection. Most courts in the United States do not admit polygraph results as evidence. The polygraph's central weakness is the same one this book has traced in emotion AI: it measures arousal, and arousal has many causes. An innocent person frightened of being disbelieved can show more arousal than a calm liar. Emotion recognition systems at the border inherited this problem, adding the further uncertainty of reading arousal and emotion remotely from face and voice rather than directly from sensors on the body. Behavior detection at American airports After the attacks of September 2001, the Transportation Security Administration in the United States sought ways to identify threats beyond searching bags and people. From the mid-2000s it deployed behavior detection officers at airports under a program called Screening of Passengers by Observation Techniques, known as SPOT. Officers observed passengers in queues and at checkpoints and scored them against a checklist of behavioral indicators, some relating to stress and fear and some to deception, drawing in part on the tradition of research on facial expression and nonverbal leakage. Passengers who scored above a threshold received additional screening and questioning. In 2013 the Government Accountability Office, the investigative arm of Congress, reviewed the program. It found that the TSA had not demonstrated that behavioral indicators could be used to identify persons who might pose a risk to aviation security, observed that the published research on deception detection showed human performance at or only slightly above chance, and recommended that Congress consider limiting future funding for the program. Figures from the program's operation showed that only a very small fraction of passengers referred by behavior detection officers to law enforcement were arrested, and those arrests were largely for matters unrelated to terrorism. Civil liberties organizations raised further concerns: a 2017 report by the American Civil Liberties Union, drawing on internal documents obtained through litigation, described the program's indicators as unsupported by science and raised evidence of profiling. The TSA later restructured its behavior detection efforts, but the episode demonstrated that a large, well-funded government program could operate for years on the basis of behavioral cues whose validity had never been established. AVATAR: the automated interviewer Researchers at the University of Arizona, working with a United States Department of Homeland Security center of excellence on border security, developed a different approach: replace the human observer with a kiosk. The system, called AVATAR, for Automated Virtual Agent for Truth Assessments in Real-time, presented travelers with an animated face on a screen that asked a series of questions. As the traveler answered, sensors recorded vocal characteristics, eye movements and pupil dilation, facial expressions and body posture. Algorithms combined these into an assessment of credibility, and travelers flagged as higher risk would be referred to a human officer. AVATAR drew on the extensive deception research of Judee Burgoon, Jay Nunamaker and their colleagues, who had been studying computer-assisted credibility assessment since the 1990s. Notably, this research group did not rest its approach on micro-expressions; Burgoon herself, as Chapter 3 noted, has argued that they are a poor basis for detecting lies. The system relied instead on a range of cues, including vocal pitch and changes in eye behavior, measured under a structured questioning protocol. It was evaluated in laboratory experiments and in field trials with border agencies. Published results reported accuracy rates well above chance in experimental conditions, but such conditions, in which participants are instructed to lie about a mock crime or to carry a fake item, differ fundamentally from real border crossings, where the stakes, the population and the base rate of deception are all different. The system did not become a standard feature at American ports of entry, though its developers sought commercial routes to market. iBorderCtrl: Europe's experiment The most widely discussed project was European. iBorderCtrl was a research project funded under the European Union's Horizon 2020 program, which ran from 2016 to 2019 with a budget of around four and a half million euros and a consortium of companies, universities and border agencies. It proposed a two-stage system to speed up crossings at the EU's external land borders for travelers from outside the Union. Before their trip, travelers would register online, upload documents, and then answer questions posed by an animated virtual border guard through their webcam. During this interview, a component called the Automatic Deception Detection System analyzed what the project described as non-verbal micro-gestures, a set of small facial and head movements, to estimate whether each answer was truthful. The results, combined with other checks, would produce a risk score, which border officers would see when the traveler arrived. Pilots took place at border crossings in Hungary, Latvia and Greece. The deception component derived from a system called Silent Talker, developed by researchers at Manchester Metropolitan University. Project publications reported accuracy of roughly three-quarters from a small initial test with a limited number of participants, many of them acting in a staged scenario. Critics pointed out that such a figure, even if it held in the field, would imply a very large number of honest travelers being flagged, and that the sample was far too small and unrepresentative to support claims about a diverse traveling public. In 2019 journalists from The Intercept went through the system's interview and reported that it flagged several of their truthful answers as deceptive. The project's coordinators emphasized that iBorderCtrl was a research effort, that it was not intended for deployment without further development, and that border officers would retain the final decision. But the opacity of the project became a story in itself. Patrick Breyer, a member of the European Parliament, requested access to documents about the project's ethical assessment, legal admissibility and results, and the EU's Research Executive Agency refused most of them on grounds of commercial confidentiality. Breyer took the case to the EU's General Court, which in December 2021 ruled partly in his favor, recognizing a public interest in democratic oversight of the development of surveillance technologies and ordering the release of some documents, while upholding protection of others. An appeal to the Court of Justice in 2023 did not change the overall outcome in Breyer's favor. The episode illustrated how publicly funded research into technologies with serious implications for fundamental rights could proceed with very limited public scrutiny of its scientific basis. Emotion recognition in China The most extensive reported uses have been in China, where emotion recognition has been marketed to public security agencies, customs authorities, schools and employers. In January 2021 the human rights organization Article 19 published a detailed report, "Emotional Entanglement", by Shazeda Ahmed and Vidushi Marda, documenting a domestic market of companies offering emotion recognition for purposes including early warning of "suspicious" individuals in public spaces, monitoring of students' attention in classrooms, and interrogation support for police. The report noted that companies made strong claims about their systems' capabilities, often citing Ekman's work, with little published evidence. A school in Hangzhou attracted international attention in 2018 for a classroom system that monitored students' facial expressions to assess their attentiveness. In May 2021 the BBC reported the account of a software engineer who said he had installed emotion detection cameras in police stations in Xinjiang, where they had been tested on detained Uyghurs; the claims could not be independently verified, and the Chinese embassy disputed the reporting. These cases matter for the border debate because they show where the logic of emotion-based screening leads when legal and institutional constraints are weak. Emotion inference becomes one more input into systems of surveillance and control, applied to people who have no ability to refuse and no means of contesting the result. Comparing the attempts The main efforts differed in their technology, sponsors and outcomes but share a common pattern. Each relied on a chain of inference from observable behavior to hidden intent, and none produced public evidence that the chain held in real conditions. Table 4 sets the main cases side by side. Table 4. Behavioral and emotion-based screening efforts in border and security settings. System Period Sponsor or operator Signals used What became of it SPOT behavior detection Mid-2000s onward US Transportation Security Administration Human observation of stress and deception indicators Criticized by GAO in 2013 for lack of validation; later restructured AVATAR kiosk 2010s University of Arizona, US homeland security research funding Voice, eye movement, face, posture Laboratory and field trials; not adopted as standard at US ports iBorderCtrl 2016 to 2019 EU Horizon 2020 research consortium Facial and head micro-gestures via webcam Piloted in three member states; ended as research; transparency litigation Commercial systems in China Late 2010s onward Various companies, public security and education buyers Facial expression, sometimes physiology Documented by Article 19 in 2021; scale and effectiveness unverified Source: GAO (2013), University of Arizona publications, iBorderCtrl project documents and Breyer litigation, Article 19 (2021). Why the border is the hardest case Several features make the border a uniquely poor setting for emotion inference, beyond the general scientific weaknesses. The first is the base rate. As the arithmetic in Chapter 3 showed, when the behavior being screened for is rare, even an accurate test produces mostly false alarms. The proportion of travelers concealing something serious is very small, so a system that flags a meaningful share of travelers will be wrong about almost all of them. The second is the population. Travelers at an international border come from every culture, speak many languages, and include people of all ages and abilities. Every source of variation in facial and vocal behavior that the previous chapters described is present at once. A traveler questioned through a webcam in a second or third language, after a long journey, will produce speech with hesitations and prosody that bear little relation to truthfulness. The third is the emotional state of those most affected. Asylum seekers and refugees may have experienced violence, persecution or loss. Trauma can produce flat or incongruent emotional expression, dissociation, gaps in memory and inconsistency in accounts, all of which credibility assessments, human or automated, may read as signs of deception. Immigration lawyers and psychologists have long warned that human officials misjudge the credibility of traumatized applicants for these reasons. Automating such judgments risks encoding the same errors at scale and lending them an appearance of scientific objectivity. The fourth is power and accountability. A job candidate who is wrongly screened out loses an opportunity; a traveler who is wrongly flagged may be detained, searched, refused entry or have an asylum claim treated with suspicion. Non-citizens at the border often have limited rights to information, representation and appeal. If the decision relied partly on an opaque emotion score, they may never learn that it did. The human in the loop Defenders of these systems almost always point out that a human makes the final decision. The machine only flags; the officer decides. iBorderCtrl's coordinators made exactly this argument, and the AI Act's requirement of human oversight for high-risk systems rests on the same idea. It is worth examining why this safeguard is weaker than it sounds. Research on automation bias, much of it from aviation, medicine and process control, has repeatedly found that people who work with automated decision aids tend to over-rely on them. A review by Raja Parasuraman and Dietrich Manzey in 2010 summarized evidence that operators given automated advice are more likely to miss problems the automation misses, and to follow the automation's recommendation even when other information contradicts it. The effect is stronger under time pressure and heavy workload, conditions that describe a busy border crossing well. An emotion or deception score also changes the officer's frame before the encounter begins. An officer who sees that a traveler has been flagged as high-risk for deception will approach the interview expecting deception, ask different questions, and interpret ambiguous behavior, such as nervousness, hesitation or inconsistency, in light of that expectation. Psychologists call this confirmation bias. The traveler's anxiety at being singled out for questioning will then produce exactly the cues that seem to confirm the flag. The human in the loop does not independently check the machine's inference; the machine's inference shapes what the human sees. Finally, the officer typically has no way to know why the system produced its score. A deception score from facial micro-gestures does not come with an explanation an officer could evaluate, and even if it did, the officer would have no means of checking whether a particular facial movement reflected deception, fatigue, a second language or trauma. Human oversight is meaningful only when the human can understand and question what the system is doing. For opaque emotion inferences about strangers, that condition is rarely met. The regulatory gap The European Union's AI Act, as Chapter 6 described, bans emotion recognition in workplaces and schools. It does not ban it at borders. Instead, AI systems used in migration, asylum and border control management, including systems used as polygraphs or similar tools and systems used to assess risks posed by persons entering a member state, are classified as high-risk. Emotion recognition systems in general also appear in the high-risk category. High-risk classification brings obligations for risk management, data quality, documentation, human oversight and accuracy, but it permits use. Civil society organizations, including European Digital Rights and Access Now, argued during the legislative process that emotion recognition and AI lie detection should be banned in migration contexts as well, pointing to the scientific critique and the vulnerability of the people concerned. They did not prevail. The result is an asymmetry that is difficult to justify on scientific grounds. The evidence that emotion inference is unreliable applies equally to employees and to travelers. If anything, the conditions at the border make the inference less reliable and its errors more harmful. The difference in treatment reflects not the science but the political weight of security arguments and the relative weakness of the constituency affected. The final chapter considers how a more coherent set of lines might be drawn. Chapter 8: Drawing the Lines The previous chapters have shown a technology that measures real signals, attaches contested meanings to them, and has been deployed in settings where those meanings bear heavy weight. The natural response might be to ban it outright, and some critics have argued for exactly that. But emotion AI is not one thing. A system that warns a drowsy driver, a tool that helps a nurse notice pain in a patient with dementia, an app that helps an autistic child practice reading expressions, and a kiosk that scores travelers for deception all draw on overlapping techniques. A rule that treats them identically will either forbid valuable uses or permit harmful ones. This chapter looks at how the main regulatory regimes have tried to draw distinctions, where they succeed and fail, and proposes a set of questions that can be applied to any particular deployment. How the European Union drew its lines The EU's AI Act, which entered into force in August 2024 with obligations phased in over the following years, is the most detailed attempt so far to regulate emotion recognition. Its approach rests on several moves. It begins with definitions. An emotion recognition system is defined as an AI system for the purpose of identifying or inferring emotions or intentions of natural persons on the basis of their biometric data. The recitals clarify that this covers emotions such as happiness, sadness, anger, surprise, disgust, embarrassment, excitement, shame, contempt, satisfaction and amusement. They expressly exclude physical states such as pain or fatigue, so a system that detects a pilot's or driver's tiredness is not an emotion recognition system under the Act. They also exclude the mere detection of readily apparent expressions, gestures or movements, such as a smile or a raised voice, unless that detection is used to identify or infer emotions. This is a significant drafting choice: it maps closely onto the distinction this book has drawn between the descriptive layer of measurement and the inferential layer of meaning. It then sorts uses by context. The Act prohibits the use of emotion recognition in the areas of workplace and education institutions, except for medical or safety reasons. It classifies emotion recognition systems more generally as high-risk, which brings requirements for risk management, data governance, technical documentation, logging, human oversight, accuracy and robustness, and registration. Separately, AI systems used in migration, asylum and border control, including those used as polygraphs or similar tools, are high-risk in their own right. Finally, anyone deploying an emotion recognition system must inform the people exposed to it, a transparency obligation that applies from August 2026. The timing of these obligations has itself been contested. The prohibitions applied from 2 February 2025, and the Commission published guidelines that month explaining how they should be interpreted, including the scope of "workplace" and of the medical and safety exception. Obligations for high-risk systems were originally due to apply from August 2026. In 2026, however, the EU adopted an amending package known as the Digital Omnibus on AI, which postponed the main high-risk obligations for stand-alone systems such as those listed for biometrics and border control to December 2027. The prohibitions and the transparency duty for emotion recognition were not delayed. For a vendor, this means that selling emotion inference to employers or schools in the EU has been unlawful since early 2025, that people must be told when they are subject to emotion recognition, and that the fuller compliance regime for permitted high-risk uses, including at borders, arrives later than first planned. The status of each kind of use is summarized in Table 5. Table 5. Treatment of emotion-related AI uses under the EU AI Act. Context or use Status Main consequence Inferring emotions of workers or job candidates Prohibited since February 2025 Unlawful unless for medical or safety reasons Inferring emotions of students in education institutions Prohibited since February 2025 Unlawful unless for medical or safety reasons Emotion recognition in migration, asylum, border control or policing Permitted, high-risk Risk management, oversight, accuracy and documentation duties Other biometric emotion recognition, such as retail or advertising Permitted, high-risk Same high-risk duties, plus duty to inform those exposed Detecting physical states such as fatigue or pain Outside the definition Not regulated as emotion recognition; other rules may apply Detecting obvious expressions without inferring emotion Outside the definition Not regulated as emotion recognition Source: Regulation (EU) 2024/1689, Articles 3, 5 and 50, Annex III and recital 18; Commission guidelines on prohibited practices (2025); Digital Omnibus on AI (2026). The structure has real strengths. It bans the use in the two settings where power imbalances are most acute and least escapable in daily life, and it builds the descriptive-inferential distinction into the law. Its weaknesses are equally visible. As Chapter 7 argued, the border and policing exemptions are hard to reconcile with the science, since the conditions there make inference less reliable and errors more harmful than in the workplace. The line between "detecting a raised voice" and "inferring anger" will be difficult to police in practice, since a vendor can describe its output in descriptive terms while customers read it inferentially. And the medical and safety exceptions, though sensible in principle, could become loopholes if employers present wellbeing monitoring as a safety measure. Other approaches Outside the EU, regulation has been more fragmented. In the United Kingdom, the Information Commissioner's Office took the unusual step in October 2022 of warning organizations about "emotional analysis" technologies, saying that they appeared immature, that they might not work, and that the regulator had yet to see any such technology developed in a way that met data protection requirements. The warning carried weight because data protection law already applies to most emotion recognition, which processes personal data and often biometric data. In the United States there is no federal law specific to emotion recognition. The most significant constraint has come from Illinois' Biometric Information Privacy Act of 2008, which requires informed written consent before collecting biometric identifiers, including scans of face geometry, and gives individuals a private right to sue. The Act has generated extensive litigation over facial analysis, and companies offering facial expression analysis to Illinois residents have had to account for it. The video interview laws in Illinois, Maryland and New York City, discussed in Chapter 6, add sector-specific rules. Federal agencies such as the Federal Trade Commission have authority over deceptive claims, and the FTC has warned companies that marketing AI products with unsubstantiated claims about their capabilities can be unlawful. Industry self-regulation has played a part as well. In June 2022 Microsoft announced that it would retire the capabilities in its Azure Face service that inferred emotional states, along with identity attributes such as gender and age, from new customers immediately and from existing ones within a year. The company cited the lack of scientific consensus on the definition of emotions, the challenges of generalizing inferences across use cases, regions and demographics, and heightened privacy concerns. It kept facial analysis available for some accessibility uses, such as tools for people with visual impairments. The decision did not end the market, since other vendors continued to sell emotion APIs, but it was a significant signal from one of the largest providers that the inferential claims could not be defended. The uses worth keeping A framework that only explains why some uses are harmful risks implying that none are acceptable. That would be a mistake. Several applications use the same techniques in ways that are both scientifically defensible and socially valuable. Driver monitoring is the clearest. EU vehicle safety rules have required new cars to be fitted with systems that detect driver drowsiness and attention lapses, phased in from 2022, with requirements for advanced distraction warnings following. These systems track eye closure, blink duration, head pose and gaze direction. They measure states, such as fatigue and inattention, that have clear behavioral signatures and well-established links to crash risk, and they respond with a warning to the driver, who remains in control. They do not need to know what the driver feels. Clinical tools offer another example. People with advanced dementia often cannot report pain, which is consequently under-recognized and under-treated. An Australian-developed app, PainCheck, guides carers to assess pain partly by analyzing facial action units associated with pain, alongside other observations, and has received regulatory clearance in several countries as a medical device. Here the measurement is focused on a specific, physically grounded state with a substantial research base, the person assessed benefits directly, and a professional interprets the result in context. Assistive technology is a third. Researchers at Stanford tested a system in which autistic children wore smart glasses that identified the expressions of people they were interacting with and displayed cues, and reported in a 2019 randomized trial in JAMA Pediatrics that children using it showed gains on a measure of socialization. Whatever the long-term value of such interventions, the use is chosen by the family, serves the child's own goals, and treats the expression label as a learning aid rather than a verdict. Research, entertainment and creative uses form a fourth group. Animators use expression tracking to drive digital characters; psychologists use automated action-unit coding to analyze large video datasets far faster than human coders could; game designers experiment with adapting difficulty to players' apparent frustration. In each case, errors have modest consequences, the descriptive layer does most of the work, and no one's opportunities depend on the output. Five questions for any deployment The cases in this book suggest that the acceptability of emotion AI turns less on the technique than on how it is used. Five questions capture most of what matters, and they can be asked of any system. The first is what the system actually claims. Does its output describe behavior, such as eye closure, a smile, a raised voice? Does it predict how humans would perceive the behavior? Or does it assert an inner state or trait, such as anger, deception or lack of enthusiasm? The further along that chain the claim goes, the stronger the evidence required, and for inner states inferred from brief observations of strangers, the evidence is generally not there. The second is what decision the output feeds, and how much weight it carries. An output that prompts a driver to take a break, or a clinician to look more closely, is a nudge toward further inquiry. An output that determines whether a candidate advances, or whether a traveler is referred for secondary inspection, is a gate. Gates require much higher standards of validity than nudges. The third is who bears the cost of error. When the person assessed is also the person who benefits, as with a driver warning, a false alarm is a minor annoyance they can dismiss. When the system serves an institution and the assessed person bears the cost of its errors, every false positive is an imposition on someone who did not choose it. Systems that shift the cost of error onto people who cannot see or contest the process deserve the deepest skepticism. The fourth is whether participation is genuinely voluntary. Consent obtained from a job candidate, a student, an employee or a traveler is rarely meaningful, because refusing has consequences that the person cannot afford. The European ban on workplace and education uses recognizes this. The same logic extends to borders, welfare offices, courts and any other setting where the person assessed depends on the assessor. The fifth is whose purpose the system serves. Tools chosen by individuals to help them understand or manage their own states, or to support them in interacting with others, stand on very different ground from tools used by others to evaluate, sort or influence them. The same expression classifier can be an aid in one arrangement and an instrument of control in another. Applied together, these questions sort the cases of this book in a way that matches most people's intuitions and the scientific evidence. Driver drowsiness detection passes easily. Pain assessment for people who cannot speak passes with appropriate clinical oversight. Assistive tools chosen by users pass. Emotion scoring in job interviews fails on nearly every question. Deception detection at borders fails on all five. What a responsible buyer should ask The five questions are useful for judging a deployment in principle. Organizations considering an emotion-related product also need practical questions to put to vendors, and the answers, or the vendor's inability to give them, are often decisive. A buyer should ask what exactly the system outputs and how those outputs are defined. If the product reports "engagement" or "stress", what measured behaviors feed that construct, and how was the mapping established? A vendor that cannot explain its constructs in terms of observable behavior is selling an inference whose basis it does not understand or will not disclose. A buyer should ask what the training data consisted of: whether expressions were posed or spontaneous, who the people were, how labels were produced, and what the labels represent. Labels from annotators viewing cropped faces represent perceived expression; labels from self-report represent something else; labels from outcomes such as hiring decisions represent past institutional judgments, with all their biases. A buyer should ask for validation evidence in a setting comparable to the intended use. Accuracy on a public benchmark says little about performance on call-center audio from older customers, or on interview video from candidates of many backgrounds. The relevant evidence shows how outputs relate to the outcome the buyer actually cares about, in a population like the one the buyer will serve. A buyer should ask for performance broken down by age, sex, ethnicity, language background and disability where possible, and for evidence that disparities have been measured and addressed rather than assumed away. A buyer should ask how scores are calibrated and aggregated over time, how the system behaves on faces or voices unlike its training data, and whether it can indicate low confidence rather than forcing a label. And a buyer should ask what the people assessed will be told, whether they can see and contest their results, and how long their data will be retained. Under European law, people must now be informed when they are subject to emotion recognition; good practice goes further and gives them a way to respond. Many products will not survive these questions. That is itself useful information. A system whose vendor cannot say what its outputs mean, what data produced them, or how well they work for the people it will be applied to, should not be making claims about anyone's feelings. The coming challenge The regulatory frameworks described here were designed with dedicated emotion recognition products in mind: a hiring tool with an emotion module, a border kiosk, an advertising analytics service. The technology is moving in a different direction. As Chapter 4 noted, conversational AI systems now process voice directly and can respond to tone, hesitation and emphasis. Video-capable assistants can see faces. These systems may never output an emotion label at all; they simply adjust their responses in light of whatever cues they have learned to use. This poses new problems. A system that adapts its persuasive strategy to a user's apparent emotional state is engaged in emotion inference whether or not it ever names the state. The EU AI Act's separate prohibition on manipulative techniques that materially distort behavior and cause significant harm may be relevant, and its transparency rules for emotion recognition would apply where the system's purpose is to infer emotions. But the boundary between a helpful assistant that notices a user is frustrated and slows down, and a sales agent that detects hesitation and applies pressure, is difficult to draw in law. The five questions above still apply: what is being claimed, what does it drive, who bears the cost, is participation voluntary, and whose purpose is served. Answering them will require more insight into how such systems use affective cues than their developers currently disclose. Conclusion It is tempting to treat the problems of emotion AI as problems of immaturity: the datasets are too small and too posed, the models too crude, the cultural coverage too narrow, and a decade of progress will close the gap. The chapters of this book point to a different conclusion. Some of the gap will close. Cameras will get better at seeing faces at awkward angles, classifiers will become more robust across skin tones and lighting, and speech models will handle more languages and accents. These are real improvements in measurement. But the central weakness lies elsewhere, and better measurement cannot fix it. The weakness is that the thing emotion AI claims to read is not fully present in the signal. A face in a thirty-second clip, a voice on a call, carries traces of what a person feels, mixed with traces of what they intend to communicate, what their culture has taught them, how their body happens to be built, what condition they live with, and what the situation demands. The science reviewed in Chapter 5 does not say that these traces are meaningless. It says they underdetermine the inner state, so that the same feeling yields different signals and the same signal arises from different feelings. No quantity of training data can extract from a signal information that it does not reliably contain. A system trained on more and better faces will become a more accurate predictor of what annotators say about faces. It will not, for that reason alone, become a better reader of minds. If that is right, three things follow that go beyond the familiar call for caution. Govern the claim, not the camera The first is that the right object of governance is the inference, not the sensor or the algorithm. The same face-tracking pipeline serves a drowsiness alarm, a pain assessment tool, an animation rig, a hiring filter and a border kiosk. Regulating the technique would either stifle the first three or license the last two. What distinguishes them is the claim each makes about a person and the decision that claim supports. The European Union's definition, which excludes the detection of obvious expressions and physical states unless they are used to infer emotions, points in this direction, and it is the most important idea in current law on the subject. It should be extended rather than worn away. In practice that means requiring anyone who sells or uses a system touching on affect to say plainly which layer the output belongs to: a description of behavior, a prediction of human perception, or an assertion about an inner state or trait. It means treating a description that is routinely read as an assertion, a "stress score" that recruiters treat as a judgment of composure, as the assertion it functions as. And it means that the evidence demanded should scale with the layer. Descriptions need ordinary validation against careful human coding. Assertions about what a particular person feels, made from brief observation and used to decide something about them, need evidence that almost no current system can provide, and in its absence should not be made. The burden belongs with the deployer The second follows from the hiring story. When HireVue removed facial analysis from its assessments, it reported that little predictive value was lost. That is a striking admission, and it generalizes. In many deployments the emotional inference adds little to what the words, the record or the ordinary judgment of a trained person already supply, while carrying most of the risk of bias, error and harm. The reasonable default is therefore to presume that an emotion-inference component should not be used in a consequential decision unless its deployer can show, for the population and setting in question, that it improves the decision and does not shift errors onto particular groups. That inverts the present practice, in which the people assessed, or advocates on their behalf, must prove harm after the fact through investigative journalism, complaints and litigation. The border cases show why the inversion matters. iBorderCtrl was piloted on real travelers on the basis of a small laboratory test, and the evidence about it came out only through a member of the European Parliament's lawsuit. SPOT ran for years before an audit found that its indicators had never been validated. In settings where those affected cannot see the system, cannot contest its output and often do not know it exists, waiting for proof of harm guarantees that harm will be found late. The burden should rest with those who choose to use the inference, and it should be discharged before deployment, in public. The asymmetry cannot hold The third concerns the uneven map of current rules. Europe bans emotion inference over workers and students, yet permits it, under high-risk safeguards, over travelers, asylum seekers and people dealing with the police. Nothing in the science justifies that line. The case for the workplace ban rests on unreliable inference and an imbalance of power from which the person assessed cannot escape. Both conditions are stronger at the border than in the office. Travelers are more varied in culture and language, more likely to be exhausted, frightened or traumatized, and less able to contest decisions. The line was drawn by political weight, not by evidence. A rule consistent with its own reasoning would extend the prohibition to every setting where the assessed person is subject to the assessor's power and cannot meaningfully refuse: borders, policing, welfare, courts, and the provision of essential services, as well as work and education. It would keep the medical and safety exceptions, tightly defined, and leave room for uses that individuals choose for their own purposes. Whether or not that extension happens soon, the inconsistency should be named whenever emotion inference is proposed in these settings, because the arguments offered for the workplace ban answer the arguments offered for the border. What remains open Some questions remain genuinely unsettled. Emotion science has not reached consensus, and it is possible that richer, contextual and multimodal approaches will one day support narrower inferences than those that exist now, especially when people consent and contribute their own reports of what they feel. The research of Alan Cowen and others suggests that expressions are more structured than the harshest critiques implied, even if not structured in the way the industry assumed. A field that learned from its mistakes might build tools that help people understand their own patterns of stress or mood, under their own control, without pretending to read strangers. The most pressing open question, though, is not scientific. It is what happens as affect inference dissolves into the general-purpose assistants that people speak to every day. A dedicated emotion recognition product can be identified, audited and, where necessary, banned. A conversational system that quietly adjusts its tone, its offers or its arguments in light of the hesitations in a user's voice does not announce itself as emotion AI at all. The five questions of Chapter 8 still apply to it, but they cannot be answered without transparency about which cues such systems use and to what end. The lesson running through this book is that the face and voice reliably carry signals and unreliably carry meanings. The responsible course is to treat the signals as observations of what someone's face or voice did, and to treat any claim about what that person felt as a question to be put to the person, rather than a verdict handed to someone with power over them. In practice that would mean emotion AI asking people how they feel, not telling others what they feel. Notes Introduction 1. Rosalind W. Picard, Affective Computing (Cambridge, MA: MIT Press, 1997). 2. Lisa Feldman Barrett, Ralph Adolphs, Stacy Marsella, Aleix M. Martinez and Seth D. Pollak, "Emotional Expressions Reconsidered: Challenges to Inferring Emotion From Human Facial Movements", Psychological Science in the Public Interest 20, no. 1 (2019): 1–68. 3. Regulation (EU) 2024/1689 of the European Parliament and of the Council (Artificial Intelligence Act), Article 5(1)(f). Chapter 1 1. G.-B. Duchenne de Boulogne, Mécanisme de la physionomie humaine (Paris, 1862). 2. Charles Darwin, The Expression of the Emotions in Man and Animals (London: John Murray, 1872). 3. Paul Ekman, E. Richard Sorenson and Wallace V. Friesen, "Pan-Cultural Elements in Facial Displays of Emotion", Science 164 (1969): 86–88. 4. Paul Ekman and Wallace V. Friesen, "Constants Across Cultures in the Face and Emotion", Journal of Personality and Social Psychology 17, no. 2 (1971): 124–129. 5. James A. Russell, "A Circumplex Model of Affect", Journal of Personality and Social Psychology 39, no. 6 (1980): 1161–1178. 6. Ming-Zher Poh et al., "Convulsive Seizure Detection Using a Wrist-Worn Electrodermal Activity and Accelerometry Biosensor", Epilepsia 53, no. 5 (2012): e93–e97. Empatica's Embrace received US FDA clearance for seizure alerting in 2018. 7. Paul Ekman and Wallace V. Friesen, Facial Action Coding System (Palo Alto: Consulting Psychologists Press, 1978); Paul Ekman, Wallace V. Friesen and Joseph C. Hager, Facial Action Coding System (Salt Lake City: Research Nexus, 2002). Chapter 2 1. Paul Viola and Michael Jones, "Rapid Object Detection Using a Boosted Cascade of Simple Features", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2001). 2. Tadas Baltrušaitis, Amir Zadeh, Yao Chong Lim and Louis-Philippe Morency, "OpenFace 2.0: Facial Behavior Analysis Toolkit", IEEE International Conference on Automatic Face and Gesture Recognition (2018). 3. Patrick Lucey et al., "The Extended Cohn-Kanade Dataset (CK+)", IEEE CVPR Workshops (2010). 4. S. Mohammad Mavadati et al., "DISFA: A Spontaneous Facial Action Intensity Database", IEEE Transactions on Affective Computing 4, no. 2 (2013): 151–160. 5. Ian J. Goodfellow et al., "Challenges in Representation Learning: A Report on Three Machine Learning Contests", Neural Information Processing (ICONIP 2013). 6. Ali Mollahosseini, Behzad Hasani and Mohammad H. Mahoor, "AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild", IEEE Transactions on Affective Computing 10, no. 1 (2019; online 2017). 7. Dimitrios Kollias and Stefanos Zafeiriou, "Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace", British Machine Vision Conference (2019). 8. Joy Buolamwini and Timnit Gebru, "Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification", Proceedings of Machine Learning Research 81 (2018). Chapter 3 1. Ernest A. Haggard and Kenneth S. Isaacs, "Micromomentary Facial Expressions as Indicators of Ego Mechanisms in Psychotherapy", in Methods of Research in Psychotherapy, ed. L. A. Gottschalk and A. H. Auerbach (New York: Appleton-Century-Crofts, 1966). 2. Paul Ekman and Wallace V. Friesen, "Nonverbal Leakage and Clues to Deception", Psychiatry 32, no. 1 (1969): 88–106. 3. Stephen Porter and Leanne ten Brinke, "Reading Between the Lies: Identifying Concealed and Falsified Emotions in Universal Facial Expressions", Psychological Science 19, no. 5 (2008): 508–514. 4. Charles F. Bond Jr. and Bella M. DePaulo, "Accuracy of Deception Judgments", Personality and Social Psychology Review 10, no. 3 (2006): 214–234. 5. Bella M. DePaulo et al., "Cues to Deception", Psychological Bulletin 129, no. 1 (2003): 74–118. 6. Aldert Vrij, Pär Anders Granhag and Stephen Porter, "Pitfalls and Opportunities in Nonverbal and Verbal Lie Detection", Psychological Science in the Public Interest 11, no. 3 (2010): 89–121. 7. Judee K. Burgoon, "Microexpressions Are Not the Best Way to Catch a Liar", Frontiers in Psychology 9 (2018): 1672. 8. Wen-Jing Yan et al., "CASME II: An Improved Spontaneous Micro-Expression Database and the Baseline Evaluation", PLoS ONE 9, no. 1 (2014): e86041. 9. Hao-Yu Wu et al., "Eulerian Video Magnification for Revealing Subtle Changes in the World", ACM Transactions on Graphics 31, no. 4 (2012). 10. Paul Ekman, Telling Lies: Clues to Deceit in the Marketplace, Politics, and Marriage (New York: W. W. Norton, 1985). Chapter 4 1. Rainer Banse and Klaus R. Scherer, "Acoustic Profiles in Vocal Emotion Expression", Journal of Personality and Social Psychology 70, no. 3 (1996): 614–636. 2. Patrik N. Juslin and Petri Laukka, "Communication of Emotions in Vocal Expression and Music Performance: Different Channels, Same Code?", Psychological Bulletin 129, no. 5 (2003): 770–814. 3. Florian Eyben et al., "The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing", IEEE Transactions on Affective Computing 7, no. 2 (2016): 190–202. 4. Carlos Busso et al., "IEMOCAP: Interactive Emotional Dyadic Motion Capture Database", Language Resources and Evaluation 42 (2008): 335–359. 5. Steven R. Livingstone and Frank A. Russo, "The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS)", PLoS ONE 13, no. 5 (2018): e0196391. 6. Björn Schuller, Stefan Steidl and Anton Batliner, "The INTERSPEECH 2009 Emotion Challenge", Proceedings of Interspeech (2009). Chapter 5 1. Barrett et al., "Emotional Expressions Reconsidered" (2019). 2. Juan I. Durán and José-Miguel Fernández-Dols, "Do Emotions Result in Their Predicted Facial Expressions? A Meta-Analysis of Studies on the Co-occurrence of Expression and Emotion", Emotion 21, no. 7 (2021): 1550–1569. 3. Rachael E. Jack, Oliver G. B. Garrod, Hui Yu, Roberto Caldara and Philippe G. Schyns, "Facial Expressions of Emotion Are Not Culturally Universal", Proceedings of the National Academy of Sciences 109, no. 19 (2012): 7241–7244. 4. Maria Gendron, Debi Roberson, Jacoba Marietta van der Vyver and Lisa Feldman Barrett, "Perceptions of Emotion from Facial Expressions Are Not Culturally Universal: Evidence from a Remote Culture", Emotion 14, no. 2 (2014): 251–262. 5. Carlos Crivelli, James A. Russell, Sergio Jarillo and José-Miguel Fernández-Dols, "The Fear Gasping Face as a Threat Display in a Melanesian Society", Proceedings of the National Academy of Sciences 113, no. 44 (2016): 12403–12407. 6. Hillel Aviezer et al., "Angry, Disgusted, or Afraid? Studies on the Malleability of Emotion Perception", Psychological Science 19, no. 7 (2008): 724–732; Hillel Aviezer, Yaacov Trope and Alexander Todorov, "Body Cues, Not Facial Expressions, Discriminate Between Intense Positive and Negative Emotions", Science 338 (2012): 1225–1229. 7. Alan S. Cowen et al., "Sixteen Facial Expressions Occur in Similar Contexts Worldwide", Nature 589 (2021): 251–257. 8. Paul Ekman, "An Argument for Basic Emotions", Cognition and Emotion 6, no. 3–4 (1992): 169–200; Klaus R. Scherer, "The Dynamic Architecture of Emotion: Evidence for the Component Process Model", Cognition and Emotion 23, no. 7 (2009): 1307–1351; Alan J. Fridlund, Human Facial Expression: An Evolutionary View (San Diego: Academic Press, 1994). Chapter 6 1. Electronic Privacy Information Center, complaint to the Federal Trade Commission regarding HireVue, Inc. (November 2019); EPIC, "HireVue, Facing FTC Complaint From EPIC, Halts Use of Facial Recognition" (January 2021). 2. AI Now Institute, AI Now 2019 Report (New York, December 2019). 3. Lauren Rhue, "Racial Influence on Automated Perceptions of Emotions", SSRN working paper (2018). 4. Center for Democracy and Technology, Algorithm-Driven Hiring Tools: Innovative Recruitment or Expedited Disability Discrimination? (2020). 5. U.S. Equal Employment Opportunity Commission, "The Americans with Disabilities Act and the Use of Software, Algorithms, and Artificial Intelligence to Assess Job Applicants and Employees" (May 2022). 6. Illinois Artificial Intelligence Video Interview Act, 820 ILCS 42 (effective 1 January 2020); New York City Local Law 144 of 2021. 7. European Commission, Guidelines on prohibited artificial intelligence practices established by Regulation (EU) 2024/1689 (February 2025). 8. Frank L. Schmidt and John E. Hunter, "The Validity and Utility of Selection Methods in Personnel Psychology", Psychological Bulletin 124, no. 2 (1998): 262–274; Society for Industrial and Organizational Psychology, Considerations and Recommendations for the Validation and Use of AI-Based Assessments for Employee Selection (2023). 9. Arlie Russell Hochschild, The Managed Heart: Commercialization of Human Feeling (Berkeley: University of California Press, 1983). Chapter 7 1. National Research Council, The Polygraph and Lie Detection (Washington, DC: National Academies Press, 2003). 2. U.S. Government Accountability Office, Aviation Security: TSA Should Limit Future Funding for Behavior Detection Activities, GAO-14-159 (November 2013). 3. American Civil Liberties Union, Bad Trip: Debunking the TSA's "Behavior Detection" Program (2017). 4. Judee K. Burgoon and Jay F. Nunamaker Jr., and colleagues, publications of the National Center for Border Security and Immigration (BORDERS), University of Arizona, on the AVATAR system. 5. Ryan Gallagher and Ludovica Jona, "We Tested Europe's New Lie Detector for Travelers — and Immediately Triggered a False Positive", The Intercept (July 2019). 6. General Court of the European Union, Case T-158/19, Breyer v European Research Executive Agency (judgment of December 2021); Court of Justice, Case C-135/22 P (2023). 7. Raja Parasuraman and Dietrich H. Manzey, "Complacency and Bias in Human Use of Automation: An Attentional Integration", Human Factors 52, no. 3 (2010): 381–410. 8. Article 19, Emotional Entanglement: China's Emotion Recognition Market and Its Implications for Human Rights (London, January 2021). Chapter 8 1. Regulation (EU) 2024/1689, Articles 3(39), 5(1)(f) and 50(3), Annex III, and recital 18. 2. Information Commissioner's Office, statement on "emotional analysis" technologies (October 2022). 3. Illinois Biometric Information Privacy Act, 740 ILCS 14 (2008). 4. Microsoft, "Microsoft's Framework for Building AI Systems Responsibly" and Responsible AI Standard announcements (June 2022). 5. Catalin Voss et al., "Effect of Wearable Digital Intervention for Improving Socialization in Children With Autism Spectrum Disorder: A Randomized Clinical Trial", JAMA Pediatrics 173, no. 5 (2019): 446–454. Further Reading Rosalind W. Picard, Affective Computing (MIT Press, 1997).** The founding statement of the field. Still worth reading for its careful distinction between observing expressions and inferring emotions, and for its early attention to privacy and consent. Lisa Feldman Barrett et al., "Emotional Expressions Reconsidered", Psychological Science in the Public Interest (2019).** The essential review of whether emotions can be inferred from facial movements. Long but clearly organized; the summary and the sections on reliability and specificity repay close attention. Lisa Feldman Barrett, How Emotions Are Made: The Secret Life of the Brain (Houghton Mifflin Harcourt, 2017).** A readable account of the theory of constructed emotion, with a sustained critique of the assumption that emotions have fixed bodily fingerprints. Paul Ekman, Emotions Revealed (Times Books, 2003).** The classic case for basic emotions and universal expressions, by the scientist whose work shaped the field. Reading it alongside Barrett gives both sides of the central debate in their own words. Andrew McStay, Emotional AI: The Rise of Empathic Media (SAGE, 2018) and Automating Empathy (Oxford University Press, 2023).** Thoughtful studies of how emotion sensing is spreading through media, workplaces, cars and cities, with attention to how ordinary people view it. Kate Crawford, Atlas of AI (Yale University Press, 2021), chapter "Affect".** A concise history of how Ekman's research became the basis for commercial emotion recognition, and a critique of the politics of that inheritance. Article 19, Emotional Entanglement (2021).** A detailed report on China's emotion recognition market, its claims and its human rights implications, with recommendations that apply well beyond China. Judee K. Burgoon, "Microexpressions Are Not the Best Way to Catch a Liar", Frontiers in Psychology (2018).** A short, accessible assessment of the micro-expression evidence by a leading deception researcher who has also built automated credibility assessment systems. European Commission, Guidelines on Prohibited AI Practices (2025). The official interpretation of the AI Act's ban on emotion recognition in workplaces and education, with worked examples of what falls inside and outside it. Hashtags: #EmotionAI #AffectiveComputing #FacialExpressionAnalysis #EmotionRecognition #FacialEmotionRecognition #ArtificialIntelligence #MachineLearning #DeepLearning #ComputerVision #FacialActionCodingSystem #FACS #FacialActionUnits #FacialLandmarks #MicroExpressionAnalysis #SpeechEmotionRecognition #VoiceAnalysis #SpeechProsody #MultimodalEmotionRecognition #EmotionClassification #ValenceAndArousal #HumanComputerInteraction #BehavioralAnalysis #EmotionDetection #AffectiveAI #EmotionScience #AIAccuracy #AlgorithmicBias #AIPrivacy #ResponsibleAI #AIEthics

Download the Book (PDF): Introduction On a lava field outside Reykjavík, a row of steel boxes the size of shipping containers hums against the wind. Each box holds fans that pull the open air through a filter coated with a chemical that grabs carbon dioxide and lets nearly everything else pass. When the filter is full, the box closes, warms itself with heat from a nearby geothermal power station, and releases a stream of nearly pure carbon dioxide. That gas is dissolved in water and pumped a kilometre or so down into young basalt, where it reacts with the rock and, within about two years, turns most of itself into white carbonate crystals lining the cracks and pores. The carbon that a car exhaust or a power plant chimney put into the sky is, at the end of the chain, stone. The idea is irresistible, and for good reason. It promises not merely to slow the accumulation of carbon dioxide in the atmosphere but to reverse it: to take back what has already been emitted. The steel boxes are real, the chemistry works, and the rock does what the geochemists said it would. Yet the plant that housed those boxes, the largest of its kind when it opened in 2024 with a stated capacity of 36,000 tonnes a year, reported removing 675 tonnes in the first half of 2026, two years after start-up, and that figure was celebrated as a sixfold improvement on the same period a year earlier. The world emits roughly 37 billion tonnes of carbon dioxide from fossil fuels and cement every year. The gap between those two numbers is not a rounding error. It is the subject of this book. What this book argues Direct air capture and geological storage are usually discussed as a single proposition, which is either a miracle or a scam depending on who is talking. Both camps are wrong in the same way: they treat the technology as one thing. It is, in fact, two quite different engineering problems joined at a wellhead. The first problem is chemical and thermodynamic. Carbon dioxide makes up about 0.04 percent of the air. Pulling it out of that enormous, dilute stream is easy in principle; chemists have done it in laboratories for a century. The hard part is letting it go again. Every material that binds carbon dioxide strongly enough to catch it from air must then be persuaded, with heat, vacuum, electricity or changes in humidity, to release it as a concentrated stream so the material can be used again. That release step sets the energy bill, and the energy bill sets everything else: the cost per tonne, the size of the power supply, and whether the whole enterprise removes more carbon than its energy supply emits. The second problem is geological. A tonne of captured gas is worth nothing for the climate unless it stays out of the atmosphere for centuries to millennia. The most proven way to do that is to compress it into a dense, liquid-like state and inject it into porous rock deep underground, beneath layers of impermeable shale or salt. Whether that works at a given place depends on things that cannot be seen from the surface and must be mapped, drilled, tested and watched: how much pore space the rock really has, how easily fluid moves through it, how much pressure it can take before it fractures, and whether old wells pierce the seal. The controlling idea of this book is that the chemistry of capture is largely solved, while the energy of release and the geology of storage are not, and that those two unsolved parts decide what direct air capture can honestly do. They make it expensive, energy-hungry, and dependent on a finite resource of well-characterised rock. That does not make it useless. It makes it a scarce instrument, suitable for cleaning up the residual emissions that are hardest to eliminate and, eventually, for drawing down some of the historical excess, but never a substitute for not emitting in the first place. A reader who understands why releasing carbon dioxide costs what it does, and why pore space must be mapped rather than assumed, will be able to judge every press release, policy and pitch deck in this field without help. How the book is built The chapters follow a molecule of carbon dioxide from the open air to its final resting place. Chapter 1 begins with the air itself. It works through the arithmetic of dilution, showing how many cubic metres of air must pass through a machine to yield one tonne of carbon dioxide, what that implies for fans and contactors, and why the distinction between gross capture and net removal matters from the very first calculation. Chapters 2 and 3 describe the two great families of capture chemistry. The first uses liquid solvents, strong alkaline solutions that react with carbon dioxide and are then regenerated through a loop of limestone chemistry and a kiln running at around 900 °C. The second uses solid sorbents, usually amines bound to porous supports, that are loaded at ambient conditions and stripped with low-temperature heat and vacuum. These chapters also cover the newer approaches that try to avoid heat altogether, using humidity, electricity, or the slow natural reactions of minerals. Chapter 4 is the hinge of the book. It explains the thermodynamics of letting go: the theoretical minimum energy needed to separate carbon dioxide from air, the much larger amounts that real processes consume, where the gap comes from, and why the source and quality of the heat matter as much as its quantity. Chapters 5, 6 and 7 go underground. Chapter 5 explains the physics and chemistry by which rock holds carbon dioxide: buoyant plumes trapped beneath seals, droplets stranded in pores, gas dissolved in brine, and carbon turned to mineral. Chapter 6 explains how geologists find and prove a storage site, from seismic surveys to the arithmetic of storage capacity, and why recent estimates of safely usable storage are an order of magnitude lower than older headline figures. Chapter 7 deals with monitoring, failure and liability, drawing on the projects that have run for decades and the ones that ran into trouble. Chapter 8 brings the two halves back together and asks what they cost, who pays, and at what scale the whole system could plausibly operate. The Conclusion argues what follows for policy and for anyone deciding whether to buy, build or regulate. A note on numbers and words This field is crowded with numbers, and many of them are aspirations dressed as facts. Throughout the book, a nameplate capacity is distinguished from what a plant has actually done, and a company's target cost is distinguished from a measured one. Where a figure comes from a specific study, the study is named in the text and listed in the Notes. Where a calculation is illustrative, it is labelled so, and the assumptions are given so the reader can redo it. Two terms deserve precision from the start. Carbon capture on its own usually means taking carbon dioxide from a concentrated industrial source, such as a cement kiln or a gas-processing plant, before it reaches the air. That prevents new emissions but does not remove old ones. Direct air capture means taking carbon dioxide from ambient air. Only when direct air capture is paired with durable storage, and only after subtracting all the emissions involved in running it, does it amount to carbon dioxide removal. The difference between these phrases is where much of the confusion, and some of the misdirection, in public debate begins. The machines on the Icelandic lava field are worth taking seriously, both for what they can do and for what they cannot. The rest of this book explains each in turn. Chapter 1: Moving the Atmosphere Every argument about direct air capture eventually comes back to one number: about 425 parts per million. That is the approximate concentration of carbon dioxide in the atmosphere in the mid-2020s, as measured at Mauna Loa and a global network of other stations. It means that of every million molecules of air, about 425 are carbon dioxide. The rest are overwhelmingly nitrogen, oxygen, argon and water vapour. The concentration is high enough to have warmed the planet by more than a degree Celsius since the nineteenth century, and low enough to make it extraordinarily laborious to pull back out. The idea of capturing carbon dioxide from the open air to stabilise the climate is younger than the chemistry but older than most people assume. The physicist Klaus Lackner, then at Los Alamos National Laboratory, proposed it in the late 1990s, arguing that because air mixes carbon dioxide globally, a capture plant could be put anywhere and could in principle offset emissions from any source, including the dispersed emissions of vehicles and aircraft that cannot be captured at a chimney. In 2011, a panel convened by the American Physical Society assessed the concept and estimated that a plant built with existing hydroxide technology would cost around 600 dollars per tonne of carbon dioxide, concluding that air capture was not then an economically sensible way to cut emissions while large concentrated sources remained uncontrolled. In 2019, the United States National Academies of Sciences, Engineering, and Medicine published a broader report on negative emissions technologies that treated direct air capture as a serious candidate requiring large-scale research and demonstration, while emphasising its energy needs and costs. Between those reports, several companies began building. Climeworks was founded in Zurich in 2009 and opened a small commercial plant at Hinwil in Switzerland in 2017, selling captured carbon dioxide to a greenhouse. Carbon Engineering began piloting its process in Squamish, British Columbia, in 2015. Global Thermostat built demonstration units in the United States. The arrival of the climate goals of the 2015 Paris Agreement, and the realisation that almost every modelled pathway to them relied on some form of carbon dioxide removal, transformed air capture from a curiosity into an industry with investors, subsidies and customers. This chapter is about what a concentration of 425 parts per million means in physical terms. Before any chemistry, before any rock, a direct air capture plant is a machine for moving a great deal of air past a surface. Its size, its noise, its energy use and its land footprint all start here. The arithmetic of dilution Begin with a cubic metre of air at sea level and a temperature of 15 °C. By the ideal gas law it contains about 42 moles of gas molecules in total. At 425 parts per million, about 0.018 moles of that are carbon dioxide. Each mole of carbon dioxide weighs 44 grams, so a cubic metre of ordinary air carries roughly 0.79 grams of carbon dioxide, less than the mass of a paperclip. To gather one tonne, then, you would have to strip every molecule of carbon dioxide from about 1.27 million cubic metres of air. That is a cube roughly 108 metres on each side. Real machines do not strip every molecule. Removing the last fraction from a stream is disproportionately expensive, so designs aim to capture somewhere between half and four-fifths of the carbon dioxide that passes through. A process engineer working with a capture fraction of 75 percent must therefore process about 1.7 million cubic metres of air, weighing about 2,000 tonnes, for every tonne of carbon dioxide obtained. Two thousand tonnes of air handled for one tonne of product is the defining ratio of the whole industry. Scale it up. A plant designed to capture one million tonnes of carbon dioxide a year, roughly the scale of the largest projects now under construction, must move about 1.7 trillion cubic metres of air a year. Spread evenly over the 31.5 million seconds in a year, that is around 54,000 cubic metres every second, comparable to the average discharge of a very large river, except that the fluid is air and must be pushed through a chemically active surface rather than simply allowed to flow. Compare this with capturing carbon dioxide at a smokestack. The exhaust from a coal-fired power station is typically around 12 to 15 percent carbon dioxide, a cement kiln's often 20 percent or more, and a natural gas turbine's around 4 percent. Coal flue gas is therefore roughly 300 times more concentrated than ambient air. A capture unit on a coal plant handles hundreds of times less gas per tonne captured. That is why point-source capture, despite its own difficulties, is consistently cheaper per tonne than air capture, and why every serious analysis of climate policy treats air capture as something to be used where point-source capture or outright elimination is impossible. The dilution also sets a thermodynamic floor, which Chapter 4 treats in detail. The minimum work needed to separate a gas present at 425 parts per million into a pure stream, at room temperature, is around 19 to 20 kilojoules per mole, which works out to about 0.44 gigajoules, or roughly 120 kilowatt-hours, per tonne of carbon dioxide. Separating it from 12 percent flue gas requires only about a quarter of that. These are ideal limits that no real machine approaches closely, but they establish a fact that cannot be engineered away: the more dilute the source, the more work it takes to concentrate it. Contactors and fans The part of a capture plant where air meets chemistry is called the contactor. Its design is a compromise among three things that pull in different directions: the area of chemically active surface the air touches, the speed at which the air moves, and the resistance the air meets as it passes through. More surface means more carbon dioxide captured per pass. Faster air means more carbon dioxide arriving per second. But both increase the pressure drop across the contactor, the loss of pressure the air suffers as it squeezes through narrow channels, and every pascal of pressure drop must be paid for in fan energy. The power a fan needs is the volumetric flow multiplied by the pressure drop, divided by the fan's efficiency. An illustrative calculation shows why designers fixate on this. Suppose a contactor imposes a pressure drop of 100 pascals, a modest figure; the atmosphere at sea level presses with about 101,000 pascals, so this is roughly a thousandth of an atmosphere. Moving 1.7 million cubic metres of air through it for each tonne of carbon dioxide requires 1.7 million times 100 joules, or 170 megajoules, of ideal work. With a realistic fan efficiency of 60 percent, that rises to about 280 megajoules per tonne, or close to 80 kilowatt-hours. If the contactor were redesigned with a pressure drop of 300 pascals to gain more surface, the fan bill would triple to around 240 kilowatt-hours per tonne, more than the entire thermodynamic minimum for the separation. The lesson is that air capture must be done gently: low velocities, short flow paths and open structures, with large frontal areas to compensate. That is why air contactors look nothing like the compact packed columns used in chemical plants. Carbon Engineering, the Canadian company whose liquid-solvent process underlies the Stratos plant in Texas, borrowed its contactor from the cooling-tower industry. Cooling towers already move enormous volumes of air across a wetted plastic packing at low pressure drop, and the industry has decades of experience making that packing cheap, durable and resistant to fouling. In Carbon Engineering's design, described in a 2018 paper in the journal Joule by David Keith and colleagues, air is drawn horizontally by fans through vertical slabs of structured plastic packing several metres deep, while a strong alkaline solution trickles down over the packing surfaces. The result is a long, low wall of fans rather than a tower, a cross-flow geometry that keeps the air path short and the frontal area large. Solid-sorbent plants take a different approach. Climeworks, the Swiss company that built the Orca and Mammoth plants in Iceland, packs its sorbent into filter structures housed in modular collector units, each with its own fans. Air is drawn through the filter until the sorbent approaches saturation; the unit then seals itself off for the release step. Because the collectors are modular, a plant is built by repeating them, which helps with manufacturing learning but means that every unit carries its own fans, dampers, seals and controls. Climeworks' third-generation design, which the company says it validated at full scale in 2024, uses large cubic collector modules about 26 metres on a side and 22.5 metres high, an attempt to gain economies of scale per unit while keeping the modular logic. A third family of contactors does without fans entirely, relying on natural wind. The most visible example is the passive approach developed from the work of Klaus Lackner, who first proposed capturing carbon dioxide from air for climate purposes in the late 1990s, in which sheets or tiles of sorbent are exposed to the breeze. Passive designs save fan energy but surrender control over airflow, which makes capture rates dependent on the weather and requires very large areas of material. Another passive approach, used by the California company Heirloom, exposes thin layers of a reactive mineral powder on stacked trays, letting the air reach them by natural movement within a covered structure. These choices trade energy for land and material, a trade examined again in Chapter 3. The arithmetic of air flow translates directly into the physical size of a plant. Take the illustrative one-million-tonne plant again, moving 54,000 cubic metres of air a second. If air passes through the contactor at about 1.5 metres per second, a typical design velocity for low-pressure-drop packing, the contactor must present a frontal area of about 36,000 square metres to the wind. Built as a single wall 10 metres high, it would run for about 3.6 kilometres. In practice the wall is broken into many shorter units arranged in rows, and the rows must be spaced apart so that one unit does not breathe the depleted exhaust of its neighbour. Air leaving a contactor carries a quarter or so less carbon dioxide than the air entering it, and if that air is drawn back in, capture per unit of fan energy falls. Plant layouts therefore resemble cooling-tower fields or solar farms more than chemical plants, with spacing set partly by the local wind regime. That gives a land footprint for the capture equipment itself of perhaps a few square kilometres per million tonnes a year, depending on spacing. It is small compared with the land needed to remove the same carbon biologically. A temperate forest growing vigorously might take up on the order of 10 tonnes of carbon dioxide per hectare per year, so removing a million tonnes a year would need around 100,000 hectares, or 1,000 square kilometres, of new forest, several hundred times the footprint of the machines. The comparison changes when the energy supply is counted. If a direct air capture plant is powered by solar panels, the solar farm will typically occupy much more land than the contactors, though still far less than a forest of equivalent removal, and on land that need not be fertile. Noise and visual impact are the other physical consequences. A field of large, slow fans produces a continuous low-frequency hum, and the plants are industrial facilities with pipework, compressors and storage tanks. They are not the kind of installation that can be dropped unobtrusively into a suburb. This is one reason most proposed plants are sited in industrial zones, oil fields or remote areas. Weather, siting and the ambient fluid Air is not a uniform feedstock. Its temperature, humidity and pressure change by the hour, and each affects capture chemistry. Humidity matters most. Water vapour is typically present at concentrations tens of times higher than carbon dioxide, and many sorbents take it up alongside the target gas. For some amine sorbents, a little moisture helps, because it opens a reaction pathway that captures more carbon dioxide per active site. But water that is adsorbed must later be desorbed, and evaporating water absorbs a great deal of heat, so a humid climate can quietly inflate the energy bill. Liquid-solvent contactors face the opposite problem: in dry, hot climates the aqueous solution loses water by evaporation, which must be replaced, and in some designs the water loss runs to several tonnes per tonne of carbon dioxide captured. Water balance is one of the reasons Carbon Engineering's process was engineered with the solution concentration and temperature chosen to suit particular climates. Temperature matters next. Colder air holds less moisture, which can help sorbents, but reaction rates slow in the cold, and liquid solutions can freeze. Iceland's cool, windy climate suits solid-sorbent capture in some respects, and its abundant geothermal heat and electricity make it attractive for the release step, but a sub-Arctic winter is hard on any outdoor machinery, and the plants there have run well below their nameplate capacities while the company worked through technical and operational problems. Siting also depends on what happens downstream. A plant must be close to cheap, low-carbon energy, close to a suitable storage site or pipeline, and in a jurisdiction that will permit both. These constraints rarely coincide. Iceland has geothermal energy and reactive basalt but a small industrial base. The Permian Basin of West Texas, where Stratos is being built, has deep saline formations, pipelines, drilling crews and a regulatory culture familiar with injection wells, but much of its energy still comes from natural gas. Louisiana, where the federally supported Project Cypress hub was planned, has thick sedimentary formations along the Gulf Coast and a large industrial workforce. The geography of air capture is in practice the geography of storage and energy, not of air, since the air is the same everywhere. Gross capture and net removal The final concept this chapter must establish is the difference between capturing carbon dioxide and removing it. A plant that captures a tonne of carbon dioxide from the air has removed a tonne from the atmosphere only if nothing it did along the way put carbon dioxide back. In practice, almost everything does. The fans and pumps use electricity, which may come from fossil sources. The release step needs heat, which may come from burning natural gas. The sorbents and solvents are manufactured chemicals with their own embodied emissions, and they degrade and must be replaced. The steel and concrete of the plant were made with energy. The captured gas must be compressed, transported and injected, all of which use more energy. Life-cycle assessment is the discipline of adding up these flows. One of the most careful studies, published in Nature Energy in 2021 by Sarah Deutz and André Bardow, examined Climeworks' process at its plants in Switzerland and Iceland. It found that the process removed carbon on a net basis, with carbon removal efficiencies, meaning net removal as a fraction of gross capture, of roughly 85 to 93 percent, depending mainly on the carbon intensity of the energy supply. Those are good results, achieved with low-carbon Icelandic energy and careful accounting. With a fossil-heavy grid, the same machine could remove little or nothing on balance. The distinction also matters because some plants deliberately capture more than they remove. In Carbon Engineering's published design, natural gas is burned in pure oxygen to heat the kiln that releases the carbon dioxide. The carbon dioxide from that combustion is captured together with the carbon dioxide taken from air, so the plant's output stream is larger than its air capture. That is a legitimate design choice, since the combustion carbon dioxide is captured rather than vented, but it means that the tonnes leaving the plant are not all tonnes removed from the air. Any accounting that reports output as removal is wrong. Chapters 2 and 3 now turn to the chemistry inside the contactors, beginning with the oldest and most industrially familiar approach: dissolving carbon dioxide in a strong alkaline liquid and then recovering it through the chemistry of lime. Chapter 2: Liquid Solvents and the Lime Loop The first practical method for removing carbon dioxide from air was not designed to fight climate change. It was designed to keep people and machines alive. Submarines, spacecraft and early air-separation plants all needed to strip carbon dioxide from enclosed atmospheres or feed streams, and the chemistry they used was alkaline: a strong base, such as sodium or potassium hydroxide, or lithium hydroxide in canisters on spacecraft, reacts eagerly with carbon dioxide, which is a weak acid when dissolved in water. The reaction is fast, the materials are cheap, and the chemistry has been understood since the nineteenth century. What those applications did not need was to recover the carbon dioxide and reuse the base. A spacecraft simply carries enough lithium hydroxide for the mission and throws the spent canisters away. A climate-scale air capture plant cannot do that; it would consume more than a tonne of hydroxide for every tonne of carbon dioxide captured. The entire engineering challenge of liquid-solvent air capture lies in closing that loop: catching carbon dioxide in a basic solution, then getting it back out as a pure gas while regenerating the base. The most developed answer runs through the oldest industrial chemistry of all, the chemistry of lime. Catching carbon dioxide in a strong base Inside a liquid-solvent contactor, a solution of potassium hydroxide flows over plastic packing while air is drawn through. When a carbon dioxide molecule meets the liquid surface, it dissolves and reacts with hydroxide ions to form carbonate. In simplified terms, one molecule of carbon dioxide and two of potassium hydroxide become one of potassium carbonate and one of water. The reaction is strongly favoured. Hydroxide solutions bind carbon dioxide so tightly that the equilibrium concentration of carbon dioxide above them is essentially zero, which is why they can capture efficiently from a gas at only 425 parts per million. In a sense, that very strength is the root of the problem that the rest of the process must solve. A sorbent that binds carbon dioxide tightly releases it only reluctantly, and the energy needed to release it is related to how tightly it was bound. Rate, not equilibrium, is what limits the contactor. Carbon dioxide must diffuse from the bulk air to the liquid surface, dissolve, and react before it is swept away. The reaction in the thin liquid film near the surface is fast enough at high hydroxide concentrations that the overall rate is governed mainly by the wetted area and the residence time of air in the packing. That is why the contactor design discussed in Chapter 1 matters so much: the cheapest way to get more carbon dioxide out of the air is not better chemistry but more wetted surface per unit of pressure drop. Potassium hydroxide was chosen over sodium hydroxide for practical reasons. Potassium carbonate is more soluble, which reduces the risk of solid salts crystallising on the packing, and potassium solutions have favourable viscosity and mass-transfer properties. The solution leaving the contactor is a mix of unreacted hydroxide and carbonate, and it now carries the captured carbon in dissolved form. The next steps take that carbon from the liquid and put it into a solid. The calcium loop: pellets and the calciner The key insight of the Carbon Engineering process, borrowed from the pulp and paper industry where a similar loop has been used for over a century, is to swap the carbonate from potassium onto calcium. Calcium carbonate is limestone. It is insoluble, so it can be separated from the liquid as a solid, and it can be heated to drive off carbon dioxide. The potassium, freed from its carbonate, becomes hydroxide again and returns to the contactor. This exchange, called causticisation, happens when the carbonate-rich solution meets calcium hydroxide, known as slaked lime. Calcium carbonate precipitates, and potassium hydroxide is regenerated. In kraft pulp mills, the analogous reaction produces a fine lime mud that must be filtered, washed and dried before it can be reheated, which is energy-intensive and messy. Carbon Engineering instead adapted a device used in municipal water softening, the pellet reactor. In a pellet reactor, the solution flows upward through a bed of small seed particles. Calcium carbonate deposits on the surface of the seeds rather than forming a fine mud, so the particles grow layer by layer into millimetre-scale pellets, somewhat like hailstones. Grown pellets sink to the bottom and are removed, while fresh seeds are added at the top. The value of this step is easy to underestimate. Pellets drain and dry far more easily than mud, they carry much less water into the next stage, and they can be handled with ordinary conveyors. Every kilogram of water that reaches the kiln must be evaporated, at a cost of about 2.3 megajoules per kilogram, so keeping the solids dry directly lowers the energy bill. The pellet reactor is one of the places where an unglamorous piece of borrowed equipment does as much for the economics as any chemical innovation. The pellets of calcium carbonate are now heated in a calciner to about 900 °C. At that temperature, calcium carbonate decomposes into calcium oxide, or quicklime, and carbon dioxide gas. This is the same reaction that makes cement kilns one of the world's largest sources of industrial carbon dioxide, and it has the same thermodynamic cost. The decomposition absorbs about 178 kilojoules per mole of carbon dioxide released, which is about 4 gigajoules per tonne, before counting the energy needed to heat the solids up to reaction temperature and the losses from a hot vessel. Two design choices make the calciner the heart of the process's energy use and its carbon accounting. The first is how to supply the heat. A conventional cement kiln burns fuel in air, which dilutes the released carbon dioxide with nitrogen and defeats the purpose. Carbon Engineering's design uses oxy-fired combustion: natural gas is burned in nearly pure oxygen, supplied by an air separation unit, so the combustion products are mainly carbon dioxide and water. When the water is condensed out, what remains is a concentrated stream containing both the carbon dioxide that came from the air and the carbon dioxide from burning the gas. Both are compressed and sent to storage. The oxygen for Stratos is supplied by a dedicated air separation plant built nearby by the industrial gas company Matheson. The second choice is the reactor type. Carbon Engineering uses a circulating fluidised bed calciner, in which the pellets are suspended and mixed in a rising stream of hot gas, giving good heat transfer and uniform temperature. Fluidised beds are well established in power generation and minerals processing, which lowers the technical risk of scaling up. The quicklime leaving the calciner is then slaked, meaning reacted with water to form calcium hydroxide. Slaking releases heat, about 65 kilojoules per mole, and in a well-integrated plant that heat is recovered and put to use, for example in drying incoming pellets or preheating streams. The slaked lime returns to the pellet reactor, and the loop is closed. Two coupled cycles turn continuously: a potassium cycle between contactor and pellet reactor, and a calcium cycle between pellet reactor, calciner and slaker. A mass balance makes the scale of the calcium loop concrete. Every tonne of carbon dioxide released in the calciner corresponds to about 2.27 tonnes of calcium carbonate decomposed, since calcium carbonate has a molar mass of 100 grams against 44 for carbon dioxide. A plant capturing a million tonnes a year from the air, plus the carbon dioxide from its own gas combustion, must therefore grow, dry, heat and decompose well over two million tonnes of limestone pellets a year, roughly the throughput of a substantial cement plant's kiln, and then slake and recycle the resulting quicklime. The calcium is not consumed, but it is not perfectly conserved either. Each pass through the calciner slightly degrades the reactivity of the lime, impurities accumulate in the loop, and fines are lost, so a fraction of the inventory must be purged and replaced with fresh calcium carbonate. The potassium loop suffers smaller losses, mostly as mist and drift carried out of the contactor by the air stream. That drift is one of several practical issues that do not appear in simplified diagrams. Potassium hydroxide is strongly caustic. A contactor that wets enormous areas of packing with it while blowing air across them must be designed to keep droplets from escaping, using mist eliminators of the kind found in cooling towers, both to conserve chemicals and to protect workers, equipment and nearby vegetation. The solution also picks up dust, pollen and other airborne impurities, which must be filtered out or purged. And because the contactor is exposed to the weather, its water balance swings with rain, temperature and humidity, so the plant must continually adjust its make-up water and solution concentration. None of these problems is exotic; cooling towers, pulp mills and water-treatment plants solve versions of each daily. But their combination, at the scale of a large air capture plant, explains part of why commissioning such plants has taken longer than planned. What the numbers say The 2018 Joule paper by Keith, Holmes, St. Angelo and Heidel remains the most detailed public engineering and cost analysis of a liquid-solvent air capture plant. It described a plant designed to capture one million tonnes of carbon dioxide a year from the atmosphere, based on data from a pilot plant that Carbon Engineering had operated in Squamish, British Columbia, since 2015. Its energy figures are worth stating exactly, because they are among the few that have been published at this level of engineering detail. In the configuration where all energy comes from natural gas, with a gas turbine generating the plant's electricity, the plant required about 8.81 gigajoules of natural gas per tonne of carbon dioxide captured from air. In an alternative configuration that took electricity from the grid, it required about 5.25 gigajoules of gas plus 366 kilowatt-hours of electricity per tonne. The authors estimated levelised costs between 94 and 232 United States dollars per tonne, depending on financial assumptions and design choices. These figures were much lower than the roughly 600 dollars per tonne suggested in a 2011 report by the American Physical Society, and they were widely cited as evidence that air capture could become affordable. Two points of caution are necessary. First, these were engineering estimates for an nth-of-a-kind plant, the cost expected once a technology has been built many times and the learning has been captured, not measured costs from an operating facility. First-of-a-kind plants almost always cost more. Second, the figures referred to carbon dioxide captured from air, not net removal. A plant that burns natural gas emits upstream methane from gas production and delivery, and that leakage must be subtracted. The scale-up has been slower and more expensive than the paper's optimism suggested. In 2023, Occidental Petroleum acquired Carbon Engineering for about 1.1 billion dollars, and its subsidiary 1PointFive began building Stratos in Ector County, Texas, designed for up to 500,000 tonnes a year. Its start-up target slipped from 2025 to the end of 2025, then to the second quarter of 2026. During commissioning in 2026, Occidental reported that the air contactors and central processing facility had performed as expected but that an unrelated component problem had delayed operations, and by August 2026 the company was pointing to the end of the year for start-up. In April 2025, Stratos received the first Class VI permits issued by the United States Environmental Protection Agency for storing carbon dioxide from a direct air capture facility, wells that will inject into deep saline formations beneath the plant. Variations, strengths and limits The core chemistry of the lime loop can be rearranged. The most notable variation is the approach taken by Heirloom, a company founded in California in 2020, which removes the liquid solvent entirely and uses the calcium cycle alone. Heirloom starts with limestone, heats it in a kiln to drive off carbon dioxide, and then hydrates the resulting quicklime. The calcium hydroxide powder is spread in thin layers on large trays stacked in tall racks and left exposed to the air. Calcium hydroxide naturally reabsorbs carbon dioxide from the atmosphere and reverts to limestone, the same process by which lime mortar hardens over years. Heirloom's contribution is to speed it up by controlling particle size, layer thickness, humidity and airflow, so that the material carbonates in days rather than years. Robots move the trays from the racks back to the kiln, where the cycle repeats. The kiln is electrically heated, which allows the carbon dioxide released to come out undiluted and lets the plant draw on low-carbon electricity rather than burning gas. Heirloom's approach removes the potassium contactor and the pellet reactor, and with them much of the complexity of handling a corrosive liquid. It also removes the fans, because natural air movement suffices for slow carbonation. What it gives up is speed: material sits on trays for days, so a large plant requires a great deal of material and space. And it keeps the most energy-intensive step, calcination at around 900 °C, so its thermodynamic floor is no lower than the liquid-solvent process. The company opened a facility in Tracy, California, in 2023, with a capacity of up to 1,000 tonnes a year, and has been planning larger facilities in Louisiana as part of the Project Cypress hub. Other researchers have explored replacing calcination with lower-temperature chemistry. One route uses electrochemistry to regenerate the base: an electrochemical cell, often using bipolar membranes, splits water into acid and base, and the acid is used to release carbon dioxide from the carbonate solution while the base is returned to the contactor. This trades heat at 900 °C for electricity, which can be zero-carbon, and avoids the calcium cycle altogether. Its challenge is cost: bipolar membranes and electrochemical stacks are expensive per unit of throughput, and their energy use per tonne remains high in published demonstrations. These pathways remain at pilot scale. The liquid-solvent route has three great strengths. It uses cheap, abundant materials: potassium hydroxide, limestone and water. It borrows its major equipment from mature industries, so its scale-up risks are those of large chemical plants rather than of new materials. And it runs continuously, with liquid and solids flowing steadily rather than in batches, which suits very large plants. Its weakness is the calciner. Releasing carbon dioxide from calcium carbonate requires high-temperature heat, and high-temperature heat is the hardest kind of energy to supply without carbon emissions. Burning natural gas with oxygen and capturing the resulting carbon dioxide works, but it ties a climate technology to fossil fuel supply chains and methane leakage. Electrifying the kiln, as Heirloom has done, or heating it with hydrogen or concentrated solar energy, are possible but not yet proven at large scale and low cost. That is the motivation for the second family of capture technologies, which uses solid materials that bind carbon dioxide less tightly and therefore release it at temperatures around 100 °C, within reach of waste heat, heat pumps and geothermal steam. The price, as Chapter 3 explains, is a set of different problems: batch operation, degrading materials, and an intimate, sometimes troublesome, relationship with water. Chapter 3: Solid Sorbents and the Art of the Swing A liquid solvent captures carbon dioxide by turning it into a dissolved salt. A solid sorbent captures it by holding it on a surface. The distinction sounds small, but it changes almost everything about how a capture plant is built and run. Solids cannot be pumped around a loop like a liquid, so the process must move conditions rather than materials: the sorbent stays in place while the plant cycles it between capturing and releasing. Engineers call each such cycle a swing, named after whatever variable is swung, whether temperature, pressure, humidity or electrical potential. The art of solid-sorbent air capture is designing a material and a swing that together capture a lot, release cleanly, use little energy and survive thousands of cycles. Amines on a scaffold Most solid sorbents now used or seriously proposed for air capture are amines, organic molecules containing nitrogen atoms with a lone pair of electrons that can bond to the carbon atom of carbon dioxide. Liquid amine solutions, especially monoethanolamine, have been used since the 1930s to scrub carbon dioxide from natural gas and, more recently, from power-plant flue gas. They work well at high concentrations, but a liquid amine scrubber operating on ambient air would need to pump and heat enormous volumes of water to recover tiny quantities of carbon dioxide. The solution is to anchor the amine to a porous solid. Three broad strategies exist. In the first, an amine-rich polymer such as polyethyleneimine is physically loaded into the pores of a support such as silica or alumina, like water soaked into a sponge. In the second, amine groups are chemically grafted onto the support surface, often using aminosilanes that bond to silica. In the third, the amine is part of the support itself, as in certain resins. Each strategy trades capacity against stability: loading more amine increases the amount of carbon dioxide captured per kilogram but can block pores and slow diffusion, while grafting gives sturdier attachment but fewer sites. The chemistry at each site depends on moisture. Under dry conditions, two amine groups typically cooperate to capture one molecule of carbon dioxide, forming a carbamate on one and a protonated ammonium on the other. That caps the theoretical capacity at one carbon dioxide per two nitrogen atoms. When water is present, a second pathway forms bicarbonate, which can capture up to one carbon dioxide per nitrogen. Humid air can therefore raise capacity, which is one reason solid-sorbent plants are often cheerful about moderate humidity. But water itself also adsorbs onto the sorbent in quantities that may far exceed the carbon dioxide, and every kilogram of it must be removed again in the release step at a large cost in heat. The binding energy of carbon dioxide to amines is moderate. Measured heats of adsorption on supported amines typically fall around 60 to 90 kilojoules per mole, strong enough to capture from very dilute air, but much weaker than the bond in calcium carbonate that the liquid-solvent process must break. That difference is what allows amine sorbents to be regenerated at around 80 to 120 °C instead of 900 °C. The temperature-vacuum swing and its hidden costs The dominant release method for amine sorbents is the temperature-vacuum swing. Its steps, as practised in Climeworks' plants and many others, follow a regular cycle. In the adsorption step, fans draw ambient air through the sorbent filter for a period that can run to hours, until the sorbent approaches a target loading. In the second step, the collector closes its inlet and outlet dampers to isolate the sorbent from the air. A vacuum pump then reduces the pressure in the chamber, removing most of the residual air so that the product will not be diluted with nitrogen and oxygen. In the third step, the sorbent is heated, typically to around 100 °C, often with steam or a heat-transfer fluid. Heat drives the equilibrium back toward free carbon dioxide, which leaves the sorbent and is drawn off by the vacuum system as a concentrated stream, with water vapour that is condensed out. In the final step, the sorbent is cooled, the dampers open, and adsorption begins again. The vacuum does two jobs. It purges air from the chamber and it lowers the partial pressure of carbon dioxide over the sorbent, which encourages release at a lower temperature than heat alone would need. Steam, where used, does something similar: by sweeping through the bed, it lowers the carbon dioxide partial pressure and delivers heat directly to the sorbent surface. Steam then condenses and is easily separated, unlike nitrogen. Global Thermostat, an American company acquired in 2023 by Zero Carbon Systems, built its process around steam-stripping of amine sorbents held on ceramic honeycomb monoliths of the kind used in car catalytic converters, chosen for their very low pressure drop. Because the cycle alternates, a plant needs many collectors, staggered so that at any moment some are adsorbing while others are desorbing. That produces a steady product flow but multiplies the dampers, seals, valves and controls. Each collector must be airtight enough to hold a vacuum, then fully open to the wind, thousands of times a year. Mechanical reliability of these parts, unglamorous as it is, has proved one of the practical constraints on solid-sorbent plants. The most important energy loss in a temperature swing has nothing to do with carbon dioxide. Every time the bed is heated from ambient temperature to around 100 °C and cooled again, the sorbent, its support, the filter structure, the housing and any adsorbed water all absorb heat. This sensible heat is lost to the cycle unless recovered, and in many designs it exceeds the heat actually needed to break the carbon dioxide bonds. A simple illustrative calculation makes the point. Suppose a sorbent holds 1 mole of carbon dioxide per kilogram in each cycle, a respectable working capacity for an amine sorbent under air-capture conditions, and that the material has a specific heat of about 1 kilojoule per kilogram per degree. Heating it by 80 degrees costs 80 kilojoules per kilogram, or 80 kilojoules per mole of carbon dioxide released, before counting the support, the housing or the water. The binding energy itself is of the same order. If the working capacity were halved, by degradation or by humid air competing for sites, the sensible heat per mole would double. That is why working capacity, rather than equilibrium capacity under ideal laboratory conditions, is the single most important property of a sorbent for air capture. Working capacity also sets how much sorbent a plant must hold, which a second illustrative calculation shows. A plant designed to capture 36,000 tonnes a year, Mammoth's nameplate, must capture about 820 million moles of carbon dioxide annually. Suppose each collector completes four full cycles a day, a plausible order for designs in which adsorption takes a few hours and desorption about an hour, giving about 1,460 cycles a year. If the sorbent releases one mole per kilogram in each cycle, the plant needs about 560 tonnes of sorbent in its collectors. If degradation and humidity reduce the working capacity to half a mole per kilogram, the requirement doubles to more than 1,100 tonnes, and if cycles must lengthen because uptake slows as the sorbent ages, it rises again. Specialised amine sorbents are not commodity chemicals, so the cost of filling and refilling collectors at this scale is substantial. For a plant a hundred times larger, one capturing several million tonnes a year, the sorbent inventory would run to tens of thousands of tonnes, a quantity that would require a dedicated chemical supply chain. Designers attack the empty heat in several ways. They choose thin structures with low mass, so that less material is heated per unit of sorbent. They use heat exchange between collectors, so that the heat released by one cooling bed preheats another. They minimise water uptake. And they try to increase working capacity through better materials. Climeworks says its third-generation sorbent, built on a structured material, doubles capture capacity per module and halves energy consumption compared with its previous generation, though independent performance data at scale have not yet been published. A sorbent that degrades has to be replaced, and replacement carries both cost and embodied emissions. Amines face several degradation mechanisms. The most damaging is oxidation: at elevated temperature, the oxygen in air attacks amine groups and gradually destroys them. That is why the vacuum step, which removes oxygen before heating, protects the sorbent as well as purifying the product. Dry carbon dioxide at high temperature can also convert amines into stable urea compounds that no longer capture. Some amines volatilise and are lost. Contaminants in the air, including sulphur dioxide, nitrogen oxides and fine dust, can poison sites or block pores. Sorbent lifetime is consequently one of the critical economic parameters that companies guard most closely. Climeworks has said its newest materials are designed to last three times longer than its previous ones. The arithmetic of lifetime is straightforward: if a kilogram of sorbent costs a certain amount and captures a certain number of kilograms of carbon dioxide over its life, that ratio sets a floor on the material cost per tonne. A sorbent that survives tens of thousands of cycles makes the material cost trivial; one that loses a third of its capacity in a year can dominate the cost. Frameworks, resins and electricity Amine-loaded silicas are not the only solid sorbents. Three other families deserve attention, because each tries to escape a limitation of the temperature swing. Metal-organic frameworks are crystalline materials built from metal ions joined by organic linkers, with internal surface areas that can exceed thousands of square metres per gram. Researchers can tune their pore sizes and chemistry with unusual precision. One notable family, studied in the group of Jeffrey Long at the University of California, Berkeley, combines a magnesium-based framework with appended diamines. In 2015, Thomas McDonald, Long and colleagues reported in Nature that carbon dioxide inserts cooperatively into these materials, so that uptake switches on sharply at a threshold pressure, like a chain of dominoes falling. That step-shaped behaviour could in principle allow very large working capacity with a small temperature swing. Metal-organic frameworks face practical hurdles, including cost of synthesis at tonne scale, stability under humid conditions, and the difficulty of forming fine crystalline powders into robust, low-pressure-drop structures, but they remain a field of active development. Ion-exchange resins open a different route, the moisture swing. Klaus Lackner, now at Arizona State University, and colleagues found that certain resins carrying quaternary ammonium groups absorb carbon dioxide when dry and release it when wet. The explanation lies in the shifting balance between carbonate and bicarbonate ions in the resin's water-starved pores: as water evaporates, the carbonate becomes a stronger base and takes up carbon dioxide; when water is added, the equilibrium shifts back and carbon dioxide is released. The energy for this swing comes from the evaporation of water, which in a dry climate is supplied for free by the atmosphere itself. The cost is water consumption and a product that emerges only modestly concentrated, often a few percent carbon dioxide, which then needs further concentration before storage. Electrochemical sorbents, finally, use electricity to change the binding affinity directly. In 2019, Sahag Voskian and T. Alan Hatton at the Massachusetts Institute of Technology described an electro-swing process in which quinone molecules on an electrode bind carbon dioxide when electrically reduced and release it when oxidised. The appeal is that electricity can be supplied precisely and efficiently, without heating a mass of support material. The difficulties are electrode cost, the sensitivity of reduced quinones to oxygen in air, and the challenge of scaling electrochemical surfaces to the vast areas that air capture requires. A related family of approaches uses electrochemistry to shift the acidity of a liquid, as noted at the end of Chapter 2. Comparing the families The approaches described in this and the previous chapter can be set side by side. Table 1 compares the main families of capture chemistry by how they release carbon dioxide, the form of energy they need, and their stage of development as of late 2026. Temperatures and maturity descriptions are drawn from the developers' published descriptions and from the engineering literature cited in the Notes. Table 1. Main families of direct air capture chemistry. Approach How CO2 is released Typical release conditions Main energy input Status, late 2026 Hydroxide solvent with lime loop (Carbon Engineering, 1PointFive) Calcining calcium carbonate About 900 °C High-temperature heat, usually oxy-fired gas 500,000 t/yr plant in commissioning Mineral looping (Heirloom) Calcining limestone after passive carbonation About 900 °C, electric kiln Electricity Up to 1,000 t/yr plant operating Supported amine, temperature-vacuum swing (Climeworks and others) Heat plus vacuum About 80–120 °C Low-temperature heat, electricity Plants of 4,000 and 36,000 t/yr nameplate, running far below Moisture-swing resin Wetting the sorbent Ambient temperature Water evaporation, electricity Pilot scale Electrochemical swing Changing electrode potential Ambient temperature Electricity Laboratory and pilot scale No family is clearly superior on all counts. The high-temperature routes use cheap materials and continuous processes but need the most demanding kind of heat. The amine route needs only modest heat but depends on expensive, degradable sorbents and complex batch hardware. The ambient-temperature routes avoid heat altogether but have not yet demonstrated their costs at scale. What all of them share is the release step. Whatever the chemistry, a tonne of carbon dioxide must be persuaded to leave a material that was chosen precisely because it holds carbon dioxide tightly. The next chapter turns to the thermodynamics of that persuasion: how much energy it fundamentally requires, how much real processes actually use, and why the difference matters for whether air capture can ever operate at the scale of billions of tonnes. Chapter 4: The Thermodynamics of Letting Go Catching carbon dioxide from air is the visible part of the process, and it attracts the attention: the long walls of fans, the filters, the photographs. But the capture step is chemically easy. Any sufficiently strong base will do it. The hard part, the part that decides cost, energy use and in the end the climate value of the whole enterprise, is letting go: releasing the captured carbon dioxide as a concentrated gas and returning the capture material to its starting state. This chapter explains why that step costs what it does, how far real processes are from the theoretical limits, and why the quality of the energy supplied matters as much as its quantity. The floor set by mixing Gases mix spontaneously. Open a bottle of perfume in a room and the scent spreads until it is everywhere; it never gathers itself back into the bottle. In thermodynamic language, mixing increases entropy, and reversing it, which is what separation means, requires work. The amount of work has a precise lower limit that depends only on temperature and the concentrations involved. For a component present at a mole fraction x in an ideal gas mixture, the minimum work to extract a small amount of it as a pure gas at the same temperature and total pressure is the gas constant multiplied by the absolute temperature and by the natural logarithm of one over x. At 15 °C, or 288 kelvin, the product of the gas constant and temperature is about 2.4 kilojoules per mole. For carbon dioxide at 425 parts per million, one over x is about 2,350, and its natural logarithm is about 7.8. The minimum work is therefore about 18.6 kilojoules per mole; at 25 °C it is slightly more, about 19.2 kilojoules. Dividing by the molar mass of 44 grams gives roughly 0.42 to 0.44 gigajoules per tonne, or about 120 kilowatt-hours. Real processes do not extract a vanishingly small amount from an infinite reservoir; they take a fraction of the carbon dioxide out of a finite stream, which slightly changes the result, and they deliver product at elevated pressure for storage, which adds compression work. But the order of magnitude stands. The floor is about half a gigajoule per tonne, and it rises logarithmically as the source becomes more dilute. For comparison, the same calculation for coal flue gas at 12 percent carbon dioxide gives about 5 kilojoules per mole, roughly a quarter of the air figure. The logarithm is merciful: air is 300 times more dilute than flue gas, yet the minimum work is only about four times higher. If air capture plants operated near this floor, the energy problem would be modest. Capturing a billion tonnes a year at the theoretical minimum would require about 0.44 exajoules of work, well under a tenth of one percent of global primary energy use. The interesting question is why real plants use ten or more times as much. Where the energy actually goes Every real separation falls short of the ideal because real processes are irreversible: they happen at finite speeds, across finite temperature differences, with materials that bind more tightly than the minimum necessary. In air capture, the gap between ideal and actual opens in four places. The first is binding strength. To capture carbon dioxide from 425 parts per million at a useful rate, a material must bind it substantially more strongly than the bare minimum. A material that bound carbon dioxide only just strongly enough to be in equilibrium with air would capture nothing at a practical rate. Chemists therefore choose sorbents whose binding enthalpy is several times the minimum work: around 60 to 90 kilojoules per mole for amines, and around 178 kilojoules per mole for the calcium carbonate decomposition at the heart of the lime loop. Releasing the carbon dioxide means paying back that enthalpy as heat. A strong bond is good for capture and bad for release, and there is no escaping the trade-off; there is only choosing where on it to stand. The second is sensible heat, discussed in Chapter 3. In a temperature swing, the whole mass of sorbent, support, structure and adsorbed water must be heated and cooled every cycle. Unless that heat is recovered, which is difficult in batch processes, it is wasted. In the lime loop the same problem appears as the heat needed to raise solid pellets from ambient temperature to 900 °C, partly recovered by heat exchange with hot outgoing streams. The third is water. Evaporating a kilogram of water takes about 2.3 megajoules, more than half the reaction enthalpy for a kilogram of carbon dioxide released from limestone. Any water that enters the release step, whether as moisture adsorbed on a sorbent or as liquid carried on calcium carbonate pellets, imposes a large and often underestimated penalty. The fourth is the mechanical energy of moving fluids: fans that push air through contactors, pumps that circulate solutions, vacuum pumps that evacuate collectors, and compressors that raise the product to pipeline pressure. Compressing carbon dioxide from atmospheric pressure to about 15 megapascals, a typical pipeline pressure, takes on the order of 100 kilowatt-hours per tonne with multistage intercooled compressors. Fans add tens to a few hundred kilowatt-hours per tonne, depending on pressure drop, as the illustrative calculation in Chapter 1 showed. Table 2 sets these contributions against the theoretical minimum for the two most developed process families. The liquid-solvent figures come from the 2018 engineering study by Keith and colleagues; the solid-sorbent figures are ranges reported in the technical literature for temperature-vacuum swing processes, including reviews by Mahdi Fasihi and colleagues in 2019 and by Christoph Beuttler and colleagues at Climeworks in the same year. They are indicative, not measured performance from today's operating plants. Table 2. Indicative energy use per tonne of CO2 captured, compared with the thermodynamic minimum. Item Thermodynamic minimum Liquid solvent, lime loop Solid amine, temperature-vacuum swing Heat Not applicable 5.25 GJ natural gas, burned at about 900 °C Roughly 5–7 GJ heat at about 100 °C Electricity Not applicable About 366 kWh from the grid Roughly 200–300 kWh Total work-equivalent About 0.44 GJ (about 120 kWh) Several times to over ten times the minimum Several times the minimum, depending on heat source Main sources of loss None Calcination enthalpy, solids heating, oxygen production Sensible heat of sorbent, water desorption, vacuum Heat quality and the limits of practice The most useful idea for anyone evaluating air capture energy claims is that a gigajoule of heat at 100 °C and a gigajoule of heat at 900 °C are not equivalent. Thermodynamics distinguishes between energy and exergy, the portion of energy that could in principle be converted to work. Heat at high temperature carries more exergy than the same quantity of heat at low temperature, because it could drive an engine with greater efficiency. The fraction of heat that is exergy is given by the Carnot factor: one minus the ratio of the ambient temperature to the temperature of the heat, both in kelvin. For heat at 100 °C, or 373 kelvin, with surroundings at 288 kelvin, the Carnot factor is about 0.23. Only about a quarter of that heat is equivalent to work. For heat at 900 °C, or 1,173 kelvin, the factor is about 0.75. Apply this to the two process families. A solid-sorbent plant using 6 gigajoules of heat at 100 °C consumes, in exergy terms, about 1.4 gigajoules from that heat, plus about 0.9 gigajoules of electricity at 250 kilowatt-hours. Its total exergy input is roughly 2.3 gigajoules per tonne, against a minimum of 0.44: a second-law efficiency of around 19 percent. A liquid-solvent plant burning 5.25 gigajoules of natural gas at high temperature consumes nearly all of that as exergy, since fuel is almost pure exergy, plus about 1.3 gigajoules of electricity. Its exergy input is around 6.5 gigajoules per tonne, giving a second-law efficiency of about 7 percent. These illustrative calculations, which ignore heat recovery details and depend on the exact figures assumed, should not be read as proving that one technology is better. Low-temperature heat is cheap and abundant only where a suitable source exists; a plant that must generate its 100 °C heat by burning fuel throws away most of the fuel's exergy, just as burning gas to heat a house does. What the calculation shows is where the opportunities lie. A solid-sorbent plant built next to geothermal steam, industrial waste heat, or a nuclear reactor's low-grade heat can use energy that has little alternative value. A liquid-solvent plant needs premium energy and can only lower its burden by recovering heat aggressively or using cheap, zero-carbon high-temperature heat, which barely exists today. The same logic illuminates heat pumps. A heat pump uses electricity to move heat from a cool source to a warmer sink. Lifting ambient heat to 100 °C is within the range of industrial heat pumps, with coefficients of performance that might plausibly fall around 2 to 3, meaning each kilowatt-hour of electricity delivers two to three kilowatt-hours of heat. For a solid-sorbent plant that needs around 1,700 kilowatt-hours of heat per tonne, a heat pump with a coefficient of 2.5 would cut the electricity required to generate that heat to about 680 kilowatt-hours. Adding 250 kilowatt-hours for fans and vacuum gives a fully electric plant using roughly a megawatt-hour of electricity per tonne. No heat pump can practically lift ambient heat to 900 °C, which is why fully electric versions of the lime loop must use resistive or other direct electric heating at a one-to-one conversion. Two famous analyses define the debate over how close real plants can get to the floor. In 2011, Kurt Zenz House and colleagues, writing in the Proceedings of the National Academy of Sciences, compiled data from many commercial separation processes and plotted their actual energy use against the theoretical minimum as a function of how dilute the target substance was. They found that processes separating very dilute substances typically operated at only a few percent of thermodynamic efficiency. Extrapolating that pattern, they argued that air capture would probably cost around 1,000 dollars per tonne and that claims of much lower costs should be treated with scepticism. The counter-argument, made implicitly by the Keith study and by others since, is that historical separation processes were never designed to minimise energy at extreme dilution because nobody needed them to, and that targeted engineering can do better than the historical pattern. The truth so far lies between. Real plant data, where available, have not shown the low costs and energy uses projected in optimistic studies, but neither have they confirmed the most pessimistic forecasts, because very few plants have run at scale long enough to establish reliable figures. The physical limit itself is not the binding constraint. Even at a second-law efficiency of 10 percent, the energy needed would be around 4.4 gigajoules per tonne, or about 1,200 kilowatt-hours of electricity-equivalent. The binding constraint is how energy use translates into cost and emissions. Gigatonnes, and the rest of the journey Consider what a billion tonnes a year of air capture would require, using the electric solid-sorbent example above at one megawatt-hour per tonne. That is about 1,000 terawatt-hours of electricity a year, roughly 3 percent of global electricity generation in the mid-2020s, which was a little over 30,000 terawatt-hours according to the energy think tank Ember. Without heat pumps, using direct electric heating for the 100 °C step, the figure would roughly double. A billion tonnes a year is about 2.7 percent of current fossil emissions. The arithmetic has a sharp implication. If that electricity is generated by fossil fuels, the plant may emit nearly as much as it captures. A megawatt-hour of electricity from an efficient natural gas plant emits about 0.35 to 0.4 tonnes of carbon dioxide; from coal, about 0.9 to 1 tonne. An air capture plant drawing a megawatt-hour per tonne from a coal-heavy grid would remove essentially nothing. Air capture is a sink only when it is powered by energy that would otherwise be either clean surplus or unavailable for other uses. Every megawatt-hour of clean electricity directed to air capture is a megawatt-hour not used to displace a fossil plant, and displacing fossil generation almost always avoids more carbon dioxide per unit of energy than air capture removes. Until grids are largely decarbonised, the climate value of a clean megawatt-hour is usually higher when it replaces coal or gas than when it runs fans and heaters. A further complication is time. Wind and solar power are cheapest when the weather is generous and scarce when it is not. An air capture plant that ran only on surplus renewable electricity would capture carbon dioxide cheaply per kilowatt-hour but would sit idle much of the time, spreading its heavy capital cost over fewer tonnes. The economics push in the opposite direction: expensive plants want to run all the time. The tension is the same one faced by electrolysers making hydrogen, and the answers are similar. A plant can draw on firm low-carbon power such as geothermal, hydroelectric or nuclear energy; it can store heat in insulated tanks or hot rock, so that cheap electricity at midday heats a reservoir that drives desorption at night; or it can accept a lower capacity factor in exchange for lower energy costs. Solid-sorbent plants have one modest advantage here: because their cycles are batch processes, they can adjust the timing of desorption to some degree, loading sorbent continuously while concentrating the energy-intensive release step in hours of cheap power. Liquid-solvent plants with large high-temperature calciners are less flexible, because kilns do not like to cycle between cold and hot. This is not an argument against air capture. It is an argument about sequence and siting. Air capture makes the most sense where its energy is genuinely additional: stranded geothermal resources, curtailed wind and solar, waste heat from industry or nuclear plants, and grids that are already nearly clean. It makes the least sense on a fossil-dominated grid. The energy story does not end when the carbon dioxide leaves the sorbent. To be stored, it must be dried, compressed to a dense phase, transported, and injected. Compression to pipeline conditions typically costs around 100 kilowatt-hours per tonne. Pipeline transport over a few hundred kilometres adds modest pumping energy. Injection into a deep formation requires sufficient wellhead pressure to overcome the formation's pressure, which may require further pumping, though the weight of the dense carbon dioxide column in the well helps. Mineralisation, the route used in Iceland by Carbfix, has a different energy profile. There, carbon dioxide is dissolved in water as it is injected, so the gas need not be compressed to a supercritical state, but large volumes of water must be pumped: the process requires on the order of 25 tonnes of water per tonne of carbon dioxide. In regions with abundant water and geothermal energy this is affordable; elsewhere it is a real constraint, and Carbfix has worked on using seawater to relax it. What happens after injection is a matter of geology rather than thermodynamics, though the two remain linked: the pressure and temperature of the rock determine the density of the carbon dioxide, the volume it occupies, and the forces that hold it in place. Chapter 5 follows the carbon dioxide underground. Chapter 5: How Rock Holds Carbon The ground beneath most continents and continental shelves is not solid in the way the word suggests. Sedimentary rocks such as sandstone and limestone formed from grains of sand, shell fragments and mud that settled in ancient seas, rivers and deserts. When those sediments were buried and cemented, they kept a fraction of open space between the grains, typically 10 to 30 percent of their volume in good reservoir rock. Those pores are filled, almost everywhere at depth, with salty water. In some places, a few of them hold oil or natural gas, trapped beneath layers of fine-grained rock that the fluids could not pass through. Those oil and gas fields are proof, by the evidence of millions of years, that the subsurface can hold buoyant fluids. Geological storage of carbon dioxide rests on that proof. This chapter explains how carbon dioxide behaves once injected into such rock and the four mechanisms by which it is held there. The mechanisms work on very different timescales, from the day of injection to tens of thousands of years, and the security of storage grows over time as one hands over to the next. Carbon dioxide under pressure At the surface, carbon dioxide is a gas about one and a half times as dense as air. Underground, with increasing depth, both temperature and pressure rise, typically by around 25 to 30 °C per kilometre and by about 10 megapascals per kilometre in water-filled rock. Above a temperature of 31.1 °C and a pressure of 7.38 megapascals, carbon dioxide passes its critical point and becomes a supercritical fluid, a state that has the density of a liquid but flows like a gas. In most sedimentary basins, those conditions are met at depths below about 800 metres. The density of supercritical carbon dioxide at storage depth typically ranges from about 500 to 800 kilograms per cubic metre, depending on the local temperature and pressure. That is several hundred times denser than the gas at the surface. A tonne of carbon dioxide that would occupy more than 500 cubic metres as a gas at surface conditions fits into one and a half to two cubic metres underground. This compaction is the reason storage is attempted at depth: it allows enormous masses to fit in a finite pore volume, and it is why the 800-metre mark appears in nearly every regulation and site-selection guideline. Even compressed, carbon dioxide is lighter than the brine around it. Formation brines typically have densities between about 1,000 and 1,200 kilograms per cubic metre. The difference means that injected carbon dioxide is buoyant: it tends to rise through the pore network until something stops it. Every trapping mechanism is, in one way or another, a way to stop or neutralise that buoyancy. Seals and stranded droplets The first and most immediate trap is a seal. Sedimentary basins contain layers of fine-grained rock, chiefly shales, mudstones and evaporites such as rock salt, whose pores are so small that buoyant fluids cannot enter them. The physical reason is capillary pressure. When two immiscible fluids such as carbon dioxide and brine meet in a narrow pore throat, surface tension creates a pressure difference that the non-wetting fluid must overcome to push through. In a shale with pore throats measured in nanometres, that threshold, the capillary entry pressure, can amount to several megapascals. As long as the buoyant pressure of a column of carbon dioxide stays below it, the carbon dioxide cannot enter the seal. The height of carbon dioxide column a seal can hold depends on its entry pressure and on the density contrast between carbon dioxide and brine. For a density contrast of 400 kilograms per cubic metre, each megapascal of entry pressure can support a column roughly 250 metres tall. Good seals can therefore hold substantial accumulations, which is why the same shales that have held natural gas for millions of years are the first candidates for carbon dioxide. A seal alone is not enough. If the reservoir beneath it is flat and continuous, the rising carbon dioxide will spread laterally along the base of the seal, potentially for tens of kilometres. To hold it in one place, geologists look for structures: domes, anticlines and fault blocks that form a closed high point, an upturned bowl in which buoyant fluid gathers. This is structural trapping. Stratigraphic traps achieve the same effect where the reservoir rock itself pinches out or changes laterally into an impermeable rock. Oil and gas accumulate in exactly these traps, which is why depleted fields are attractive storage targets. Many of the largest prospective storage resources, however, are in open saline formations, extensive reservoir layers without a single closed structure. There, the carbon dioxide plume migrates slowly upslope along the seal. Storage in such formations relies on the slower mechanisms described next to immobilise the carbon dioxide before it travels far. Sleipner, the longest-running storage project in the world, injects into exactly such a formation, and its plume behaviour has been tracked in detail for nearly three decades, as Chapter 7 describes. The second mechanism begins as soon as the carbon dioxide plume starts to move. As carbon dioxide pushes brine out of a pore, it occupies the pore centre while a film of brine clings to the grain surfaces, because most sandstones are water-wet. When the plume moves on, perhaps because it is rising buoyantly or spreading laterally, brine flows back into the pore behind it. The returning brine does not displace all the carbon dioxide. Some is cut off into isolated droplets, called ganglia, stranded in the larger pores and held there by the same capillary forces that make the seal work. Those droplets cannot move again without a pressure gradient far larger than any that buoyancy alone can supply. This process, called residual or capillary trapping, can immobilise a substantial fraction of the injected carbon dioxide within years to decades of injection. Laboratory measurements on reservoir rocks typically find that residual saturations, the fraction of pore volume left occupied by trapped carbon dioxide after brine reinvades, range from about 10 to 30 percent, depending on the rock's pore structure and wettability. A 2015 review led by Samuel Krevor at Imperial College London compiled these measurements and their implications for storage security. Residual trapping has an important engineering corollary: it is enhanced when the plume is forced to sweep through a large volume of rock rather than accumulating in a thick column. Some proposed injection strategies deliberately inject brine after carbon dioxide, or alternate the two, to accelerate trapping. And because residual trapping works without a closed structure, it is the main reason open saline formations can store carbon dioxide securely despite lacking the dome-shaped traps that hold oil and gas. Dissolving and turning to stone Carbon dioxide dissolves in water. Under typical storage conditions, a kilogram of brine can dissolve a few tens of grams of carbon dioxide, the amount decreasing with salinity. Once dissolved, the carbon dioxide is no longer a separate buoyant phase. Indeed, brine saturated with carbon dioxide is slightly denser, by around one percent, than the brine around it. That small density increase has a large consequence. Dense, carbon dioxide-laden brine at the top of the formation sits above lighter, unsaturated brine below, an unstable arrangement. Over time, it sinks in descending fingers, a process called convective mixing, while fresh brine rises to contact the remaining carbon dioxide and dissolve more. The dissolved carbon dioxide migrates downward, away from the seal, into the body of the formation. The rate depends on the permeability of the rock and the thickness of the formation and is generally measured in centuries to millennia for complete dissolution of a large plume, though onset can occur within years to decades in highly permeable formations. Solubility trapping is therefore a slow but powerful contributor. Studies of natural carbon dioxide reservoirs, places where volcanic or magmatic carbon dioxide has accumulated in rock over geological time, have confirmed its role. A 2009 paper in Nature by Stuart Gilfillan and colleagues used noble gas and carbon isotope tracers to study nine natural carbon dioxide gas fields in North America, China and Europe and concluded that dissolution into formation water was the principal mechanism by which the carbon dioxide had been removed from the gas phase over millions of years. The fourth mechanism is the most permanent. Dissolved carbon dioxide forms carbonic acid, which attacks minerals in the rock and releases cations such as calcium, magnesium and iron. Those cations combine with dissolved carbonate to precipitate new carbonate minerals such as calcite, magnesite and siderite. The carbon is then locked into solid rock, as secure as it can ever be in the Earth's crust. In most sandstone reservoirs, mineral trapping is extremely slow, because the dominant mineral, quartz, does not react with carbonic acid, and the reactive minerals present are scarce. Models typically suggest that mineralisation in sandstones takes thousands to tens of thousands of years to sequester a meaningful fraction of injected carbon dioxide. Basalts and other mafic and ultramafic rocks are different. They are rich in calcium, magnesium and iron silicates that react readily with carbonic acid. The Carbfix project in Iceland, begun in 2007 by Reykjavík Energy, the University of Iceland, the French National Centre for Scientific Research and Columbia University, set out to test how fast. Carbon dioxide was dissolved in water at the wellhead and injected into basalt at depths of 400 to 800 metres. In a 2016 paper in Science, Juerg Matter and colleagues reported that more than 95 percent of the injected carbon dioxide had mineralised within two years, as measured by carbon-14 tracer techniques. A parallel pilot near Wallula in Washington State, run by the Pacific Northwest National Laboratory, injected about 1,000 tonnes of supercritical carbon dioxide into the Columbia River basalts and found carbonate minerals in core samples recovered two years later. Basalt mineralisation has two great attractions. It reduces the long-term monitoring burden, because the carbon dioxide does not remain as a mobile fluid, and it opens storage options in places without thick sedimentary basins, including volcanic regions and ocean ridges. Its limits are the need for large water volumes when carbon dioxide is pre-dissolved, the uncertainty of injecting at large scale into fractured basalts whose permeability is hard to predict, and the possibility that precipitated minerals could clog the very fractures needed to keep injecting. What nature has tested, and security through time The strongest evidence that rock can hold carbon dioxide for geological time comes from places where nature has done the experiment. In several parts of the world, carbon dioxide released from deep magmatic sources or from the breakdown of carbonate rocks has accumulated in sedimentary traps, exactly as natural gas does. In the American Southwest, fields such as McElmo Dome in Colorado and Bravo Dome in New Mexico hold large accumulations of nearly pure carbon dioxide beneath shale and evaporite seals. These fields have held their carbon dioxide for very long periods; at Bravo Dome, studies of noble gases have suggested that the gas accumulated over geological timescales and has been dissolving slowly into the underlying brine ever since. Since the 1970s and 1980s, the oil industry has produced carbon dioxide from these fields and sent it by pipeline to the Permian Basin for enhanced oil recovery. The irony is instructive: the infrastructure now being repurposed for climate storage was built to take carbon dioxide that nature had safely stored and bring it back toward the surface. Natural analogues also show what failure looks like. At Crystal Geyser in Utah, carbon dioxide-charged water rises along a fault zone and erupts periodically at the surface from an abandoned exploration well. Geologists have studied the site and the surrounding area, where ancient travertine deposits record the escape of carbon dioxide along the same faults over hundreds of thousands of years. The lesson is consistent with everything engineers have learned from storage projects: leakage, when it happens, follows specific pathways, in this case faults that cut the seal and a well that pierced it. A carbon dioxide accumulation without such pathways stays put. One with them does not. Natural analogues have their limits as evidence. Natural accumulations formed slowly, at pressures that equilibrated with their surroundings over geological time, whereas engineered storage injects large volumes quickly and raises pressure in a way nature rarely does. The analogues show that seals and traps can hold carbon dioxide for millions of years; they do not show that a given seal will hold under the pressure pulse of industrial injection. That question is answered site by site. The four trapping mechanisms work together, and their combined effect is a security that grows with time. Immediately after injection, structural and stratigraphic trapping beneath a seal hold almost all the carbon dioxide, and storage depends on that seal and on the wells that penetrate it. Over decades, residual trapping immobilises a growing fraction. Over centuries and millennia, solubility trapping takes most of the remainder into the brine, and over still longer times, mineral trapping locks part of it into rock. The Intergovernmental Panel on Climate Change's 2005 Special Report on Carbon Dioxide Capture and Storage summarised the evidence then available by judging that, for appropriately selected and managed sites, the fraction retained was very likely to exceed 99 percent over 100 years and likely to exceed 99 percent over 1,000 years. Nearly two decades of further operating experience have not overturned that judgment, but they have sharpened its conditions. The phrase "appropriately selected and managed" carries the weight. The risks of geological storage are concentrated early, during and soon after injection, when pressure is highest and the plume is most mobile, and in a few specific pathways: poorly sealed wells, faults that cut the seal, and pressure high enough to fracture rock. Those risks can be reduced, but only by knowing the subsurface well. That knowledge does not come from general principles. It comes from mapping a specific piece of ground, drilling into it, and testing it. The next chapter describes how that work is done. Chapter 6: Mapping Pore Space A geologist asked whether carbon dioxide can be stored in a particular region will not answer with a number of tonnes. She will answer with a question: how much do you know? The storage capacity of a piece of ground is not a property that can be read off a map. It is an estimate that tightens, often downward, as each new kind of data arrives: a regional geological map, then old well logs, then seismic surveys, then a new well with cores and pressure tests, then the first months of injection. At each stage some prospective sites drop out, and the capacity of those that remain is revised. This chapter describes that progression, the tools used at each stage, the arithmetic of capacity, and the recent research that has sharply revised the global estimate of how much storage can prudently be used. A site that can store carbon dioxide safely and economically must satisfy three conditions, sometimes summarised as capacity, injectivity and containment. Capacity is the amount of carbon dioxide the formation can hold. It depends on the volume of rock, its porosity, how much of that porosity the carbon dioxide can actually reach, and the density of carbon dioxide at the formation's temperature and pressure. Injectivity is the rate at which carbon dioxide can be pushed into the formation through a well without exceeding safe pressures. It depends chiefly on permeability, the ease with which fluid flows through the rock, and on the thickness of the formation. A formation with vast capacity but low permeability is like a huge warehouse with a single narrow door: it may hold a great deal eventually, but not at the rate a large capture plant delivers. Containment is the assurance that the carbon dioxide will stay. It depends on the integrity and continuity of the seal, the presence and behaviour of faults, the number and condition of wells that penetrate the seal, and the pressure the formation and seal can withstand before fracturing. Each condition can rule out a site independently, and each requires different data. The tools of subsurface mapping Characterising a storage site uses methods developed over a century of oil and gas exploration. Table 3 summarises the principal tools, what each measures, and its main limitation; the descriptions reflect standard practice in petroleum geoscience and in storage-site characterisation guidance from bodies such as the United States Department of Energy. Table 3. Principal tools for characterising a storage site. Tool What it measures Coverage Main limitation Regional geological mapping Basin structure, rock types, depth ranges Hundreds of kilometres Coarse; cannot confirm local properties Seismic reflection surveys (2D and 3D) Geometry of layers, faults, traps Tens of kilometres, resolution of metres to tens of metres Images structure, not directly porosity or permeability Well logs Porosity, lithology, fluid content along the borehole A few metres around each well Sample only where wells exist Core samples Porosity, permeability, capillary entry pressure, mineralogy Centimetres Tiny fraction of the rock volume Injection and pressure tests Injectivity, formation pressure, fracture pressure Hundreds of metres to kilometres Require a drilled well; costly The workhorse is reflection seismology. A seismic survey sends sound waves into the ground, from vibrator trucks on land or air guns towed behind ships at sea, and records the echoes returning from boundaries between rock layers with different acoustic properties. Processed into a three-dimensional image, a seismic survey reveals the shape of reservoir and seal layers, the domes and fault blocks that form traps, and the faults that might provide escape routes. Modern 3D surveys can resolve layers a few metres to tens of metres thick, depending on depth and rock properties. Seismic data show structure, but they say little directly about porosity and permeability. For that, geologists need wells. When a well is drilled, instruments lowered into the borehole record logs of properties including natural radioactivity, which distinguishes shale from sandstone, electrical resistivity, which indicates the fluid in the pores, and the speed of sound and density of the rock, from which porosity is calculated. Cores, cylinders of rock cut and brought to the surface, allow laboratory measurement of permeability, capillary entry pressure of the seal, and mineral composition. The practical challenge is integrating these scales. A core sample is a few centimetres across; a seismic survey covers hundreds of square kilometres. Geologists combine them through a geological model, a three-dimensional numerical representation of the subsurface that honours the seismic geometry and the well data and fills the space between with statistically plausible estimates. Reservoir engineers then simulate injection within this model, predicting pressure build-up and plume migration. Each new well and each test is used to update the model. The uncertainty never vanishes, but it shrinks. Areas with extensive oil and gas history have a large advantage. In basins such as the North Sea, the Permian Basin of Texas, the Gulf Coast and the Alberta Basin, thousands of wells and decades of seismic data already exist, and much of the regional geology is known in detail. That data advantage is double-edged, however. The same wells that provide information are potential leakage pathways, a theme taken up in Chapter 7. The arithmetic of capacity The basic formula for estimating capacity in a saline formation is a product of five quantities: the area of the formation, its thickness, its porosity, the density of carbon dioxide at reservoir conditions, and an efficiency factor representing the fraction of pore volume that carbon dioxide can actually occupy. A worked example, with illustrative values, makes the reasoning concrete. Consider a saline sandstone formation covering 100 square kilometres, with a net thickness of 50 metres of reservoir-quality rock and an average porosity of 20 percent. Its bulk rock volume is 100 million square metres times 50 metres, or 5 billion cubic metres, and its pore volume is a fifth of that, 1 billion cubic metres. If carbon dioxide at reservoir conditions has a density of 700 kilograms per cubic metre and could fill every pore, the formation could hold 700 million tonnes. It cannot. The efficiency factor accounts for the fact that brine already fills the pores and must be displaced or compressed, that buoyancy makes the plume ride along the top of the formation and bypass much of the rock beneath, that heterogeneities divert flow, and that some residual brine always remains. Methodologies developed by the United States Department of Energy and published by Angela Goodman and colleagues in 2011 estimated efficiency factors for saline formations that typically fall in the low single-digit percentages, depending on rock type and the confidence level chosen. Taking 2 percent as an illustrative middle value, the capacity of the example formation falls to 14 million tonnes. That is enough to take the output of a 500,000-tonne-a-year capture plant for about 28 years. It is a respectable site. But the arithmetic exposes how sensitive capacity estimates are. Halve the net thickness because some of the formation proves to be shaly, and capacity halves. Change the efficiency factor from 2 to 1 percent because pressure limits injection, and it halves again. A site initially estimated at 14 million tonnes could, on closer inspection, hold 3.5 million. Pressure is often the real constraint. Injecting carbon dioxide into a closed or poorly connected formation raises its pressure, because the brine already there must be compressed or pushed aside. Every formation has a fracture pressure above which rock, and especially the seal, may crack. Regulations typically require injection pressure to stay below a safety margin under that limit. In a formation that is laterally confined by faults or by changes in rock type, pressure can approach the limit long before the pore volume is filled. In such cases, operators may need to produce brine from the formation to relieve pressure, a strategy that costs money and creates the problem of disposing of large volumes of salty water. The Gorgon project in Western Australia ran into exactly this constraint, as Chapter 7 describes. Injectivity imposes its own arithmetic. A single well in a thick, permeable sandstone can accept a great deal: the one injection well at Sleipner has taken about a million tonnes a year for most of its life, because the Utsira Formation is unusually thick and permeable. Formations with lower permeability may accept only a few hundred thousand tonnes a year per well, or less, without pressures near the wellbore rising toward the fracture limit. Gorgon was planned with nine injection wells; In Salah used three horizontal wells, drilled sideways through a thin reservoir layer precisely to expose more rock to each well. Because each deep well can cost tens of millions of dollars, especially offshore, the number of wells needed is a major component of storage cost, and a formation's permeability is as important economically as its capacity. Pressure also extends much farther than carbon dioxide. A plume of injected carbon dioxide may spread a few kilometres from the well over decades, but the pressure disturbance it causes travels through the connected brine far beyond that, sometimes tens of kilometres. That distinction matters for regulation, because the area within which old wells must be found and checked is defined by where pressure could push brine or carbon dioxide up an unsealed well, not merely by where the plume itself goes. It also matters between projects. Two storage sites injecting into the same formation tens of kilometres apart may interfere with each other's pressure budget, so basin-scale planning is needed once storage becomes common. In the North Sea and the Gulf Coast, where many projects are planned in the same formations, regulators and operators have begun to treat pore space as a shared resource whose pressure capacity must be allocated. Ownership of that pore space is a question of law rather than geology, and it is unsettled in many jurisdictions. In the United Kingdom and Norway, the state controls offshore storage and licenses it. In much of the United States, where landowners own subsurface rights, several states have legislated who owns the pore space, generally assigning it to the surface owner, and have created procedures, sometimes called unitisation or amalgamation, that allow a storage project to proceed when a large majority of affected owners agree, with compensation for the rest. Without such rules, a single holdout landowner above a plume could block a project, and the uncertainty alone can deter investment. Resource, capacity and prudence Storage estimates are often presented as a single global number, and those numbers have been very large. Industry and research bodies have produced figures for theoretical global storage resource in the range of thousands to over ten thousand gigatonnes of carbon dioxide. Since cumulative fossil emissions since the industrial revolution are on the order of two thousand gigatonnes, such figures appear to suggest that storage space is effectively unlimited. Geologists distinguish, however, between several classes of estimate. A theoretical resource includes all pore volume in suitable formations. An effective resource applies technical constraints such as efficiency factors and minimum depths. A practical capacity also applies economic, regulatory and social constraints, such as distance from sources, protected areas, population density and competing uses of the subsurface. A matched capacity, the most conservative class, includes only sites that have been specifically characterised and matched with a source of carbon dioxide. Each step down can cut the estimate by a large factor. In September 2025, a study published in Nature by Matthew Gidden of the International Institute for Applied Systems Analysis and colleagues, including Siddharth Joshi and Joeri Rogelj, applied a set of exclusion criteria to sedimentary basins around the world. They screened out areas of seismic hazard, places where storage could threaten groundwater, areas close to population centres, environmentally protected areas and the regions within the Arctic and Antarctic circles, formations too shallow or too deep for secure storage, offshore locations in water too deep to develop, and waters outside national jurisdiction or in disputed territory. The remainder they described as a prudent planetary limit for geological storage: about 1,460 gigatonnes of carbon dioxide, roughly an order of magnitude below the theoretical estimates of ten thousand gigatonnes or more that had become common. The authors calculated that if this entire prudent resource were used only for carbon dioxide removal, it would lower global temperatures by about 0.7 °C, far less than the multiple degrees sometimes implied by larger estimates. Several points about this study deserve emphasis. It does not claim that storage beyond its limit is physically impossible; it argues that relying on storage beyond that limit would require accepting risks or costs that society has not yet shown it is willing to bear. Its exclusion criteria are judgments, and other researchers may draw them differently. But its central conclusion has changed the conversation. Geological storage is a finite, intergenerational resource. Every tonne of fossil carbon dioxide stored to avoid emissions now uses pore space that could later hold carbon dioxide removed from the air. The study's authors suggested that storage should be prioritised for uses with no alternative, which in the long run includes removing historical emissions. From map to permit A site that passes characterisation must then be permitted. In the United States, carbon dioxide injection for storage requires a Class VI well permit under the Safe Drinking Water Act's Underground Injection Control program, introduced by the Environmental Protection Agency in 2010. The permit application must demonstrate the site's suitability, define an area of review based on predicted plume and pressure footprints, identify and address all wells within that area, specify monitoring plans, and provide financial assurance for corrective action, closure and emergency response. Post-injection site care defaults to 50 years of monitoring, though it can be shortened if the operator demonstrates that the plume has stabilised. Several states have taken over responsibility for issuing these permits, a status known as primacy. North Dakota and Wyoming were early, Louisiana followed in 2024, and Texas was granted primacy in November 2025, with permitting now handled by the Railroad Commission of Texas, the state's oil and gas regulator. Supporters argue that state regulators know local geology and move faster; critics worry that agencies traditionally close to the oil and gas industry may be less exacting. In Europe, the 2009 Directive on the geological storage of carbon dioxide sets a similar framework, requiring site characterisation, monitoring, and transfer of liability to the state after closure once the operator demonstrates that stored carbon dioxide will remain permanently contained. The European Union's Net-Zero Industry Act, adopted in 2024, set a target of at least 50 million tonnes a year of carbon dioxide injection capacity by 2030 and placed obligations on oil and gas producers to contribute. Mapping, modelling and permitting can make a site credible. They cannot make it certain. That requires watching what the carbon dioxide does once it is in the ground, and responding when it surprises. Chapter 7: Watching the Plume In August 1996, Statoil, now Equinor, began injecting carbon dioxide beneath the North Sea from the Sleipner gas platform. The company was not trying to fight climate change for its own sake. Natural gas from the Sleipner West field contained around 9 percent carbon dioxide, far above the 2.5 percent permitted for sale, so the carbon dioxide had to be removed before the gas could be marketed. Norway had introduced a tax on offshore carbon dioxide emissions in 1991. Venting the separated gas would have cost the company the tax on roughly a million tonnes a year; injecting it into a saline sandstone called the Utsira Formation, about a kilometre beneath the seabed, was cheaper. Sleipner has injected on the order of a million tonnes a year ever since, and it has become the most closely watched body of carbon dioxide in the world. This chapter is about watching. Geological storage is not a fire-and-forget operation. The models built during site characterisation are predictions, and predictions about rock kilometres down are always partly wrong. Monitoring is how operators find out where the carbon dioxide has actually gone, whether pressure is behaving as expected, and whether anything is leaking. The record of storage projects shows that monitoring works, that surprises are common, and that most surprises can be managed if they are detected early. It also shows how expensive and embarrassing the ones that are not managed well can become. Seeing through rock The most powerful tool for watching carbon dioxide underground is time-lapse seismic surveying, also called 4D seismic, in which a 3D survey is repeated at intervals over the same area. Carbon dioxide replacing brine in pores changes the speed at which sound travels through the rock and creates strong reflections. By subtracting an earlier survey from a later one, geophysicists can map where the carbon dioxide has spread. At Sleipner, the first repeat survey in 1999 revealed something the pre-injection models had not fully anticipated. The Utsira Formation, previously thought of as a fairly uniform sandstone, contains several thin shale layers, each only about a metre thick. The rising carbon dioxide had pooled beneath each of them, creating a stack of about nine thin, bright layers, like a pile of pancakes, before reaching the top of the formation beneath the main seal. The plume had climbed through the entire formation within about three years, finding small gaps in the intermediate shales. Later surveys tracked the topmost layer spreading laterally under the main caprock, with its shape controlled by small topographic features in the seal's underside, just as oil accumulates in subtle traps. What makes Sleipner reassuring is not that it behaved as originally predicted but that monitoring revealed what it was doing and the main seal has held. Repeated surveys have found no evidence of carbon dioxide above the caprock. Seismic methods are sensitive enough to detect accumulations on the order of thousands of tonnes in favourable conditions, which means a significant leak into overlying layers would very likely be visible. Gravity surveys on the seabed, which measure the tiny changes in the Earth's gravitational pull caused by replacing dense brine with lighter carbon dioxide, have provided an independent check on how much carbon dioxide is where and helped constrain its density, and with it how much has dissolved. Seismic imaging is complemented by monitoring in and around wells. Pressure and temperature gauges at the bottom of injection and observation wells show how the formation responds to injection, which is the most direct check on capacity and injectivity models. Fibre-optic cables cemented behind well casing can measure temperature and acoustic signals continuously along the whole well. Fluid samples from observation wells can detect carbon dioxide arrival and changes in water chemistry. Chemical tracers added to the injected stream allow any carbon dioxide found elsewhere to be fingerprinted. On land, satellites can help. Interferometric synthetic aperture radar, known as InSAR, compares radar images of the ground taken at different times to detect changes in surface elevation as small as a few millimetres. Injection raises reservoir pressure, which can lift the ground above it by a small amount. That signal, as one project discovered, can reveal more than anyone expected. Lessons from surprise Projects that ran into trouble have taught at least as much as those that did not. Table 4 summarises seven storage projects, chosen because they illustrate the range of experience, with their start dates, target formations and principal lessons. Figures are from the operators' public reports and the news coverage and scientific papers cited in the Notes. Table 4. Selected geological storage projects and what they taught. Project Start Storage formation Scale Principal lesson Sleipner, Norway 1996 Utsira saline sandstone, about 1 km deep About 1 Mt/yr 4D seismic can track a plume through intra-formation layers In Salah, Algeria 2004 Carboniferous sandstone, about 1.9 km About 3.8 Mt in total; suspended 2011 InSAR revealed pressure-driven fracturing into the lower seal Snøhvit, Norway 2008 Tubåen, then Stø formation About 0.7 Mt/yr Pressure rose faster than predicted; injection rerouted Quest, Canada 2015 Basal Cambrian Sands, over 2 km Over 1 Mt/yr design Extensive monitoring; performance near design Gorgon, Australia 2019 Dupuy Formation, about 2 km About 10 Mt by 2024 against a 4 Mt/yr design Pressure management limited injection Decatur, United States 2011 and 2017 Mount Simon Sandstone, about 2 km Pilot of 1 Mt, then commercial Corroded monitoring well let fluids into an unauthorised zone Northern Lights, Norway 2025 Johansen-area formation, 2.6 km below seabed 1.5 Mt/yr phase one Open-access shipping and storage network In Salah, in the Algerian Sahara, is the most instructive. Gas from the Krechba field contained excess carbon dioxide, which BP, Sonatrach and Equinor separated and injected through three horizontal wells into the water-bearing part of the same Carboniferous sandstone, about 1.9 kilometres deep. The reservoir layer was thin and had low permeability, so injection pressures were high. InSAR showed the ground above the injection wells rising by around 5 millimetres a year. Above one well, the uplift pattern formed two parallel lobes separated by a trough, which analysis interpreted as the signature of a vertical fracture or fault zone opening in the lower part of the seal under the injection pressure. Separately, carbon dioxide reached an old appraisal well about 1.3 kilometres from an injection well within a couple of years, apparently along a fracture pathway, and that well was then sealed. Injection was suspended in 2011 after about 3.8 million tonnes had been stored. No carbon dioxide is known to have reached the surface through the main seal, which included a very thick upper section, but the project showed how quickly pressure can find weaknesses and how valuable cheap surface monitoring can be. Snøhvit, in the Barents Sea, taught a related lesson about pressure. Carbon dioxide separated from gas produced for a liquefied natural gas plant was injected from 2008 into the Tubåen Formation. Pressure in the well rose much faster than predicted, indicating that the formation was more compartmentalised and less permeable than models had assumed. By 2011, operators had moved injection to the shallower Stø Formation, which had better connectivity. The project continued, but only because the operators were watching pressure closely and had a fallback. Gorgon, operated by Chevron on Barrow Island off Western Australia, is the largest storage project measured by design capacity, intended to inject around 4 million tonnes a year, at least 80 percent of the carbon dioxide separated from the gas processed for its liquefied natural gas trains. Injection did not begin until August 2019, more than three years after gas production started. Reporting since then has shown that injection fell well short of design; in the financial year to mid-2024 the project stored about 1.6 million tonnes, and around 10 million tonnes cumulatively over five years. The main cause has been reservoir pressure: the plan relied on producing water from the formation to relieve pressure, and the water-handling system did not perform as needed. Costs rose accordingly, and the project has had to buy offsets to cover the shortfall against its environmental conditions. At Decatur, Illinois, the agricultural processing company Archer Daniels Midland injected carbon dioxide from ethanol fermentation into the Mount Simon Sandstone, first in a pilot project led by the Illinois State Geological Survey that stored about a million tonnes between 2011 and 2014, and from 2017 under a commercial Class VI permit. In 2024, the company found that corrosion in one of its deep monitoring wells had allowed carbon dioxide and formation fluid to migrate into a formation where they were not permitted, at depths of about 5,000 feet and below. The Environmental Protection Agency issued a notice of violation in September 2024. ADM stated that no drinking water sources were threatened, and the injection well itself was unaffected. The incident illustrated two things at once: wells, including monitoring wells, are the weakest links in containment, and monitoring detected the problem. Wells, faults and pressure The record of storage projects points to three main pathways by which things go wrong. Wells are the most likely. A well is a hole drilled through the seal, sealed again with steel casing and cement. If that seal fails, it creates a direct path for fluids to move upward. Carbon dioxide dissolved in brine forms carbonic acid, which can attack both steel and the cement used in well construction. Field evidence, however, is more encouraging than laboratory worst cases. In a widely cited 2007 study, William Carey and colleagues recovered cement samples from a well in the SACROC oil field in West Texas that had been exposed to carbon dioxide for about 30 years of enhanced oil recovery operations; they found that the cement adjacent to the casing had been altered by carbonation but continued to act as a barrier to flow. The greater danger lies in old, poorly documented wells, of which the United States alone has millions, many drilled and plugged before modern standards. Any storage project in a mature petroleum basin must locate and evaluate the wells within its area of review, and some will need to be re-plugged. Faults are the second pathway. A fault that cuts through the seal may be sealed by clay smear or mineral cement, or it may be permeable. Faults may also slip if pore pressure rises enough to reduce the friction holding them locked, causing earthquakes. In 2012, Mark Zoback and Steven Gorelick argued in the Proceedings of the National Academy of Sciences that because the Earth's continental crust is often close to failure, large-scale storage would be likely to trigger earthquakes that could compromise seals. Ruben Juanes and colleagues replied that small earthquakes need not create leakage pathways through thick, ductile seals, and that careful site selection and pressure management could control the risk. Operating experience since has been broadly consistent with the more optimistic view, but monitoring for induced seismicity is now a standard requirement. At Decatur, microseismic monitoring recorded many small events, mostly in the Precambrian basement beneath the storage formation, none large enough to be felt or to compromise the seal. Pressure is the third, and it underlies the other two. Every project that has run into trouble did so in part because pressure rose faster or further than expected. Pressure, not pore volume, is often what limits storage, and pressure management through careful injection rates, multiple wells, or brine production is a central part of storage engineering rather than a refinement. Who answers for a tonne a thousand years from now Monitoring raises the question of how long it must continue and who will pay for it. Carbon dioxide removal is sold on the promise of permanence on timescales of centuries to millennia, but companies rarely last a century. Regulatory systems answer this in different ways. The European Directive allows liability to transfer to the state once the operator has shown that the stored carbon dioxide is behaving as predicted and will remain contained, normally not earlier than 20 years after closure, and requires the operator to contribute to the cost of post-closure monitoring. In the United States, Class VI rules require 50 years of post-injection monitoring by default and financial assurance, but federal law has no general mechanism for transferring long-term liability, and several states have created their own. For carbon removal credits, the accounting adds another layer. A credit buyer is paying for a tonne removed from the atmosphere and kept out. Permanence is usually guaranteed through monitoring requirements, insurance-like buffer pools of reserve credits, and contracts that oblige the seller to replace any reversed tonnes. The arithmetic of leakage shows why careful site selection matters more than any guarantee. A storage site that leaked 0.1 percent of its contents per year would retain only about 37 percent after a thousand years, which would make it close to useless as a climate measure. One that leaked 0.001 percent per year would retain about 99 percent. The difference between these rates is the difference between a poorly chosen site and a well-chosen one, and it is precisely the difference that the characterisation and monitoring described in this chapter and the previous one are designed to secure. Turning monitoring data into a tradeable claim requires a further layer of rules. A credit for carbon removal is only as good as the methodology behind it: how gross capture is measured, which emissions are subtracted, what proof of storage is required, and how reversals would be detected and replaced. Independent standards bodies have developed methodologies specifically for direct air capture with geological storage. In 2024, Climeworks announced that its Orca plant had become the first direct air capture facility certified under the Puro Standard, one of the main registries for durable removals. The International Organization for Standardization published ISO 27914 in 2017, setting out requirements for the geological storage component of carbon capture and storage. These frameworks differ in detail, but the best of them share a principle: a removal tonne is counted only when the carbon dioxide has been delivered into storage and verified, net of every emission in the chain, and only by a party that has a continuing obligation to make good any reversal. The weakest point in the chain is often not the rock but the record. Over centuries, companies dissolve, archives are lost, and wells drilled today become the abandoned wells of the future. Some regulators now require that well locations, construction records and monitoring data be lodged with public authorities, so that the knowledge needed to protect a storage site outlives the organisation that created it. That is the institutional analogue of a good seal. Northern Lights, the open-access storage project developed by Equinor, Shell and TotalEnergies, embodies the institutional response. Its first carbon dioxide was injected in August 2025, delivered by ship from the Heidelberg Materials cement plant at Brevik in Norway and stored about 2.6 kilometres beneath the seabed off western Norway. It operates as a service: industrial emitters across Europe contract for transport and storage, and the operator takes on the work of monitoring and reporting. The first phase has a capacity of 1.5 million tonnes a year, and an expansion to at least 5 million tonnes a year is planned from 2028. The model separates the business of capture from the business of storage and concentrates subsurface responsibility in organisations with the skills and balance sheets to bear it. Storage, then, works where it is done carefully, and the care is expensive. The final chapter asks what all of this, capture and storage together, costs, who pays, and what that implies for how large the enterprise can become. Chapter 8: Cost, Scale and the Honest Role of Air Capture The previous chapters have treated air capture and geological storage as engineering problems. This chapter treats them as an industry. An industry has costs, customers, subsidies, investors and competitors, and its size is set as much by those as by physics. The question is not whether a tonne of carbon dioxide can be taken from the air and stored for a thousand years. It can. The question is at what price, how many tonnes, powered by what, paid for by whom, and in place of what. The state of the industry in late 2026 It helps to begin with what actually exists. Table 5 lists the largest direct air capture facilities that have operated or are near operation, with their nameplate capacities and what has been publicly reported about their performance. The figures come from company announcements and the reporting cited in the Notes. Table 5. Largest direct air capture facilities, nameplate versus reported status. Facility Developer and technology Location Nameplate capacity Reported status, late 2026 Orca Climeworks, solid amine Iceland 4,000 t/yr Operating since 2021, well below nameplate Mammoth Climeworks, solid amine Iceland 36,000 t/yr Operating since 2024; 675 t removed in first half of 2026 Tracy Heirloom, mineral looping California Up to 1,000 t/yr Operating since 2023 Stratos 1PointFive, hydroxide solvent Texas 500,000 t/yr In commissioning; start-up expected end of 2026 Project Cypress Battelle, Climeworks, Heirloom Louisiana 1 Mt/yr target by 2030 Planned; federal support reviewed in 2025–26 Two facts stand out. The first is how small the operating capacity remains. If every plant in the world ran at nameplate, total direct air capture would still be measured in tens of thousands of tonnes a year until Stratos starts, against fossil emissions of about 37 billion tonnes. The independent State of Carbon Dioxide Removal assessments, led by Stephen Smith at the University of Oxford, have estimated that conventional land-based methods such as afforestation remove about two billion tonnes a year, while all novel methods together, including bioenergy with carbon capture, biochar and direct air capture, remove only on the order of one to two million tonnes. Direct air capture was a small share of that small share. The second is the gap between nameplate and reality. Climeworks' Mammoth plant, designed for 36,000 tonnes a year, reported 675 tonnes removed in the first half of 2026, an annualised rate under 4 percent of nameplate, and the company presented that as a sixfold improvement on the same period of 2025. Stratos was originally expected to start in 2024 or 2025 and has been delayed several times. None of this is unusual for first-of-a-kind industrial plants, which routinely take years to reach design performance. But it means that public discussion of costs and capacities has been running on projections, not measurements. What a tonne costs and who pays The cost of a tonne of carbon dioxide removed by direct air capture and storage can be divided into five parts: the capital cost of the plant, amortised over its life; energy; capture materials, including replacement of degraded sorbent; operation and maintenance; and transport and storage. Capital cost is the largest and least certain. A first-of-a-kind plant is costly because it is custom-built, because its designers lack experience, and because lenders charge more for unproven technology. Stratos, with a design capacity of 500,000 tonnes a year, was reported to cost over a billion dollars; Occidental has not published a precise final figure. If a plant costs 1.3 billion dollars, is financed at a cost of capital of around 8 percent over 25 years, and runs at its full design capacity, the annual capital charge is roughly 120 million dollars, or about 240 dollars per tonne. If it runs at half capacity, the charge doubles. Capital cost per tonne is therefore not only a function of what the plant costs but of how reliably it runs, which is why the operational problems described above matter so much financially. Energy costs follow from the thermodynamics of Chapter 4. A plant using a megawatt-hour of electricity per tonne at 50 dollars per megawatt-hour spends 50 dollars per tonne on electricity. A plant using 5 gigajoules of natural gas at 3 dollars per gigajoule spends 15 dollars on gas, plus the cost of its electricity and its oxygen. These are significant, but in current plants they are smaller than the capital charge. Transport and storage, in favourable locations with short pipelines and well-characterised sites, are generally estimated at on the order of ten to a few tens of dollars per tonne. In less favourable locations, or where carbon dioxide must be shipped, costs rise. Adding these up gives costs well above the figures in early optimistic studies. Climeworks itself has set a target for 2030 of 250 to 350 dollars per tonne captured and 400 to 600 dollars per tonne of net removal, the difference reflecting the costs of transport, storage, certification and the emissions that must be subtracted. Credits for direct air capture removal have commonly sold for prices in the high hundreds of dollars per tonne or more. The figure of 100 dollars per tonne, long promoted as the threshold at which air capture would become a mass-market climate solution, remains a target with no plant yet near it. Nobody needs carbon dioxide removed from the air in the way people need electricity or steel. Demand for air capture exists only because governments or buyers create it. Three sources of demand matter. The first is public subsidy. In the United States, the Section 45Q tax credit pays 180 dollars per tonne of carbon dioxide captured by direct air capture and stored in dedicated geological storage, and 85 dollars per tonne for carbon dioxide captured from industrial sources and stored. The 2025 budget reconciliation law, known as the One Big Beautiful Bill Act, preserved those values while eliminating many other clean-energy credits, and it raised the credit for carbon dioxide used in enhanced oil recovery or in products to parity with the storage credit. Projects must begin construction before 2033 to qualify. Separately, the 2021 Bipartisan Infrastructure Law allocated 3.5 billion dollars to regional direct air capture hubs. In 2025 the Department of Energy reviewed and cancelled many clean-energy awards, and the two flagship hubs, Project Cypress in Louisiana and the South Texas hub that includes Stratos, appeared on lists of projects slated for cancellation. Reporting in the spring of 2026 indicated that both had survived the review, but the episode demonstrated how exposed air capture is to political change. The second is voluntary corporate purchasing. Technology companies have been the most important buyers of carbon removal, motivated by net-zero commitments and, increasingly, by the emissions growth that has come with energy-hungry data centres. In 2022, Stripe, Alphabet, Shopify, Meta and McKinsey launched Frontier, an advance market commitment of 925 million dollars to buy permanent carbon removal between 2022 and 2030, with the explicit aim of helping early technologies reach scale. Microsoft has become by far the largest buyer of carbon removal of all kinds. Stratos has signed agreements with buyers including Amazon, Airbus, AT&T and Microsoft. These purchases matter because they create a price signal beyond subsidy, but their size is small relative to the scale of the climate problem and they depend on corporate strategies that can change. The third is compliance markets. Most emissions trading systems, including the European Union's, do not yet recognise permanent carbon removals for compliance, though the European Union has adopted a framework for certifying them and is considering how to integrate them. If removals were admitted into a compliance market with a high carbon price, the demand picture would change. Until then, air capture relies on subsidies and voluntary buyers. Oil, learning and the limits of optimism Any honest account of the field must address its relationship with the oil and gas industry. The largest air capture project in the world is owned by Occidental, one of the largest oil producers in the Permian Basin. Occidental has presented air capture as a way to produce what it calls net-zero oil: carbon dioxide captured from the air would be injected into oil fields to boost production through enhanced oil recovery, and the stored carbon dioxide would offset the emissions from burning the extra oil. Stratos itself is planned to inject into dedicated saline storage, but the logic of combining air capture with enhanced recovery remains part of the company's strategy, and the 2025 change to 45Q, which gave enhanced oil recovery the same credit as dedicated storage, makes that combination more attractive. The arguments on both sides are real. Enhanced oil recovery does leave a large fraction of the injected carbon dioxide underground, and oil fields are well-characterised, well-monitored places to store it. The oil industry brings subsurface skills, drilling capacity, and capital that no other sector has in comparable measure. On the other hand, carbon dioxide used to produce more oil is not removal in any net sense if the oil is burned; at best, it reduces the carbon intensity of a barrel. And an industry whose business is selling fossil fuels has an obvious interest in technologies that might justify continuing to sell them. The responsible position is neither to reject the industry's involvement nor to accept its framing. It is to insist on accounting that distinguishes removal from offsetting, and to judge projects by the net tonnes they keep out of the atmosphere. The hope behind public support for air capture is that costs will fall with deployment, as they did for solar panels, wind turbines and lithium-ion batteries. Those technologies followed learning curves: each doubling of cumulative production cut costs by a roughly constant percentage. Some analyses have applied similar learning rates to air capture and projected costs falling to 100 dollars per tonne or below by mid-century. There are reasons for caution. Solar panels and batteries are mass-produced consumer goods, manufactured in billions of identical units on automated lines. Air capture plants are closer to chemical plants: large, site-specific, and built in small numbers. Technologies of that type have historically shown slower learning. Modular solid-sorbent designs might learn faster, because their collectors can be mass-produced, but their thermodynamic floor and their reliance on manufactured sorbents limit how far costs can fall. The lime loop, built from mature components, has less room for dramatic learning because its components are already mature. Neither family can learn its way below the energy cost of release, which is why the thermodynamics of Chapter 4 is ultimately an economic argument. It is worth sketching, in round numbers, what a billion tonnes a year of air capture with storage would physically involve, because the sketch disciplines both enthusiasm and dismissal. At the scale of Stratos, 500,000 tonnes a year, it would take 2,000 plants, about 40 a year for half a century. At the illustrative capital cost used above, about 1.3 billion dollars per plant, the bill would be around 2.6 trillion dollars before any learning; even if learning cut capital costs by two-thirds, it would approach a trillion. Running the plants at the roughly one megawatt-hour per tonne of a heat-pump-assisted solid-sorbent design would need about 1,000 terawatt-hours of clean electricity a year, roughly the annual output of Japan's entire power system. Storing the carbon dioxide would require, at an average of half a million tonnes a year per well, around 2,000 injection wells, plus monitoring wells, plus pipelines connecting plants to storage sites. Over 50 years, the stored total would reach 50 gigatonnes, a meaningful share of the 1,460-gigatonne prudent storage limit described in Chapter 6. None of these numbers is impossible. The global oil and gas industry drills tens of thousands of wells a year and spends several hundred billion dollars a year on new supply. The world added hundreds of gigawatts of solar capacity in 2024 alone. But each number describes a project of industrial scale comparable to building a major new energy sector, and it would remove less than 3 percent of today's fossil emissions. That ratio is the strongest single argument for treating air capture as a complement to emissions cuts rather than an alternative. The honest role Where does this leave air capture among the climate tools available? Three comparisons help. Against cutting emissions, air capture almost always loses. A tonne of carbon dioxide not emitted, by replacing a coal plant with wind or solar, insulating a building, or electrifying a car, typically costs far less than 500 dollars to avoid, and often costs nothing or saves money. Every tonne removed by air capture at hundreds of dollars could, in most cases today, be avoided more cheaply. This is not a reason to abandon air capture; it is a reason not to let it substitute for emissions cuts. The danger that the promise of future removal will weaken the resolve to cut emissions now, which social scientists call mitigation deterrence, is real and has been documented in debates over climate targets. Against other forms of carbon removal, air capture has distinctive strengths. Forests and soils can store carbon cheaply but impermanently, at risk from fire, drought and land-use change, and with uncertain measurement. Bioenergy with carbon capture uses large areas of land and competes with food production. Enhanced rock weathering spreads crushed rock on fields, which is promising but hard to measure. Air capture with geological storage uses little land, produces a metered stream of carbon dioxide that can be measured precisely, and stores it in a form that can last millennia. Those properties make it the closest thing to an auditable, permanent removal, and that is why buyers pay a premium for it. Against the scale of need, air capture is a long-term option, not a near-term rescue. The Intergovernmental Panel on Climate Change's Sixth Assessment Report found that pathways limiting warming to 1.5 °C included substantial carbon removal, with cumulative direct air capture with storage ranging from zero to about 310 gigatonnes by 2100 across the assessed scenarios, alongside larger quantities of bioenergy with carbon capture and land-based removal. Those removals serve two roles: offsetting residual emissions from sectors such as aviation, agriculture and some industrial processes that are very hard to eliminate, and, later in the century, drawing down historical emissions. Neither role requires air capture to compete with cutting emissions today. Both require it to be available, proven and affordable by the time it is needed. That framing, air capture as a tool whose time is decades away and whose foundations must be built now, is the honest one. It justifies continued public and private investment in research, demonstration and the characterisation of storage sites. It does not justify claims that air capture can solve the climate problem, or that emissions can continue on the assumption that they will later be removed. The Conclusion draws out what follows from that position. Conclusion The molecule followed through this book travelled from the open air, through a fan and a filter or a film of caustic liquid, through a kiln or a heated vacuum chamber, into a compressor, down a well and into the pores of a sandstone or the fractures of a basalt. At each stage, it encountered a physical fact that no amount of investment or optimism can change. It was dilute, so two thousand tonnes of air had to be moved for every tonne of it. It was held by a bond chosen to be strong, so energy had to be spent to break that bond. It was buoyant underground, so it needed a seal, and the seal had to be found, mapped, tested and watched. Those facts do not make direct air capture and geological storage impossible. They make it a particular kind of thing, and the argument of this book has been that the policy, the investment and the public debate should be shaped by what kind of thing it is. Four conclusions follow. Pay for removal, not for capture The first is that the unit that matters is the net tonne removed and durably stored, and nothing else. Throughout this book, the same confusion has recurred: nameplate capacity reported as performance, captured carbon dioxide reported as removed carbon dioxide, carbon dioxide used to produce oil reported as stored. Each of these confusions inflates the apparent contribution of the technology and, worse, misdirects money toward the wrong goals. Public subsidies and private purchases should therefore be tied to net removal measured at the point of storage, after subtracting the emissions from energy, materials, construction and transport. That sounds like a technicality. It is the heart of the matter. A subsidy that pays per tonne captured rewards a plant for burning gas to capture gas. A subsidy that pays per tonne of net removal stored in dedicated formations rewards exactly what the climate needs. The 2025 change in the United States that gave carbon dioxide used for enhanced oil recovery the same credit as carbon dioxide stored permanently moved policy in the wrong direction by this test, whatever its other merits. Buyers in the voluntary market, who are not bound by tax law, can and should apply the stricter standard themselves. Treat energy as the binding constraint The second conclusion is that the energy of release, not the chemistry of capture, sets the limit on what air capture can do, and that energy policy must therefore come first. A tonne of carbon dioxide removed by a plant running on fossil electricity is partly or wholly cancelled at the power station. A megawatt-hour of clean electricity directed to air capture removes less carbon dioxide than the same megawatt-hour would avoid by displacing coal or gas generation, for as long as there is coal or gas generation to displace. The practical implication is that air capture should be sited where its energy is genuinely additional, close to stranded geothermal heat, curtailed renewable power, nuclear plants with spare low-grade heat, or industrial waste heat, and that its large-scale growth should follow, not precede, the decarbonisation of the grids it draws on. Developers who claim removals from plants on carbon-intensive grids should be required to show their accounting in full. Governments deciding how to spend scarce clean-energy budgets should, for the next decade or two, favour displacing fossil generation, while supporting enough air capture to drive learning and keep the option alive. The research priority follows as well. The most valuable advances will be those that cut the energy of release or allow it to be supplied with low-value heat: sorbents with higher working capacity and longer lives, release processes that waste less sensible heat, ways to keep water out of the release step, and electrically driven processes that avoid heat altogether. Advances in contactor design and modular manufacturing matter too, but they cannot outrun the thermodynamics. Map the pore space before it is needed The third conclusion is that geological storage is a finite resource and should be managed as one. For decades, storage was described as effectively unlimited, and the arithmetic of cumulative emissions against theoretical capacity seemed to support that. The more careful estimates now available, such as the prudent limit of about 1,460 gigatonnes proposed in 2025, suggest that safely usable storage is an order of magnitude smaller than older headline figures, and that every tonne stored is drawn from a shared, intergenerational account. Two things follow. First, storage capacity should be characterised, publicly and early, well before it is needed. Mapping pore space requires seismic surveys, appraisal wells and injection tests that take years and cost money that no single project wants to spend. That is a public good, comparable to geological surveys of minerals or groundwater, and public agencies are well placed to lead it. The countries and regions that know their subsurface will be able to permit storage quickly and safely when demand arrives; those that do not will be forced to choose between delay and risk. Second, the allocation of storage between uses should be a deliberate choice. Storing carbon dioxide captured at cement plants, whose process emissions have few alternatives, is a sound use. Storing carbon dioxide removed from the air to offset aviation or agriculture is a sound use. Storing carbon dioxide captured from fossil power stations that could be replaced by renewables may be a poorer use of a limited resource over the long run. Basin-scale planning, pressure-budget allocation and pore-space licensing will be as important to the future of storage as any technical advance. Build the option without spending it The fourth conclusion concerns time. Almost every credible pathway to stabilising the climate at well below 2 °C involves removing carbon dioxide from the atmosphere at a scale of billions of tonnes a year by the second half of the century, both to balance emissions that cannot be eliminated and, eventually, to reduce the stock of carbon dioxide already in the air. Direct air capture with geological storage is one of the few methods that could provide that removal with high permanence and precise measurement. It will not be ready at that scale unless it is built, tested and improved now, at a scale small enough to be affordable and large enough to teach. That is the case for continued public support, for advance purchase commitments by corporate buyers, and for patient investment in demonstration plants that may underperform at first. The early record of plants such as Orca, Mammoth and Stratos, with their delays and shortfalls, is not evidence that the technology has failed. It is evidence of what building a first-of-a-kind industry looks like, and of why claims about its near-term contribution should be discounted. The same reasoning sets a firm limit on how the option should be used. It should not be spent in advance. Climate targets that assume large future removals in order to permit higher emissions now are borrowing against an asset that does not yet exist, at a scale it may never reach, using storage that is finite. The responsible position is to cut emissions as fast as possible, to build removal capacity in parallel, and to reserve that capacity for the emissions that genuinely cannot be avoided and the historical excess that must eventually be drawn down. The steel boxes on the Icelandic lava field, and the long walls of fans rising in West Texas, are the beginnings of an industry that may one day matter a great deal. Whether it does will depend less on the cleverness of its chemistry than on whether its builders, buyers and regulators respect the two things that chemistry cannot change: the energy it takes to let carbon dioxide go, and the rock that must hold it afterwards. Notes The sources below are listed by chapter, in the order in which the material they support appears. Company figures for plant capacity and performance are as reported by the companies or by the press at the dates given. Introduction and Chapter 1 Climeworks' Mammoth plant, Iceland: nameplate capacity of up to 36,000 tonnes a year; operations began in 2024; 675 tonnes removed in the first half of 2026, described as a sixfold increase on the prior-year period. Reported by Bloomberg and syndicated in Energy Connects, September 2026. Global fossil carbon dioxide emissions of about 37 billion tonnes a year: Global Carbon Project, Global Carbon Budget 2024, Earth System Science Data (2025). American Physical Society, Direct Air Capture of CO2 with Chemicals: A Technology Assessment for the APS Panel on Public Affairs (2011). National Academies of Sciences, Engineering, and Medicine, Negative Emissions Technologies and Reliable Sequestration: A Research Agenda (Washington, DC: National Academies Press, 2019). K. S. Lackner, H.-J. Ziock and P. Grimes, "Carbon dioxide extraction from air: is it an option?", Proceedings of the 24th International Conference on Coal Utilization and Fuel Systems (1999). S. Deutz and A. Bardow, "Life-cycle assessment of an industrial direct air capture process based on temperature–vacuum swing adsorption", Nature Energy 6 (2021): 203–213. Climeworks, "Next-gen tech powers Climeworks' megaton leap", press release, June 2024, describing third-generation collector dimensions, capacity and energy claims, and 2030 cost targets. Chapter 2 D. W. Keith, G. Holmes, D. St. Angelo and K. Heidel, "A process for capturing CO2 from the atmosphere", Joule 2 (2018): 1573–1594. Source of the energy inputs (8.81 GJ natural gas, or 5.25 GJ gas plus 366 kWh electricity, per tonne) and the levelised cost range of 94–232 dollars per tonne. Occidental Petroleum's acquisition of Carbon Engineering for about 1.1 billion dollars: Oil & Gas Journal, August 2023. Stratos Class VI permits: Occidental and 1PointFive, "Occidental and 1PointFive secure Class VI permits for STRATOS direct air capture facility", April 2025. Stratos start-up timing and commissioning issue: RBN Energy, "Stratos DAC project target Q2 2026 start-up"; Decarbonfuse, "Stratos DAC delay: inside the holdup at the world's largest carbon capture plant" (2026); E&E News, "World's largest carbon removal plant to open at the end of 2026", 7 August 2026. Oxygen supply for Stratos by Matheson Tri-Gas: Gasworld, 2024. Heirloom's Tracy, California facility and Louisiana plans: Heirloom, "Heirloom to build two direct air capture facilities in Northwest Louisiana", 2024. Chapter 3 T. M. McDonald and others, "Cooperative insertion of CO2 in diamine-appended metal-organic frameworks", Nature 519 (2015): 303–308. T. Wang, K. S. Lackner and A. Wright, "Moisture swing sorbent for carbon dioxide capture from ambient air", Environmental Science & Technology 45 (2011): 6670–6675. S. Voskian and T. A. Hatton, "Faradaic electro-swing reactive adsorption for CO2 capture", Energy & Environmental Science 12 (2019): 3530–3547. E. S. Sanz-Pérez, C. R. Murdock, S. A. Didas and C. W. Jones, "Direct capture of CO2 from ambient air", Chemical Reviews 116 (2016): 11840–11876. A comprehensive review of sorbent chemistry, including amine mechanisms and degradation. Chapter 4 K. Z. House and others, "Economic and energetic analysis of capturing CO2 from ambient air", Proceedings of the National Academy of Sciences 108 (2011): 20428–20433. M. Fasihi, O. Efimova and C. Breyer, "Techno-economic assessment of CO2 direct air capture plants", Journal of Cleaner Production 224 (2019): 957–980. C. Beuttler, L. Charles and J. Wurzbacher, "The role of direct air capture in mitigation of anthropogenic greenhouse gas emissions", Frontiers in Climate 1 (2019): 10. Global electricity generation of a little over 30,000 terawatt-hours: Ember, Global Electricity Review 2025. Chapter 5 Intergovernmental Panel on Climate Change, Special Report on Carbon Dioxide Capture and Storage (Cambridge: Cambridge University Press, 2005), including the judgment on retention fractions over 100 and 1,000 years. S. Krevor and others, "Capillary trapping for geologic carbon dioxide storage: from pore scale physics to field scale implications", International Journal of Greenhouse Gas Control 40 (2015): 221–237. S. M. V. Gilfillan and others, "Solubility trapping in formation water as dominant CO2 sink in natural gas fields", Nature 458 (2009): 614–618. J. M. Matter and others, "Rapid carbon mineralization for permanent disposal of anthropogenic carbon dioxide emissions", Science 352 (2016): 1312–1314. B. P. McGrail and others, "Field validation of supercritical CO2 reactivity with basalts", Environmental Science & Technology Letters 4 (2017): 6–10. Chapter 6 A. Goodman and others, "U.S. DOE methodology for the development of geologic storage potential for carbon dioxide at the national and regional scale", International Journal of Greenhouse Gas Control 5 (2011): 952–965. M. J. Gidden and others, "A prudent planetary limit for geologic carbon storage", Nature (published 3 September 2025); summary and figures from the International Institute for Applied Systems Analysis press release of the same month and a guest post by the authors in Carbon Brief. Texas Class VI primacy: Railroad Commission of Texas, "Texas granted primacy over Class VI wells", November 2025. European Union, Directive 2009/31/EC on the geological storage of carbon dioxide; Regulation (EU) 2024/1735, the Net-Zero Industry Act, setting the 50-million-tonne injection capacity objective for 2030. Chapter 7 R. A. Chadwick and others, "Quantitative analysis of time-lapse seismic monitoring data at the Sleipner CO2 storage operation", The Leading Edge 29 (2010): 170–177. A. Rinaldi and J. Rutqvist, "Modeling of deep fracture zone opening and transient ground surface uplift at KB-502 CO2 injection well, In Salah, Algeria", International Journal of Greenhouse Gas Control 12 (2013): 155–167. Gorgon performance figures: RenewEconomy, "Expensive failure: flagship Gorgon CCS collects less CO2 in worst year", 2 December 2024. Decatur monitoring well incident: E&E News, "First US CO2 injection well violates permit — EPA", September 2024. J. W. Carey and others, "Analysis and performance of oil well cement with 30 years of CO2 exposure from the SACROC Unit, West Texas, USA", International Journal of Greenhouse Gas Control 1 (2007): 75–85. M. D. Zoback and S. M. Gorelick, "Earthquake triggering and large-scale geologic storage of carbon dioxide", Proceedings of the National Academy of Sciences 109 (2012): 10164–10168; R. Juanes, B. H. Hager and H. J. Herzog, "No geologic evidence that seismicity causes fault leakage that would render large-scale carbon capture and storage unsuccessful", Proceedings of the National Academy of Sciences 109 (2012): E3623. Northern Lights first injection, August 2025: Oil & Gas Journal, "Northern Lights achieves first injection, storage of CO2 volumes in North Sea"; Equinor project pages. Chapter 8 Section 45Q changes under the One Big Beautiful Bill Act: Global CCS Institute, "U.S. preserves and increases 45Q credit in One Big Beautiful Bill Act", July 2025. Direct air capture hub funding review and restoration: MIT Technology Review, "The US is set to cancel funding for two major direct air capture plants", 7 October 2025; Decarbonfuse, "DOE restores 1.2B for DAC hubs" (2026). S. M. Smith and others, The State of Carbon Dioxide Removal (successive editions, 2023 onward), stateofcdr.org. Intergovernmental Panel on Climate Change, Climate Change 2022: Mitigation of Climate Change, Working Group III contribution to the Sixth Assessment Report, including ranges for cumulative direct air capture with storage in 1.5 °C pathways. Frontier advance market commitment: Frontier Climate, launch announcement, April 2022. Glossary Calcination. Heating calcium carbonate (limestone) to about 900 °C so that it decomposes into calcium oxide and carbon dioxide. The energy-intensive release step of liquid-solvent and mineral-looping air capture. Capillary entry pressure. The pressure a non-wetting fluid such as carbon dioxide must exceed to push into the narrow pores of a rock filled with water. High entry pressure is what makes shale an effective seal. Carbon dioxide removal. Taking carbon dioxide out of the atmosphere and storing it durably, counted net of all emissions caused by the process. Distinct from capture at an industrial source, which prevents new emissions. Class VI well. The category of injection well defined by the United States Environmental Protection Agency for storing carbon dioxide underground, with its own permitting, monitoring and financial assurance rules. Contactor. The part of an air capture plant where air meets the capture chemical, whether a wetted packing or a sorbent filter. Exergy. The part of a quantity of energy that could in principle be converted into work. High-temperature heat carries more exergy than the same amount of low-temperature heat. Injectivity. The rate at which fluid can be injected into a formation through a well without exceeding safe pressure, governed mainly by permeability and thickness. Permeability. A measure of how easily fluid flows through rock. Distinct from porosity, which measures how much open space the rock contains. Residual trapping. Immobilisation of carbon dioxide as isolated droplets held in pores by capillary forces after brine flows back behind a moving plume. Sorbent. A material that captures a gas by binding it on its surface or within its structure. In air capture, usually an amine attached to a porous solid. Supercritical fluid. The state of carbon dioxide above 31.1 °C and 7.38 megapascals, with liquid-like density and gas-like flow. Reached at depths below about 800 metres in most basins. Working capacity. The amount of carbon dioxide a sorbent actually takes up and releases in each practical cycle, as opposed to its maximum capacity under ideal conditions. Further Reading Intergovernmental Panel on Climate Change, Special Report on Carbon Dioxide Capture and Storage (2005).** Still the best single overview of the physical basis of geological storage, trapping mechanisms and risk. Some numbers are dated; the science is not. National Academies of Sciences, Engineering, and Medicine, Negative Emissions Technologies and Reliable Sequestration: A Research Agenda (2019).** A sober, thorough comparison of removal methods, with detailed chapters on direct air capture and geological storage and an explicit research programme. American Physical Society, Direct Air Capture of CO2 with Chemicals (2011).** The sceptical assessment that framed a decade of debate. Read it alongside the Keith study below to see how the argument about cost moved. D. W. Keith and others, "A process for capturing CO2 from the atmosphere", Joule (2018).** The most detailed public engineering description of a liquid-solvent plant, with mass and energy balances that reward careful reading. K. Z. House and others, "Economic and energetic analysis of capturing CO2 from ambient air", PNAS (2011).** The classic argument that dilute separations are inherently costly, with the thermodynamic reasoning set out clearly. E. S. Sanz-Pérez and others, "Direct capture of CO2 from ambient air", Chemical Reviews (2016).** A comprehensive scientific review of sorbent chemistry for readers who want the mechanisms in depth. M. J. Gidden and others, "A prudent planetary limit for geologic carbon storage", Nature (2025).** The study that reframed storage as a finite resource; its methods and maps are worth examining directly. S. M. Smith and others, The State of Carbon Dioxide Removal (stateofcdr.org, successive editions).** Independent, regularly updated data on how much removal is actually happening, by method, and on the gap to climate goals. Global CCS Institute, Global Status of CCS (annual).** An industry-oriented but useful annual census of capture and storage projects worldwide, with policy summaries by country. Hashtags: #CarbonCaptureTechnologies #CarbonCapture #DirectAirCapture #DAC #GeologicalSequestration #GeologicalCarbonStorage #CarbonDioxideRemoval #CDR #CarbonCaptureAndStorage #CCS #NegativeEmissions #ClimateChangeMitigation #ClimateTechnology #Decarbonization #NetZeroEmissions #CarbonNeutrality #SolidSorbents #LiquidSolvents #CarbonCaptureChemistry #CO2Adsorption #SorbentRegeneration #CarbonMineralization #MineralCarbonation #BasaltMineralization #SupercriticalCO2 #SalineAquifers #SubsurfaceStorage #CarbonStorageMonitoring #CarbonRemovalVerification #SustainableEnergy

Download the Book (PDF): Introduction: The Model Is Not the Product Most machine learning projects that fail do not fail in the notebook. They fail in the months after a model has been declared finished, when the accuracy figure that justified the project has quietly stopped describing anything real. A fraud model trained on last year's transactions meets a new payment method and begins waving through a pattern it has never seen. A demand forecast built before a competitor opened across the road keeps ordering stock for customers who have left. A recommendation system learns from the clicks it caused and narrows, week by week, into a loop of its own making. None of these failures throws an exception. The service stays up, the latency graphs stay green, and the predictions keep arriving, each one formatted perfectly and wrong in a way nobody has arranged to notice. This is the central fact of machine learning in production, and the reason a discipline called MLOps exists at all. A conventional piece of software encodes its behaviour in code. If the code does not change, the behaviour does not change, and a test that passed yesterday will pass today. A machine learning model encodes its behaviour in a combination of code and data, and the data that matters most is the data it has not yet seen. The model's correctness depends on an assumption — that tomorrow's inputs will resemble yesterday's training set — that the world is under no obligation to honour. Every deployed model is therefore a bet with an expiry date, and the expiry date is not printed on the label. Two cultures, one system The organisational difficulty follows from the technical one. Data scientists are trained to work in a loop of exploration: load data, try an idea, measure, try another. Their tools — notebooks, interactive sessions, ad hoc scripts — are built for speed of iteration, and their natural unit of output is a result: a metric, a plot, a model file that scored well on a held-out set. Software engineers are trained to work in a loop of delivery: specify, build, test, deploy, operate. Their tools — version control, continuous integration, containers, orchestration — are built for repeatability, and their natural unit of output is a running service that other systems can depend on. Neither culture is wrong, and neither is sufficient. The data scientist's model file is not a service: it has no defined interface, no record of the exact data and code that produced it, no tests that will catch a broken input, and no plan for what happens when it decays. The engineer's deployment pipeline, applied naively to a model, will happily ship a model file with every unit test passing and no way of knowing whether the model is any good. Organisations that let the two cultures meet only at a handoff — "here is the pickle file, please put it into production" — discover that the handoff is where the project dies. The research on this is consistent. When Paleyes, Urma and Lawrence surveyed published case studies of machine learning deployment in 2022, they found the hard problems clustered not in model design but in data management, integration, monitoring and the organisational seams between teams.1 An interview study of practising ML engineers by Shankar and colleagues the same year found the same thing from the inside: the work that consumed people was keeping models healthy after launch, not building them in the first place.2 MLOps is the set of practices, tools and organisational habits that close that seam. It borrows heavily from DevOps — automation, version control for everything, small reversible changes, monitoring as a first-class concern — but it cannot simply copy it, because a machine learning system has two additional moving parts that ordinary software lacks. One is data, which changes on its own schedule and must be versioned, validated and tracked as carefully as code. The other is the model itself, which is not written but trained, and which must therefore be reproducible from its inputs, evaluated statistically rather than with pass/fail assertions, and replaced on a cadence set by the world rather than by the team's release calendar. The argument of this book The argument of this book can be put in one sentence. A production model is a running system whose correctness decays, and MLOps is the engineering of a closed loop — from data, through features, training, registry and serving, to monitoring and back to retraining — in which every handoff carries an explicit contract that makes the decay visible and the recovery routine. Each phrase in that sentence does work. "A running system" means the unit of concern is not the model file but everything required to turn a raw input into a served prediction and to know whether that prediction was right. "Correctness decays" means that degradation is the default, not the exception, and the design question is how fast you notice and how cheaply you respond. "A closed loop" means the lifecycle is not a line from research to production but a circuit, where the output of monitoring is the input to retraining, and where the circuit has to keep turning without heroics. And "every handoff carries an explicit contract" names where the failures actually live. Models rarely break because the algorithm was wrong. They break because the features computed at serving time differ subtly from those computed in training, because a schema changed upstream and nobody told the model, because a retrained model was promoted without being compared to the one it replaced, or because the monitoring watched latency and never looked at the inputs. Each of these is a broken contract at a boundary. The chapters follow the loop. Chapter 1 sets out why models decay and why the surrounding system, rather than the model, is where most of the engineering effort goes. Chapter 2 deals with reproducibility and lineage — the discipline of being able to say exactly which data, code and parameters produced the model now serving traffic — using MLflow as the working example of an experiment tracker and model registry. Chapter 3 turns to feature stores, the architectural answer to the single most common source of silent failure, the mismatch between features in training and features in serving. Chapter 4 covers data validation and drift detection: what kinds of change happen to a model's world, which statistical tools detect them, and why drift alerts so often generate more noise than signal. Chapter 5 builds on that to treat automated retraining — when to retrain, how to gate a new model before it replaces the old one, and how to avoid automating your way into a worse model. Chapter 6 covers the packaging and serving of models: batch, online and streaming inference, and the interface contract a model must offer. Chapter 7 takes serving onto Kubernetes, the platform most organisations now use to run inference at scale, and explains autoscaling, model servers such as KServe, GPU scheduling and safe rollout patterns. Chapter 8 treats monitoring and incident response as a system in their own right. Chapter 9 steps back to the organisation: who owns what, how to build a platform rather than a collection of one-off pipelines, and where a team that has none of this should start. What this book assumes and what it leaves out The book is written for two readers who need to meet in the middle. One is a data scientist who can train a good model and wants to understand what it takes to keep it good in production. The other is a software or platform engineer who can run services reliably and wants to understand what is different about running models. Both should finish with a shared vocabulary and a clear picture of the whole loop. It assumes familiarity with the basic ideas of supervised learning — training and test sets, overfitting, common evaluation metrics — and with the general shape of modern software delivery: version control, containers, continuous integration. It does not teach either from scratch. It leaves several things out deliberately. It is not a survey of tools; tools change every year and the problems they solve do not. Where it names specific systems — MLflow for tracking and registry, Feast as an example of an open-source feature store, Kubernetes and KServe for serving — it does so because they are widely used and concrete enough to make the ideas tangible, and the principles transfer to their alternatives. It does not treat the training of large language models, whose operational problems overlap with those discussed here but add their own. And it does not attempt a full treatment of responsible AI, fairness or model governance, although Chapter 9 explains where those concerns attach to the lifecycle. One more thing it does not do is pretend there is a single correct architecture. A team serving one model to a few thousand users a day needs a fraction of what a company serving hundreds of models to hundreds of millions of users needs, and building the larger system too early is its own kind of failure. The aim throughout is to explain the forces — the reasons each piece of the loop exists — so that a reader can decide how much of it their own situation actually requires. CHAPTER 1 Why Production Models Decay In 2015 a group of Google engineers led by D. Sculley published a short paper with a memorable title: "Hidden Technical Debt in Machine Learning Systems."1 Its most quoted contribution is a simple observation about proportion. In a real-world machine learning system, they wrote, the code that actually implements the learning algorithm is a small box in the middle of a much larger structure — configuration, data collection, feature extraction, data verification, resource management, analysis tools, process management, serving infrastructure and monitoring. The model is the part everyone talks about. The surrounding system is the part that takes most of the effort and causes most of the failures. A decade later the observation has hardened into the founding premise of MLOps, and it is worth understanding why it is true rather than merely repeating it. The reason is not that model code is easy. It is that a model's behaviour is determined by things outside its code, and those things change without asking permission. A model is a frozen assumption about the world Training a supervised model means finding a function that maps inputs to outputs well on a particular sample of data. The resulting function is only as good as the assumption that the sample represents the cases the model will meet later. Statisticians call this the assumption that training and deployment data are drawn from the same distribution. In a research setting, it is enforced by construction: you split one dataset into training and test portions, so the test set is, by definition, a fair sample of the same world. In production it is enforced by nothing at all. Consider what changes between the moment a model is trained and the moment it makes its ten-millionth prediction. The population of users changes, as a product grows into new markets or attracts a different kind of customer. The behaviour of users changes, with seasons, with economic conditions, with fashions and with the product itself. The systems that produce the input data change: a mobile app ships a new version that reports a field differently, a data vendor revises its methodology, an upstream team renames a column or changes a unit from seconds to milliseconds. And in many applications the relationship between inputs and the correct output changes too: what a fraudulent transaction looks like shifts as fraudsters adapt to whatever the model is catching. Each of these is a form of what the literature calls dataset shift or drift, and Chapter 4 treats the taxonomy properly. The point here is simpler. A deployed model is a frozen snapshot of a relationship that is not frozen. Its quality is highest on the day it is trained and declines from there at a rate that depends entirely on how fast its corner of the world is moving. A model of the physical properties of steel alloys may stay accurate for years. A model of which news articles a user will click may be stale in days. Neither can be judged by looking at the model alone. This is the first and most important difference between machine learning and ordinary software. A sorting function written in 1995 still sorts correctly today. A credit model trained in 2019 met a pandemic in 2020 that changed employment, spending and repayment patterns almost overnight, and many such models had to be recalibrated or overridden within weeks. The code was identical; the world it was making assumptions about was not. Silent failure The second difference is that models fail quietly. When conventional software breaks, it usually does so loudly: an exception is thrown, a request times out, a process crashes. Monitoring built for conventional software — error rates, latency, CPU and memory — catches these failures well. A model that has become wrong does none of these things. It accepts its input, performs the same arithmetic it always performed, and returns a well-formed number. The prediction service is healthy by every infrastructure metric while its outputs are degrading. Worse, the evidence that a prediction was wrong often arrives late or never. A loan default is known months or years after the decision. A churn prediction is confirmed only when the customer leaves, or fails to. A recommendation that was not clicked might have been bad, or the user might simply have been busy. A content moderation decision that removed a legitimate post generates a complaint only if the author bothers to complain. In each case, the ground truth that would let you compute accuracy is delayed, partial or biased by the model's own decisions. You cannot simply watch the accuracy number, because for most production systems there is no accuracy number available in real time. This is why so much of the MLOps toolkit consists of indirect evidence. If you cannot measure correctness directly, you watch the things that predict a loss of correctness: whether the inputs still look like the training data, whether the distribution of predictions has shifted, whether upstream data pipelines delivered on time and in the expected shape, whether the features computed at serving time match those computed in training. These are proxies, and like all proxies they generate false alarms and miss real problems. Much of the craft lies in choosing which proxies to watch and how to act on them. Where the debt accumulates Sculley and his colleagues catalogued the specific forms that hidden debt takes in machine learning systems, and several of their categories remain the most useful map of where things go wrong. The first is entanglement, summarised in the paper's principle that "changing anything changes everything." In conventional software, good design isolates components so that a change to one does not ripple into others. In a model, every input feature interacts with every other through the learned weights. Adding a feature, removing one, or changing how one is computed can alter the model's use of all the rest. There is no such thing as a small, local change to the inputs of a trained model, and therefore no such thing as a safe untested change. The second is undeclared consumers. A model's predictions, once available, tend to be picked up by other systems that its owners never knew about. A team builds a propensity score for one marketing campaign; another team discovers it in a shared table and starts using it to rank customer service queues. Now any change to the model has consequences its owners cannot see, and, worse, the second use may feed back into the data the first model is trained on. The third is data dependency debt, which the paper argued is costlier than code dependency debt because it is harder to detect. A model may depend on a signal produced by another team's pipeline, which depends on a third team's logging. If any link in that chain changes its behaviour — begins emitting nulls instead of zeros, starts deduplicating events, switches time zone — the model's inputs change, and nothing in a compiler or a type checker will flag it. Unstable data dependencies are the most common root cause of the production incidents that machine learning teams report. The fourth is feedback loops. Many models influence the data that will later be used to train them. A recommender shows certain items, users click on the items they are shown, and the next training set says those items are what users want. A fraud model blocks certain transactions, so their true labels are never observed, and the next model has no evidence that some of those blocks were wrong. Direct feedback loops of this kind can be designed around — by reserving a small fraction of traffic for exploration, or by logging the probability with which each decision was made — but only if someone recognises them first. Hidden feedback loops, in which two separate models influence each other through the world, are harder still. The fifth is glue code and pipeline jungles. Because models are built by experimentation, the code around them tends to accrete: a script to pull data, another to clean it, a notebook to compute features, a hand-run job to retrain. Each piece made sense when it was written. Together they form a system nobody fully understands, which cannot be rerun from scratch, and in which a change in one step silently breaks another. The paper's prescription — treat the pipeline as a first-class engineered artefact with tests, not as scaffolding around the model — is the beginning of what every later chapter of this book describes. Finally there is configuration debt. A production model may depend on hundreds of configuration choices: which features to use, which date range to train on, which hyperparameters, which thresholds to apply to the output, which fallback to use when a feature is missing. These choices are often less carefully managed than the code, and a mistake in configuration is as consequential as a bug. How the debts compound: an illustration These categories are easier to recognise in combination than in isolation, because real incidents rarely involve only one. Consider a composite case of the kind that practitioners describe again and again, here a delivery platform that predicts how long each order will take to arrive. The model is trained on a year of completed deliveries, using features such as restaurant preparation time, distance, time of day, weather and the number of couriers active nearby. It performs well at launch, and the estimates it shows customers are noticeably more accurate than the rule of thumb they replaced. The product team, pleased, starts using the same estimate to decide how many couriers to offer bonuses to at peak times — an undeclared consumer, from the point of view of the team that built the model. Several months later, three things happen in the same quarter. The restaurant partnerships team signs a large chain whose kitchens are much faster than the average, and the population of orders shifts. The mobile engineering team changes how courier locations are reported, sending updates every thirty seconds instead of every ten, so the "active couriers nearby" feature is computed from sparser data and its values drop. And because the courier bonus scheme, driven by the model's own estimates, has drawn more couriers into busy areas, deliveries in those areas have become faster than the model expects. No single change is dramatic. The chain's orders are well formed; the courier feature still has plausible values; the feedback through the bonus scheme is gradual. The model's average error rises slowly, and because the team monitors only service latency and error rates, nobody notices until customer complaints about inaccurate estimates reach the support team. By then the model has been degrading for weeks, and untangling which of the three causes contributed what takes longer still, because nobody recorded which data the model was trained on, and the feature computation in serving had been written separately from the one used in training. Every element of that story corresponds to one of the categories above: a data dependency on another team's logging, an undeclared consumer, a feedback loop through the world, entanglement that made the contribution of each feature hard to isolate, and a pipeline that could not be rerun to test hypotheses. And every one of them could have been caught earlier by a practice described later in this book: a data contract with the mobile team, input distribution monitoring, a registry of consumers, a reproducible training pipeline, and an evaluation that sliced error by restaurant type. The lifecycle as a loop, not a line The usual picture of a machine learning project is a line. Someone frames a problem, gathers data, engineers features, trains and evaluates models, picks the best one, deploys it and moves on. That picture is accurate for the first iteration and misleading for everything after. In production, the lifecycle is a loop. Data flows from operational systems into storage. Features are computed from that data, both for training and for serving. A training pipeline produces candidate models, each tracked with its inputs and metrics. Candidates that pass evaluation are registered as versions of a named model. One version is deployed to a serving system that responds to requests or scores batches. The serving system logs its inputs and outputs. Monitoring compares those logs to what the model expected and, as labels arrive, to what actually happened. When monitoring detects that the model's assumptions no longer hold, or a schedule says it is time, the training pipeline runs again on fresher data, and the loop turns. Every stage in this loop is a handoff between components, and often between teams. The data engineering team hands raw data to the feature pipeline. The feature pipeline hands features to training and to serving. Training hands a model to the registry. The registry hands a model to deployment. Serving hands logs to monitoring. Monitoring hands a signal back to training. At each boundary there is an implicit contract — this is the schema, this is the meaning of each field, this is the freshness you can expect, this is the quality bar a model must clear — and almost every production failure can be traced to one of those contracts being violated without anyone noticing. The rest of this book is, in effect, an account of how to make each of those contracts explicit and enforced. Version and track every model with its data and code, so the contract between training and registry can be audited. Compute features once, in one place, for both training and serving, so the contract between feature pipeline and model holds by construction. Validate data against an expected schema and distribution before it reaches a model, so upstream changes announce themselves. Gate new models against the incumbent before they serve traffic. Package models behind a stable interface. Monitor inputs and outputs, not just infrastructure. Each of these practices is unremarkable on its own. What makes MLOps a discipline is doing all of them together, so that the loop can turn without a person walking a model through it by hand. Levels of automation It helps to have a rough sense of what maturity looks like, because few teams need the whole loop on day one. A widely cited framing, published in Google Cloud's architecture guidance, distinguishes three levels.2 At the lowest level, everything is manual: data scientists train models in notebooks, hand a file to engineers, and the model is deployed once and rarely touched. Retraining, if it happens, is a fresh project. At the middle level, the training pipeline itself is automated: the steps from data extraction to model evaluation run as a single orchestrated workflow that can be triggered on a schedule or by an event, so retraining is cheap and the model in production is the output of a reproducible process rather than a notebook session. At the highest level, the pipeline's own code is under continuous integration and delivery: changes to feature logic or training code are tested and deployed automatically, so the team can change the system, not just rerun it. These levels are a useful vocabulary but a poor goal in themselves. The right level depends on how fast the model's world changes, how costly its errors are, and how many models the organisation runs. A team with one model that degrades slowly can live comfortably at the first level with good monitoring and a written retraining procedure. A team running fifty models that each need weekly refreshes cannot survive without the second. The third level pays for itself only when the training code itself changes often. Chapter 9 returns to the question of how to choose, but the principle is set here: automate the parts of the loop that turn most often and fail most expensively, and leave the rest manual until it hurts. What changes when you accept decay Accepting that models decay changes the questions a team asks. The research question — how accurate can we make this model? — remains important, but it is joined by operational questions that are at least as consequential. How will we know when this model is no longer accurate? How long will it take us to find out? What will we do when we do find out, and how long will that take? Can we reproduce the model now in production if we need to debug it? Can we roll back to the previous version in minutes? Who will be woken up when it goes wrong, and will they understand what they are looking at? A model with slightly lower offline accuracy but excellent answers to these questions is usually worth more than a better model with none. A team that has internalised this tends to make different design choices from the start: simpler features that can be computed identically in training and serving, models whose behaviour can be explained when something goes wrong, data sources whose owners have agreed to a contract, and a deployment path that a new model can travel on a Tuesday afternoon without a meeting. None of this requires heavy tooling. It requires recognising that the deliverable of a machine learning project is not a model but a capability: the ongoing ability to produce, deploy, observe and replace models as the world moves. The chapters that follow describe how to build that capability, starting with the most basic requirement of all — knowing exactly what you deployed. CHAPTER 2 Reproducibility, Lineage and the Model Registry At some point in the life of every production model, someone will ask a question that sounds simple. Why did the model make this prediction? Why is the new version worse than the old one on this segment? What data was the model in production last March trained on? Can we rebuild it to check? For a team without discipline around reproducibility, each of these questions starts an archaeology project: digging through notebooks, chat logs and file shares, trying to work out which version of which script was run against which snapshot of which table. Often the answer is that nobody knows, and the model in production cannot be reproduced at all. This chapter is about making those questions cheap to answer. The contract it establishes is the one between training and everything downstream: every model that might serve traffic must carry an unambiguous, machine-readable record of where it came from. That record has three parts — code, data and configuration — and a fourth thing that ties them together, the model registry. What reproducibility actually requires In software engineering, reproducibility usually means that the same source code, built with the same toolchain, produces the same binary. Version control and pinned dependencies get you most of the way. In machine learning, the source code is only one of the inputs. A trained model is a function of the training code, the training data, the configuration (hyperparameters, feature lists, date ranges), the software environment (library versions, which can change numerical results), and often a source of randomness (initialisation seeds, data shuffling, sampling). Change any one of these and you get a different model. Strict bit-for-bit reproducibility is often impractical. Training on GPUs involves non-deterministic floating-point operations; distributed training introduces ordering effects; some libraries do not guarantee determinism even with fixed seeds. Teams that insist on identical weights often spend effort that would be better spent elsewhere. The practical goal is weaker but more useful: given the record of a training run, you can re-execute it and obtain a model whose behaviour is equivalent within a stated tolerance, and you can explain any difference. That requires recording every input precisely enough to re-execute. Code is the easiest of the inputs to pin down: record the commit hash of the repository from which training ran, and refuse to train from uncommitted changes in any pipeline that might produce a production model. Environment is next: record the container image digest, or at minimum the resolved list of package versions. Configuration should live in files under version control, not in command-line flags typed by hand, and the resolved configuration for each run should be logged with it. Data is the hard part. Production training data usually lives in tables that are appended to and sometimes rewritten. A query that selected "the last ninety days of transactions" last March returns different rows today. Three strategies are common. The first is to snapshot: copy the exact training set to immutable storage and record its location and a content hash. This is simple and robust, and its cost is storage, which is usually cheaper than the engineering time lost to irreproducible models. The second is to use a storage layer with time travel — table formats such as Delta Lake, Apache Iceberg and Apache Hudi keep a versioned history of each table, so a run can record the table version or timestamp it read and re-read exactly that state later, within the retention window. The third is to version data alongside code with a tool such as DVC, which stores content-addressed pointers in Git and the data itself in object storage. All three serve the same purpose: replacing "the transactions table" with "the transactions table as of version 1,842," a reference that means the same thing forever. Experiment tracking Before a model reaches production it is typically one of hundreds of candidates. Data scientists try feature sets, architectures and hyperparameters; most runs are discarded. Experiment tracking is the practice of recording every run — its inputs, its metrics and its outputs — in a central, queryable store, so that the eventual winner arrives with its history attached and the losers remain available for comparison. MLflow, an open-source platform originally released by Databricks in 2018 and described by Zaharia and colleagues that year, is the most widely used example, and it makes the concepts concrete.1 Its tracking component organises work into experiments, each containing runs. A run records parameters (the configuration choices), metrics (numerical results, possibly logged over time, such as loss per epoch), tags (free-form metadata such as the Git commit or the dataset version) and artifacts (files, including the trained model itself, evaluation plots and data samples). The tracking server stores metadata in a database and artifacts in an object store, and exposes them through an API and a web interface where runs can be searched, sorted and compared. The instrumentation is deliberately light. A training script needs only a few lines to become a tracked run: import mlflow mlflow.set_experiment("churn-prediction") with mlflow.start_run(): mlflow.log_params({"max_depth": 6, "learning_rate": 0.05, "train_window_days": 90}) mlflow.set_tag("data_version", "transactions@v1842") model.fit(X_train, y_train) mlflow.log_metric("val_auc", roc_auc_score(y_val, model.predict_proba(X_val)[:, 1])) mlflow.sklearn.log_model(model, "model", registered_model_name="churn-classifier") For many common libraries MLflow also offers autologging, which captures parameters, metrics and the model without explicit calls. Autologging is convenient for exploration, but production pipelines usually log explicitly, because explicit logging documents what the team considers important and survives library upgrades that change what autologging captures. What makes tracking valuable is not any single run but the ability to ask questions across runs. Which configuration produced the best validation score on the most recent data? How has the best achievable score changed over the last six retrainings — is the problem getting harder? Did the run that is now in production use the same feature set as the run before it? These questions are trivial with a tracking server and nearly impossible without one. Two habits make tracking more useful than it would otherwise be. The first is to log evaluation on slices, not just overall: metrics for each major customer segment, region or product line, so that a model which improves on average while getting much worse for one group is visible before deployment. The second is to log the data itself, or a reference to it, as a first-class input. A run whose parameters are recorded but whose training data is described only as "latest" is not reproducible, whatever else it captures. The model registry Experiment tracking records everything. A model registry records the subset that matters for production: the named models the organisation depends on, the sequence of versions of each, and which version is meant to be serving traffic in each environment. If the tracking server is a laboratory notebook, the registry is the inventory. In MLflow's registry, a registered model is a named entity — "churn-classifier" — and each time a run logs a model under that name, a new version is created, numbered sequentially and linked back to the run that produced it. That link is the lineage: from any registered version you can navigate to the run, and from the run to its parameters, metrics, code commit, data version and environment. The registry thus answers the archaeologist's question directly. The model in production is version 14; version 14 was produced by run so-and-so; that run trained on data version such-and-such using commit such-and-such. Earlier versions of MLflow assigned each model version a fixed lifecycle stage — None, Staging, Production or Archived — and many teams built their deployment processes around transitions between stages. Recent releases have deprecated that rigid scheme in favour of aliases and tags, which are more flexible. An alias is a mutable named pointer to a specific version: a team might define a "champion" alias for the version that serves production traffic and a "challenger" alias for a candidate under evaluation. Deployment code then loads a model by alias rather than by number: model = mlflow.pyfunc.load_model("models:/churn-classifier@champion") Promoting a new version becomes a matter of moving the alias, which is a single, auditable operation: from mlflow import MlflowClient MlflowClient().set_registered_model_alias("churn-classifier", "champion", 15) The advantage of this indirection is that it separates two decisions that are often confused: producing a model and deploying it. Training pipelines can register new versions as often as they like without affecting production. Promotion — moving the champion alias — is a separate, deliberate step, which can be gated by automated checks, human approval, or both. Rollback is moving the alias back. Because serving systems resolve the alias to a concrete version when they load the model, every prediction can still be attributed to a specific version number, and the registry keeps the history of which version held the alias when. Many organisations separate registries by environment, or use a single registry with environment-specific aliases, so that a model version must be promoted through development and staging before it reaches production. The details matter less than the principle: there is exactly one authoritative answer to "which model is serving production right now?", and it lives in a system, not in someone's memory. Metadata beyond the minimum A registry entry can carry more than lineage. Two kinds of additional metadata repay the effort. The first is a model's signature and input example. MLflow can record the expected input schema — column names, types, and shapes — and an example input alongside the model. Serving infrastructure can then validate incoming requests against the schema and reject malformed ones before they reach the model. This is a small thing that catches a surprising number of integration errors, such as a client sending a feature as a string instead of a number or omitting a field that the model silently imputes. The second is documentation for humans. Mitchell and colleagues proposed in 2019 that every released model be accompanied by a "model card": a short structured document describing its intended use, the data it was trained on, its evaluation results including performance across relevant subgroups, its known limitations, and the contexts in which it should not be used.2 The idea has since been widely adopted, and a registry is the natural place to attach such a card, alongside the version it describes. For teams subject to regulatory review — in lending, insurance, healthcare and, increasingly, any domain covered by the European Union's AI Act — this documentation is not optional, and producing it from metadata the pipeline already records is far easier than writing it after the fact. Lineage as a debugging tool The value of all this becomes obvious the first time something goes wrong. Suppose monitoring reports that the churn model's predicted churn rate jumped overnight from eight percent to fifteen. The on-call engineer opens the registry and sees that the champion alias moved to version 15 the previous afternoon. Version 15's run shows it was trained on data version 1,903, while version 14 was trained on 1,842. Comparing the two runs' logged data statistics shows that a feature measuring days since last login has a very different distribution in the newer data. Following the lineage further — to the feature pipeline, which Chapter 3 discusses — reveals that an upstream change began recording logins from a new authentication service in a different table, so most users now appear to have not logged in for weeks. The fix is to move the alias back to version 14 within minutes, then repair the feature and retrain. Every step in that sequence depended on a recorded link. Without the registry, the engineer would not know a new model had been deployed. Without run metadata, they could not compare the training data. Without data versioning, they could not see what had changed. A team with these links answers the question in an hour. A team without them may not notice the problem for weeks, and when it does, may never establish the cause. Testing that reproducibility actually holds Reproducibility is a property that decays like everything else. A pipeline that could rebuild its model last year may not be able to today: a base container image has been rebuilt with newer system libraries, a data version has passed outside the retention window of the table format, a package has been yanked from the index, a credential to an old bucket has been revoked. Teams discover this at the worst moment, when they need to rebuild an old model to investigate a complaint or answer a regulator, and find that the recorded inputs no longer resolve to anything. The remedy is to test reproducibility as deliberately as any other property. A practical approach is a periodic rebuild: once a month, or once per release of the training pipeline, re-execute the training run that produced the current champion from its recorded inputs alone, in a clean environment, and compare the result with the registered model. The comparison should not be on weights, which may differ for the non-deterministic reasons discussed above, but on behaviour: run both models on a fixed evaluation set and check that their aggregate metrics agree within a stated tolerance and that individual predictions agree within a stated margin for the large majority of examples. A rebuild that fails, or produces a model that behaves differently, is a defect to fix before it matters. Several design choices make this test easier to pass. Pin dependencies to exact versions, and keep the resolved environment as a lock file under version control, rather than relying on loose version ranges that resolve differently over time. Build training images from digests rather than mutable tags. Set and record every random seed the pipeline uses, including those inside data splitting and sampling, even if full determinism on accelerators is not achievable. Retain training data snapshots, or table versions, for at least as long as any model trained on them might remain in production, plus the period for which regulators or customers might ask about decisions made with it. In regulated domains that period can be many years, and a table format's default retention of a few weeks is nowhere near enough. There is also a cheaper, continuous check. Every training run can log a small, fixed "canary" evaluation set's predictions as an artefact. When the pipeline code changes, CI can retrain on a small fixed dataset and compare the canary predictions with those from the previous code version. Unexpected changes signal that something in the pipeline — a preprocessing step, a library upgrade, a changed default — has altered the model's behaviour, and the team can decide whether the change was intended. Common failures and how to avoid them The most common failure of reproducibility practice is partial adoption. A team installs a tracking server, data scientists use it for some experiments, and the production model is still trained by a script someone runs by hand with parameters that are not logged. The fix is a rule, enforced by the pipeline rather than by good intentions: no model reaches the registry except through a pipeline run that records code, data, configuration and environment automatically. Manual experiments remain free to be messy; the path to production is not. A second failure is recording the data as a mutable reference. A run tagged "training data: s3://bucket/churn/latest/" records nothing useful. Any path that can be overwritten should be replaced by a versioned or content-hashed reference. A third is treating the registry as an archive rather than a control point. If the serving system loads a model from a file path that someone copied by hand, the registry's record of which version is in production can drift from reality. The serving system must resolve its model from the registry — by alias or by version — so that the registry is the cause of what is deployed, not merely a description of it. The fourth failure is subtler: reproducing the model but not its inputs at serving time. A model can be retrained exactly and still behave differently in production if the features it receives are computed differently from those it was trained on. That problem — training–serving skew — is serious enough to have produced an entire category of infrastructure, and it is the subject of the next chapter. CHAPTER 3 Feature Stores and the Training–Serving Contract The most dangerous bug in a machine learning system is one in which the model receives, at prediction time, slightly different information from what it learned on. The model does not know. It applies weights learned for one meaning of a feature to values that carry another, and its predictions are quietly worse. Nothing errors. Offline evaluation still looks excellent, because offline evaluation uses the training-time computation. This is training–serving skew, and in practice it is responsible for more silent failures than any other single cause. Feature stores emerged as the architectural answer. The term was popularised by Uber, which described its Michelangelo platform in 2017 and explained that a shared repository of curated features had become one of its most valuable components.1 Since then, feature stores have appeared as open-source projects, notably Feast, which began at the Indonesian technology company Gojek in collaboration with Google Cloud, and as managed services from every major cloud and data platform vendor. The variety of products obscures the fact that they solve a small number of well-defined problems. Understanding those problems is more useful than comparing products, and it also makes clear when a team does not need a feature store at all. How skew happens Skew arises because training and serving typically compute features in different places, with different code, from different data. Training happens in batch. A data scientist writes a query or a dataframe transformation that computes, for every historical example, features such as the customer's number of purchases in the last thirty days, their average order value, or the time since their last support ticket. This code runs against a data warehouse or lake, over months of history, in a language such as SQL or Python, and it can take hours. Serving happens online. When a request arrives — a customer opens the app, a transaction needs scoring — the model needs the same features for this customer, now, in tens of milliseconds. The warehouse query cannot run in that time. So engineers reimplement the feature logic in the serving path: perhaps in Java against an operational database, perhaps with a cache updated by a streaming job. The two implementations are written by different people, in different languages, against different data sources. They will not agree. Common discrepancies include a time window counted in calendar days in one place and rolling twenty-four-hour periods in the other; nulls treated as zero in one and as missing in the other; time zones handled differently; events deduplicated in the warehouse but not in the stream; a lookup table refreshed daily in batch but hourly online. Each difference is small. Taken together they can shift a feature's distribution enough to degrade a model significantly, and because each implementation is internally consistent, neither team's tests catch it. There is a second, subtler source of skew that has nothing to do with duplicated code: data leakage through time. When constructing a training set, each example must be paired with feature values as they were at the moment the prediction would have been made — not as they are now, and not as they were at the end of the day. A churn model trained with "number of support tickets" computed as of today, for customers who churned last month, will learn that churners have many tickets; but some of those tickets were filed after the customer had already decided to leave, or while cancelling. The feature looks predictive in training because it contains information from the future, and the model disappoints in production, where the future is not available. Leakage of this kind is pervasive in hand-built training sets and very hard to spot after the fact. What a feature store does A feature store addresses both problems by making a feature a defined, managed object rather than a fragment of code that happens to exist in two places. Its core responsibilities can be stated in four parts. A single definition. Each feature, or group of related features, is defined once — its source data, its transformation, the entity it describes (a customer, a merchant, a device), and its timestamp semantics. That definition is the only place the logic lives, and both training and serving consume values derived from it. An offline store for training. Feature values are materialised with their timestamps into a historical store, typically tables in the organisation's warehouse or lake. This store holds the full history, so that a training set can be assembled for any period. An online store for serving. The latest value of each feature for each entity is materialised into a low-latency key-value store — Redis, DynamoDB, Cassandra, Bigtable or similar — so that a serving system can retrieve a feature vector for a given customer in a few milliseconds. Point-in-time correct retrieval. When a training set is requested, the feature store joins each labelled example to the feature values that were valid at that example's timestamp, and no later. This is sometimes called an "as-of" join. Done correctly, it eliminates the temporal leakage described above. The two stores have different jobs and are built on different technology, as Table 1 sets out. Table 1. Offline and online feature stores compared. Property Offline store Online store Primary consumer Training pipelines, batch scoring Real-time inference services Data held Full timestamped history Latest value per entity key Typical backing system Warehouse or lakehouse tables Key-value store or in-memory cache Access pattern Large scans and point-in-time joins Single-key lookups Latency expectation Minutes to hours Single-digit milliseconds Main correctness risk Temporal leakage in joins Staleness of materialised values The materialisation process — computing feature values and writing them into both stores — is what keeps them consistent. Because the same definition produces the values in each, a feature called "orders in last thirty days" means the same thing in both, and the residual difference is only one of freshness: the online store holds the value as of the most recent materialisation. Feast, as an open-source example, makes the shape of this visible in its interface. A training job asks for historical features by passing a dataframe of entity keys and event timestamps: training_df = store.get_historical_features( entity_df=labels_df, # columns: customer_id, event_timestamp, label features=["customer_stats:orders_30d", "customer_stats:avg_order_value"], ).to_df() and a serving path asks for the same features, for the current moment, by key: features = store.get_online_features( features=["customer_stats:orders_30d", "customer_stats:avg_order_value"], entity_rows=[{"customer_id": 1042}], ).to_dict() The feature references are identical. That identity is the contract. Batch, streaming and on-demand features Not every feature can be computed the same way, and much of the architectural complexity of a feature store comes from accommodating three different freshness regimes. Batch features are computed on a schedule from data at rest — nightly or hourly jobs over the warehouse. They are cheap, easy to backfill, and adequate for anything that changes slowly relative to the decision being made: a customer's lifetime value, their tenure, their typical spending category. Most features in most systems are batch features, and teams starting out should be suspicious of any claim that they need more. Streaming features are computed continuously from event streams, typically with a stream processor such as Apache Flink or Spark Structured Streaming reading from Kafka or a similar log. They are needed when the signal is in the last few minutes: the number of transactions on a card in the past ten minutes, the number of failed logins in the past hour. Fraud detection, abuse prevention and real-time personalisation rely on them. Streaming features are considerably harder to operate. The streaming computation must produce the same values as a batch backfill of the same definition, or training data (computed from history) will not match serving data (computed live) — skew again, in a new form. Good feature platforms address this by compiling a single definition into both a batch and a streaming job, or by logging the streamed values and using the logs as the historical record. On-demand features are computed at request time from data in the request itself, or by combining request data with stored features: the distance between a transaction's location and the customer's home address, or the ratio of this order's value to the customer's average. They cannot be precomputed because they depend on the request. A feature store handles them by storing the transformation, not the value, and executing it in the serving path and the training path from the same code. A useful discipline is to assign every feature explicitly to one of these three regimes and to justify any that are not batch. Every move from batch to streaming multiplies operational cost, and many teams discover that a feature they believed needed second-level freshness performs nearly as well when updated hourly. Logging features at serving time An alternative, or complement, to the materialise-and-retrieve approach deserves attention because it eliminates skew by construction. Instead of reconstructing historical feature values from source data, the serving system logs the exact feature vector it used for every prediction, and future training sets are built by joining those logged features to labels as they arrive. The advantages are considerable. Whatever the serving path computed, the model will be trained on precisely that, so there is no possibility of the two computations diverging. Point-in-time correctness is automatic, because each logged vector was by definition computed at prediction time. Google's TFX platform and many large-scale recommendation systems have relied on this pattern for years. The disadvantages are equally real. You can only train on features that were logged, so a new feature has no history until it has been in production for a while — you cannot backfill it and train immediately. Logging every feature vector for every prediction can be expensive at high volume. And logged features reflect only the population the model scored, which may be biased by the model's own decisions. In practice, mature systems often combine the two: log features for existing models to guarantee consistency, and use point-in-time joins over source data to bootstrap new features. Sharing, discovery and governance The original motivation at Uber was as much organisational as technical. Different teams kept building the same features — a rider's trip count, a driver's rating, a restaurant's average preparation time — with slightly different definitions and separate pipelines. A central catalogue, with documentation, ownership and usage statistics, meant that a feature built for one model could be reused by another, and that improvements to a shared feature benefited every consumer. That benefit is real, but it comes with the undeclared-consumer problem identified in Chapter 1. Once a feature is shared, its owner can no longer change it freely. A modification that improves one model may break another. A feature store therefore needs the same governance any shared interface needs: versioned feature definitions, so that a changed feature is a new version rather than a silent alteration; a record of which models consume which features; and a deprecation process that notifies consumers before a feature disappears. Without these, a feature store simply centralises the data dependency debt it was meant to reduce. The catalogue also creates a natural place for data quality expectations. A feature definition can carry its expected range, its expected null rate and its expected freshness, and the materialisation process can check them on every run. A feature whose null rate jumps from one percent to forty percent is a far earlier warning of trouble than a model whose accuracy declines a week later. Chapter 4 takes up this kind of validation in detail. Proving that training and serving agree A feature store reduces the opportunity for skew; it does not prove its absence. Materialisation jobs can lag, streaming and batch computations can disagree, on-demand transformations can depend on library versions that differ between the training environment and the serving image. The only reliable assurance is a test that compares the two paths directly, and it is worth building whether or not a team uses a feature store. The test is conceptually simple. Take a sample of real serving requests from the prediction log, which records the feature vector the model actually received. For each, recompute the features as the training pipeline would have computed them for the same entity at the same timestamp, using the point-in-time retrieval path. Then compare, feature by feature. For numeric features, count the share of examples where the two values differ by more than a small tolerance; for categorical features, count exact mismatches; for both, count cases where one side is missing and the other is not. Any feature with a mismatch rate above a low threshold — a fraction of a percent is a reasonable starting point — is a candidate for investigation. Run regularly, this comparison produces a skew report that is one of the most useful artefacts a team can have. It catches the obvious cases, such as a feature that is always null at serving time because a lookup failed. It also catches the subtle ones: a rolling window that differs by a day at month boundaries, a rounding difference that matters only for large values, a categorical encoding that maps a rare value to "unknown" in one path and to its own index in the other. Each of these would otherwise surface only as an unexplained gap between offline and online performance. When mismatches are found, the comparison also tells the team which side is wrong, or at least where to look. If the logged serving value is stale — say, it matches the offline value from six hours earlier — the cause is materialisation lag. If it differs in a consistent way, the cause is usually a difference in transformation logic. If it differs only for entities created recently, the cause is often a backfill gap. Each of these points to a different fix, and none would be diagnosable from the model's metrics alone. Embeddings as features A growing share of features are not hand-crafted aggregates but learned representations: vectors produced by one model and consumed by another. A recommendation system may represent each user and item as an embedding learned from interaction data; a fraud model may use an embedding of a merchant's description; a search ranker may use embeddings of queries and documents from a language model. Embeddings bring a specific versioning problem. An embedding is only meaningful relative to the model that produced it. If the embedding model is retrained, every vector changes, and the coordinates of one version have no relationship to those of the next. A downstream model trained on version three embeddings will produce nonsense if served version four. The feature store, or whatever system holds the embeddings, must therefore treat the embedding model's version as part of the feature's identity, materialise embeddings for a new version alongside the old rather than overwriting them, and switch consumers only when they have been retrained on the new version. In effect, upgrading an embedding model is a coordinated migration across every model that consumes it, and the undeclared-consumer problem becomes acute: an unknown consumer of an embedding will break silently the day the embeddings are regenerated. When you do not need one Feature stores solve real problems, but they are infrastructure with a running cost: another system to operate, another set of pipelines to monitor, another abstraction for data scientists to learn. Several situations do not justify one. If all predictions are made in batch — a nightly job scores every customer and writes the results to a table that applications read — there is no online serving path and therefore no duplicate implementation. Training and scoring can share the same transformation code against the same warehouse, and point-in-time correctness can be achieved with careful SQL. A well-organised set of versioned transformation models, such as those maintained with dbt, may be all the feature infrastructure required. If a team runs only one or two models and their online features are few and simple, the reuse benefit is small. Sharing transformation code between the training job and the serving service as a library, with tests that run both paths on the same inputs and assert equal outputs, can give most of the protection at a fraction of the cost. If the model's inputs arrive complete in the request — an image to classify, a document to summarise, a sensor reading to evaluate — there may be no stored features at all. The decision rule is to adopt a feature store when the cost of skew, leakage and duplicated feature work across the organisation exceeds the cost of running one. That point usually arrives when several teams serve models online with overlapping features, or when a single team's real-time features become numerous enough that keeping two implementations in step is a full-time job. Before that point, the essential thing is not the product but the contract it embodies: one definition per feature, point-in-time correctness in every training set, and a test that proves training and serving agree. CHAPTER 4 Data Validation and Drift Detection If decay is the default condition of a production model, the first operational task is to see it coming. This is harder than it sounds, for the reason set out in Chapter 1: the direct measure of a model's quality — how often it is right — usually arrives late, partially, or not at all. Most of what teams call drift detection is therefore an exercise in watching indirect signals, choosing statistical tests that can distinguish meaningful change from noise, and deciding in advance what to do when a test fires. This chapter separates two activities that are often conflated. Data validation asks whether the data reaching the model is well formed: the right columns, the right types, values in plausible ranges, nulls where nulls are expected and nowhere else. It catches broken pipelines. Drift detection asks whether well-formed data has changed in ways that might make the model wrong. It catches a changing world. Both are necessary, they use different tools, and the first should always run before the second. Validation first A large share of what looks like model degradation is actually data breakage. An upstream service starts sending a price in cents instead of dollars. A join fails and a feature becomes entirely null. A categorical field acquires a new value — a new country code, a new product type — that the model has never seen and encodes as unknown. A timestamp field shifts by an hour after a daylight-saving change. These are not subtle statistical shifts; they are contract violations, and they should be caught by assertions, not by hypothesis tests. The practice of validating data against an explicit schema was described clearly by Polyzotis, Breck and colleagues at Google, whose 2019 paper on data validation for machine learning set out the approach embodied in TensorFlow Data Validation.1 A schema records, for each feature, its type, whether it may be missing, its allowed domain for categorical values, and its expected range for numeric ones. The schema is initially inferred from training data, reviewed by a person, and checked into version control alongside the model. Every new batch of data — every training set, and samples of serving traffic — is checked against it, and violations either block the pipeline or raise an alert. Open-source tools such as Great Expectations, Pandera and the validation components of TFX make this straightforward, and many feature stores and data warehouses now offer built-in checks. The specific tool matters less than three habits. First, the schema is a reviewed artefact, not an automatically regenerated one; if it is re-inferred from each new batch, it will happily accept whatever broken data arrives. Second, validation runs at both boundaries — before training, so that a corrupted training set never produces a model, and at serving, so that a corrupted input never produces a prediction without someone knowing. Third, every check has a defined action. A missing required field might reject the request and return a fallback; a five percent rise in null rate might log a warning; a new categorical value might be passed through but counted. Checks without actions become ignored dashboards. Validation also covers freshness. A feature that has not been updated in three days because a scheduled job silently failed will pass every type and range check while feeding the model stale information. Recording the timestamp of each feature's last successful update and alerting when it exceeds an agreed age is one of the cheapest and most valuable checks a team can implement. A taxonomy of shift Once data is known to be well formed, the question becomes whether it has changed in ways that matter. The machine learning literature distinguishes several kinds of change, and the distinctions are practical rather than academic, because each kind is detected differently and calls for a different response. Write X for a model's inputs and Y for the quantity it predicts. The world that generated the training data can be described by the joint distribution of X and Y, which can be factored either as the distribution of inputs times the conditional distribution of outputs given inputs, or as the distribution of outputs times the conditional distribution of inputs given outputs. Different kinds of shift correspond to different factors changing. Covariate shift is a change in the distribution of inputs, P(X), while the relationship P(Y | X) stays the same. A lender's model trained mostly on applicants aged thirty to fifty begins receiving many more applications from people in their twenties after a marketing campaign. The rules connecting applicant characteristics to default have not changed, but the model is now operating in a region of input space where it saw little data and may be poorly calibrated. Covariate shift is the easiest kind to detect, because it is visible in the inputs alone, without labels. Label shift, sometimes called prior probability shift, is a change in the distribution of outcomes, P(Y), while the way each outcome manifests in the inputs, P(X | Y), stays the same. A disease becomes more prevalent during an outbreak; the symptoms of the disease are unchanged, but a far higher fraction of patients have it. A classifier calibrated to the old base rate will underpredict. Lipton, Wang and Smola showed in 2018 that label shift can be detected and corrected even without labels at serving time, by using the model's own confusion matrix from validation data to estimate how the class proportions have moved.2 Concept drift is a change in the relationship itself, P(Y | X). The same inputs now imply a different output. Fraud is the classic case: a pattern of transactions that was benign last year is now characteristic of a new scheme. Customer preferences are another: the same browsing history now predicts a different purchase because tastes have moved. Concept drift is the most damaging kind and the hardest to detect, because the inputs may look exactly as they did before. It can only be confirmed with labels, or inferred indirectly from changes in how the model's predictions relate to downstream outcomes. Gama and colleagues' 2014 survey of concept drift adaptation added a useful vocabulary for the temporal pattern of change: sudden drift, where the relationship switches abruptly; gradual drift, where old and new concepts alternate with increasing frequency of the new; incremental drift, where the relationship moves continuously; and recurring drift, where old concepts return, as with seasonal patterns.3 The pattern matters for the response. Sudden drift calls for rapid retraining on recent data only; recurring drift calls for models that include seasonal features, or for keeping old models available. In practice, real shifts mix these types, and the categories are best treated as questions to ask rather than labels to assign. Are the inputs different? Are the outcomes different? Is the mapping between them different? Each question has its own detection method. Detecting change in inputs and predictions Detecting covariate shift is a two-sample testing problem: given a reference sample (usually the training data or a recent stable window of production data) and a current sample (the latest window of production inputs), decide whether they come from the same distribution. A handful of methods dominate practice, each with characteristic strengths, as Table 2 summarises. Table 2. Common methods for detecting distribution change. Method Applies to What it measures Practical caution Population Stability Index One feature, binned Weighted divergence across bins Results depend on binning choice Kolmogorov–Smirnov test One numeric feature Largest gap between cumulative distributions Flags trivial shifts at large sample sizes Chi-squared test One categorical feature Departure of category counts from expected Unstable with rare categories Wasserstein distance One numeric feature Effort to move one distribution onto the other Scale-dependent; needs normalising Maximum mean discrepancy Many features jointly Distance between distributions in a kernel space Costly on large samples; kernel choice matters Domain classifier Many features jointly How well a model separates reference from current Needs training and careful validation The Population Stability Index comes from credit scoring, where it has been used for decades to check whether the applicant population has moved away from the population a scorecard was built on. Values of the feature are divided into bins; for each bin, the share of reference and current observations is compared, and the index sums the difference in shares weighted by the logarithm of their ratio. An industry rule of thumb treats an index below 0.1 as stable, between 0.1 and 0.25 as a moderate shift worth investigating, and above 0.25 as a significant shift. These thresholds are convention, not statistical theory, but they have the virtue of being widely understood by risk teams and regulators. The Kolmogorov–Smirnov test compares the cumulative distribution functions of two samples of a numeric variable and reports the largest vertical gap between them, with a p-value. It makes no assumption about the shape of the distributions. Its weakness in production is the weakness of all significance tests at scale: with hundreds of thousands of observations, it will detect shifts so small that they have no practical consequence. A p-value tells you whether a difference is detectable, not whether it matters. Teams that alert on raw p-values from large samples drown in alerts. The chi-squared test does the corresponding job for categorical features, comparing observed category counts in the current window with those expected from the reference proportions. It struggles when many categories are rare, and is usually applied after grouping the long tail into an "other" bucket. Wasserstein distance, sometimes called earth mover's distance, measures how much probability mass must be moved, and how far, to turn one distribution into the other. Unlike a p-value it is an effect size, expressed in the units of the feature, which makes it easier to set thresholds that correspond to meaningful change, though it must be normalised to compare across features. All of these are univariate: they examine one feature at a time. A model with two hundred features, each monitored separately at a five percent significance level, will produce about ten false alarms per check by chance alone. And univariate tests miss shifts that appear only in combinations — two features whose individual distributions are unchanged but whose correlation has reversed. Multivariate methods address this. Maximum mean discrepancy, introduced as a kernel two-sample test by Gretton and colleagues, compares distributions in a high-dimensional feature space and can detect differences in any aspect of the joint distribution.4 A domain classifier takes a more direct route: label reference data zero and current data one, train a classifier to tell them apart, and treat its accuracy as a measure of shift. If no classifier can do better than chance, the distributions are indistinguishable; if one can, its feature importances point to what changed. Rabanser, Günnemann and Lipton compared these approaches systematically in a 2019 study whose title — "Failing Loudly" — states the goal.5 One of their more practical findings was that applying simple univariate tests to a reduced representation of the data, such as the outputs of the model itself or a dimensionality-reduced embedding, often performed as well as more elaborate methods. This points to a strategy many mature teams adopt: monitor the distribution of the model's predictions as a first-line signal. The prediction distribution is one-dimensional, it summarises everything the model sees through the lens of what the model cares about, and a shift in it is almost always worth investigating. Feature-level monitoring is then used to diagnose a shift once one has been detected, rather than as the primary alarm. Detecting change in outcomes Input monitoring detects covariate shift and some label shift. It does not detect concept drift, because concept drift can leave inputs unchanged. For that, labels are required. Where labels arrive quickly — an ad is clicked or not within seconds, a search result is selected or not within the session — teams can compute model quality metrics on a rolling window and monitor them directly. This is the gold standard, and where it is available, the statistical machinery above becomes secondary. Where labels are delayed, a standard approach is to monitor quality on cohorts as their labels mature. A loan model's predictions from January can be scored once January's loans have had six months to default; the team watches the sequence of monthly cohort scores and investigates when one departs from the trend. This is slow, but it is the only direct evidence available, and it is essential for retraining decisions. Some teams supplement it with early proxies — thirty-day delinquency as an early indicator of eventual default — which carry a known, measured relationship to the true outcome. Where labels are biased by the model's own decisions — as when a fraud model blocks transactions whose fraud status is then never observed — the problem is harder. The standard remedy is to hold out a small, randomly selected share of decisions from the model's influence, or to randomise near the decision boundary, so that some labels are observed for every kind of case. This costs something in the short term: some fraud gets through, some good customers get extra friction. It is the price of knowing whether the model still works. A worked example: computing stability on one feature The mechanics of the Population Stability Index are worth walking through once, because the choices involved illustrate the judgement every drift method requires. The numbers below are illustrative. Suppose an insurer's pricing model uses the age of the insured vehicle as a feature. In the reference window, the team divides vehicle age into five bins chosen so that each holds about a fifth of the reference data: up to two years, three to five, six to eight, nine to twelve, and thirteen or more. In the current month's applications, the shares in those bins are 0.14, 0.18, 0.20, 0.22 and 0.26 respectively, against 0.20 in each for the reference. For each bin, the index takes the difference between the current and reference shares and multiplies it by the natural logarithm of their ratio. In the first bin, the difference is −0.06 and the ratio is 0.7, whose logarithm is about −0.357, giving a contribution of about 0.021. In the last bin, the difference is 0.06 and the ratio 1.3, whose logarithm is about 0.262, giving about 0.016. The middle bins contribute much less: about 0.002 for the second, nothing for the third, and about 0.002 for the fourth. The sum is roughly 0.041, comfortably below the conventional threshold of 0.1. By the rule of thumb, the population is stable. But the pattern within the index tells a story the single number hides. Share has moved steadily from newer vehicles to older ones. If the model's pricing is particularly sensitive to vehicle age, or if older vehicles carry risks the model learned poorly because they were scarce in training, a shift of this shape may matter even though the index is small. Conversely, a shift in the middle bins of a feature the model barely uses could produce a larger index with no consequence. The example also shows how sensitive the method is to binning. Had the team used ten bins, or bins based on fixed year boundaries rather than reference quantiles, the index would have been different. Had any current bin been empty, the logarithm would have been undefined, which is why implementations add a small constant to every share. None of these choices is wrong, but they must be fixed in advance and applied consistently, or the index becomes incomparable from month to month. The general lesson applies to every method in Table 2: a drift statistic is a summary that depends on choices, and the person reading it should know what those choices were and look beneath the summary when it moves. Turning signals into decisions The practical difficulty of drift detection is not computing statistics but deciding what to do with them. A system that monitors two hundred features with four tests each, every hour, will generate a steady stream of statistically significant changes, the large majority of which have no effect on the model's usefulness. Teams that wire every such signal to an alert soon learn to ignore alerts, and then miss the one that matters. Several practices help. Prioritise features by importance: a shift in a feature the model barely uses is unlikely to matter, while a shift in one of its top five inputs is worth a look. Monitor effect sizes, not p-values, and set thresholds from historical variation — how much did this feature move in ordinary weeks over the past year? — rather than from textbook significance levels. Use a reference window that reflects the latest stable period of production, not just the training data, so that the alert fires on new change rather than on the known and accepted difference between training and production. Account for seasonality explicitly, comparing this Monday with recent Mondays rather than with the weekly average. And route signals to different destinations by severity: schema violations to an immediate page, a large prediction shift to a same-day ticket, a moderate shift in a minor feature to a weekly review. Above all, connect drift detection to a decision. A drift alert should answer the question: should we retrain, roll back, investigate upstream, or do nothing? If the team cannot say which, the alert is not yet useful. The next chapter treats the most common response — retraining — and explains how to make it safe to automate. CHAPTER 5 Automated Retraining Without Automated Regret Retraining is the most obvious response to decay. If a model was accurate when trained on last year's data and has drifted since, train it again on this year's. The idea is simple enough that teams often automate it early: a scheduled job retrains the model every week and deploys the result. Then, one week, the newly trained model is worse than the one it replaced — because the latest data contained a broken feature, or a holiday anomaly, or a batch of mislabelled examples — and the automated pipeline pushes it into production without anyone looking. Automated retraining is therefore two problems, not one. The first is building a pipeline that can produce a new model reliably and cheaply, without a data scientist in the loop. The second, which matters more, is building the gates that decide whether the new model should replace the old one. A pipeline without gates automates the production of regressions. A pipeline with good gates turns retraining into a routine, low-risk event, which is the whole point. The training pipeline as a product At the manual end of the maturity range described in Chapter 1, retraining means a data scientist opening a notebook, rerunning cells, and handing over a file. Everything that went into the model lives in their head and their scratch directory. The first step towards automation is to turn that notebook into a pipeline: a sequence of discrete, testable steps, each with defined inputs and outputs, orchestrated by a system that can run it on demand, on a schedule, or in response to an event. A typical training pipeline has six or seven steps. Data extraction pulls the training window from the warehouse or feature store, with a point-in-time correct join to labels, and records exactly which data version it read. Data validation checks the extract against the reviewed schema and expected statistics described in Chapter 4, and stops the pipeline if the data is broken. Feature transformation applies any preprocessing that is part of the model rather than the feature store — scaling, encoding, tokenisation — and saves the fitted transformers with the model, so that serving applies identical preprocessing. Training fits the model, possibly with a hyperparameter search. Evaluation scores the candidate on held-out data, overall and on important slices, and compares it with the current production model. Validation and registration applies the promotion gates and, if they pass, registers the candidate as a new model version with full lineage. Deployment, which may be a separate pipeline, moves the candidate into service through a staged rollout. Orchestration tools for this work are plentiful: Apache Airflow, Kubeflow Pipelines, Argo Workflows, Prefect, Dagster, and the managed pipeline services of the major clouds all serve. What they have in common is that each step runs in its own container with declared inputs and outputs, steps can be retried independently, every run is logged, and the pipeline definition is code under version control. The choice among them matters less than the discipline of keeping each step small, deterministic given its inputs, and individually testable. The pipeline must itself be tested. Breck and colleagues' "ML Test Score" rubric, published by Google in 2017, remains a useful checklist.1 It proposes tests in four areas — data, model development, infrastructure, and monitoring — and several of its infrastructure tests apply directly to retraining pipelines: the training process is reproducible; the full pipeline is tested end to end on a small sample of data; the model can be rolled back; a model's quality is checked before it is allowed to serve. A pipeline that has never been run on a small fixture dataset in continuous integration will eventually fail in production at the step nobody had tested since it was written. When to retrain The question of when to retrain has three standard answers, which differ in cost, responsiveness and risk, as Table 3 compares. Table 3. Strategies for triggering retraining. Trigger How it works Strengths Weaknesses Fixed schedule Retrain every day, week or month Simple; predictable cost; easy to operate May retrain needlessly or too late Drift-based Retrain when input or prediction drift crosses a threshold Responds to change in inputs Noisy signals; misses pure concept drift Performance-based Retrain when measured quality falls below a threshold Acts on the quantity that matters Depends on timely, unbiased labels Data-volume Retrain after a set amount of new labelled data accrues Matches learning to new evidence Ignores whether anything has changed Scheduled retraining is where most teams should start. Its cadence should come from evidence about how quickly the model decays, not from habit. A simple way to obtain that evidence is a backtest: train models on historical data ending at several different dates, evaluate each on the months that followed, and plot how quality falls with the age of the model. If a model trained in January loses two points of accuracy by March and five by June, a monthly or bimonthly schedule is reasonable. If it loses almost nothing over a year, retraining quarterly is enough, and the effort is better spent elsewhere. Schedule-based retraining is predictable, its costs are easy to budget, and it does not depend on a fragile drift signal. Its weakness is that it cannot react to sudden change between scheduled runs. Drift-triggered retraining fires when monitoring detects that inputs or predictions have shifted beyond a threshold. It reacts faster to sudden change, but inherits all the noise problems of Chapter 4. It also has a conceptual limitation: retraining on drifted inputs helps only if there are enough recent labelled examples to learn the new pattern, which for slowly labelled problems may not be true for months. A drift alert in such a domain is better treated as a signal to investigate than as a signal to retrain. Performance-triggered retraining fires when a quality metric computed from fresh labels falls below an agreed level. It is the most principled trigger, because it acts on the thing the business cares about. It requires labels that arrive quickly enough to matter and that are not biased by the model's own decisions. Mature systems usually combine these: a regular schedule as the baseline, with drift or performance signals able to trigger an early run, and data-volume checks to ensure that each run has enough new evidence to be worth doing. Whatever the combination, every retraining run is just a candidate; whether it ships is decided by the gates. Choosing the training window Every retraining decision is also a decision about which data to train on, and the choice involves a genuine trade-off. A longer window gives the model more examples and more stable estimates, including of rare events and seasonal effects. A shorter window gives it data that better reflects the world as it is now. When the world has changed suddenly, old data is actively harmful, teaching the model patterns that no longer hold. Several approaches manage the trade-off. A sliding window retrains on the most recent fixed period, say twelve months, and forgets everything older. An expanding window keeps all history and lets the model learn from the full record. Weighted windows keep all history but weight recent examples more heavily, so the model leans towards current conditions without forgetting rare cases entirely. The right choice depends on the drift pattern: sudden drift favours short windows; recurring seasonal patterns require windows long enough to contain at least one full cycle, or explicit seasonal features. As with retraining cadence, the window should be chosen by experiment. The backtest that established how fast the model decays can also compare windows: train with six, twelve and twenty-four months of data ending at each historical cut-off, and see which generalises best to the following period. The answer often differs from intuition. A related question is whether to retrain from scratch or update incrementally. For many classical models, training from scratch on the new window is cheap and avoids accumulating errors. For large neural networks, training from scratch is expensive, and fine-tuning the previous model on new data is common; but fine-tuning introduces its own risk of drifting gradually away from good behaviour on older cases, a phenomenon studied under the name of catastrophic forgetting. Pipelines that fine-tune incrementally should periodically retrain from scratch as a check. Gates before promotion The gates between a trained candidate and production traffic are where automated retraining becomes safe. They fall into four groups. Data gates run before training. If the training extract fails schema validation, has too few rows, has an unexpected label rate, or contains features whose distributions differ wildly from recent history, the pipeline stops. Most bad models are caused by bad data, and it is far cheaper to refuse to train than to detect the bad model afterwards. Baylor and colleagues, describing Google's TFX platform in 2017, emphasised exactly this ordering: data analysis and validation are first-class components of the pipeline, not afterthoughts.2 Absolute quality gates check that the candidate meets minimum standards on held-out data: an area under the curve above some floor, a calibration error below some ceiling, a latency on representative hardware within budget, a model size within memory limits. These catch candidates that are broken outright. Relative quality gates compare the candidate with the current champion on the same evaluation data. This comparison is the heart of the champion–challenger pattern. A candidate that is better than the floor but worse than the model it would replace should not be promoted, however recently it was trained. The comparison should be made on the most recent data available, which the champion has not seen, so that it reflects how each would perform today. A common rule requires the challenger to be at least as good as the champion overall and not worse beyond a set tolerance on any important slice. Slice-level checks matter because a model can improve on average while degrading badly for a specific region, product line or customer group; averages hide such regressions. Behavioural gates test properties that aggregate metrics miss. A credit model's predicted default probability should increase, all else equal, as debt-to-income rises; a test can construct pairs of inputs that differ only in that feature and confirm the ordering. A text classifier should not change its prediction when a name in the input is swapped for another of different apparent gender or ethnicity. A set of known difficult cases — previous incidents, important customers, regulatory edge cases — should be predicted correctly. Such tests, in the spirit of software unit tests, document what the team believes the model must always do, and they catch regressions that held-out metrics average away. Gates should be specified in configuration, versioned with the pipeline, and logged with every run, so that the decision to promote or reject a candidate can be explained after the fact. In MLflow terms, a candidate that passes is registered and assigned a "challenger" alias; promotion to "champion" happens only after the online stage below. Online validation Offline gates use historical data. They cannot fully predict how a model will behave on live traffic, where the population, the feature freshness and the interactions with other systems differ. Two patterns provide online evidence before full promotion, and Chapter 7 describes how to implement them on Kubernetes. Shadow deployment runs the challenger alongside the champion on live traffic. Both receive every request; only the champion's prediction is used. The challenger's predictions are logged and compared. Shadowing carries no risk to users and reveals operational problems — latency, errors, missing features — and differences in prediction distribution. It cannot measure the effect of the challenger's decisions on outcomes, since those decisions are never acted upon. Canary and A/B deployment route a small share of real traffic to the challenger and act on its predictions. This measures real outcomes but exposes some users to a possibly worse model. Canaries are usually small and short, intended to catch gross failures; A/B tests are larger and longer, designed to measure differences with statistical confidence. The decision to promote can then rest on the metric that matters — conversion, loss rate, complaint rate — rather than on a proxy. Not every retraining run needs an A/B test. A routine weekly refresh of a stable model, whose candidate passes every offline gate with a performance close to the champion's, can reasonably go through a short shadow period and a canary. A new model architecture, a new feature set, or a candidate that differs markedly from the champion warrants a full experiment. Matching the scrutiny to the size of the change keeps the pipeline fast for routine work and careful for consequential changes. One retraining run, end to end It helps to see the pieces working together. Consider a subscription business's weekly churn model, retrained every Sunday night by a scheduled pipeline. At two in the morning the orchestrator starts the run. The extraction step requests a training set from the feature store covering the twelve months up to four weeks ago — the most recent four weeks are excluded because churn is defined as cancellation within twenty-eight days, so those examples do not yet have final labels. It records the feature store's data version and the label table's snapshot identifier as run tags. The validation step checks the extract against the reviewed schema. All required features are present; null rates are within their expected bands except one, a feature counting support contacts, whose null rate has risen from two percent to nine. The schema marks this as a warning rather than a failure, so the run continues, but the warning is logged prominently and a ticket is opened automatically for the feature's owner. The label rate — the share of customers who churned — is 4.1 percent, within the historical range. Training runs with the fixed hyperparameters recorded in the pipeline configuration; a full hyperparameter search runs only quarterly. Evaluation scores the candidate on the most recent fully labelled four weeks, which neither the candidate nor the champion has seen. The candidate's area under the curve is slightly higher than the champion's on the same data. Slice evaluation, however, shows that for customers on annual plans — a small but valuable segment — the candidate is noticeably worse. Investigation later shows that the rise in nulls in the support-contacts feature falls almost entirely on annual-plan customers, because of a change in how their support tickets are routed. The relative gate is configured to reject any candidate that is worse than the champion by more than a set tolerance on any defined slice. The candidate fails. It is still registered, because its lineage and evaluation are useful evidence, but it is tagged as rejected and the champion alias does not move. The pipeline posts a summary to the team's channel: candidate rejected, with the slice that failed and the data warning that probably explains it. On Monday morning a data scientist reads the summary, follows the ticket to the feature owner, and confirms the routing change. The feature is fixed by Wednesday. The following Sunday's run passes all gates, enters a shadow period of three days during which its predictions are logged alongside the champion's, shows prediction distributions close to the champion's on every segment, and is promoted by moving the alias on Thursday. Nobody had to rebuild anything by hand, and the bad model never served a single prediction. This is what success looks like. It is undramatic: a gate fired, a person read a report, a data problem was fixed at its source. The value of the system lies in the incident that did not happen. Failure modes of automation Automating retraining introduces risks that manual retraining does not have, and they deserve naming. The first is automating a feedback loop. A recommender that retrains daily on clicks from its own recommendations will narrow steadily, each day learning more confidently that the items it showed are the items users want. A fraud model that retrains on labels only from transactions it allowed will learn nothing about the ones it blocked. Automated retraining amplifies such loops because it closes them faster. Countermeasures — exploration traffic, randomised holdouts, logging of decision probabilities so that training can reweight examples — must be in place before retraining is automated, not after. The second is training on poisoned or anomalous data. A one-off event — a system outage that generated a burst of failed transactions, a viral moment that sent unusual traffic, a bot attack — can make the latest window unrepresentative. If the pipeline trains on it and the gates evaluate on data from the same window, the anomaly can pass every check. Data gates that compare the window with longer history, and human review of significant changes in training data statistics, are the defence. The third is silent gate erosion. When gates reject candidates repeatedly, there is pressure to loosen them, and thresholds drift downwards until they no longer protect anything. Gate thresholds should be changed only by reviewed commits, with a recorded rationale. The fourth is unbounded cost. A hyperparameter search that runs on every retraining, or a retraining triggered by every drift signal, can consume large amounts of compute for little benefit. Cost per run, and runs per month, should be tracked like any other operational metric. Handled with care, automated retraining changes the economics of model maintenance. A model that once required a quarterly project to refresh can be refreshed every week with no human effort beyond reviewing the gate reports, and the team's attention moves from rebuilding models to improving the system that builds them. But the new model still has to reach users, and how it does so — the interface it exposes and the infrastructure that serves it — is the subject of the next two chapters. CHAPTER 6 Packaging Models and Choosing a Serving Pattern A trained model, on its own, is a file of numbers and a dependence on a particular library. To be useful it must be packaged so that it can be loaded anywhere, given an interface that other systems can call, and placed in a serving pattern that matches how its predictions will be consumed. Each of these decisions creates a contract: between the model and its runtime environment, between the model and its callers, and between the prediction service and the business process it supports. Most deployment failures are breaches of one of the three. This chapter covers the decisions that come before infrastructure. The choice between batch, online and streaming inference is made first, because it determines almost everything else; packaging and interface design follow; and the chapter closes with the operational concerns — latency, fallbacks and versioning — that apply whatever the pattern. Batch, online and streaming inference Every model is consumed in one of three broad ways, and Table 4 compares them. Table 4. The three main inference patterns. Pattern When predictions are made Typical latency budget Typical uses Main operational concern Batch On a schedule, for many entities at once Minutes to hours per run Churn scores, demand forecasts, lead ranking Freshness and job reliability Online (request–response) On demand, per request Tens to hundreds of milliseconds Search ranking, fraud checks, personalisation Latency, availability, scaling Streaming Continuously, as events arrive Seconds Anomaly detection, real-time alerts Ordering, state and back-pressure Batch inference computes predictions for a large set of entities on a schedule — every customer's churn risk each night, every product's demand for the coming week — and writes them to a table or cache that applications read. It is the simplest pattern to build and operate. The serving infrastructure is the same as the data pipeline infrastructure the organisation already runs; the model can be loaded once per run; throughput matters but latency does not; failures can be retried; and the outputs can be validated before anyone sees them. Training and serving can share code and data sources, which eliminates many sources of skew. Batch inference is limited by freshness. Predictions are only as current as the last run, and they exist only for entities known at run time. A new user who signed up an hour ago has no score. A customer whose behaviour changed dramatically this morning still carries last night's prediction. For many applications that is fine; a weekly demand forecast has no use for minute-level freshness. Teams often default to online serving because it feels more sophisticated, when batch would serve the business equally well at a fraction of the cost and risk. The first question to ask of any new model is whether batch is good enough. Online inference computes a prediction when a request arrives and returns it synchronously. It is necessary when the input is not known in advance — a search query, a transaction being authorised, an uploaded image — or when predictions must reflect the very latest information. It is the most demanding pattern: the model must be loaded in memory in a long-running service; feature retrieval and inference together must fit within a latency budget set by the calling application, often a small fraction of a user-facing request's total time; the service must scale with traffic and stay available through deployments and failures. Most of the infrastructure discussed in Chapter 7 exists to meet these demands. Streaming inference sits between the two. A stream processor consumes events from a log such as Kafka, computes features over windows of recent events, applies the model, and emits predictions as new events. There is no caller waiting for each response, so latency budgets are looser than online serving, but the system must handle continuous throughput, ordering, late-arriving events and the state required for windowed features. Streaming suits applications where the trigger is an event rather than a request — detecting anomalous sensor readings, flagging suspicious account activity as it happens, updating a user's recommendations after each interaction. Many systems combine patterns. A common hybrid precomputes expensive parts in batch — candidate sets, embeddings, slowly changing features — and does a lightweight final step online. A recommendation service might generate a few hundred candidate items per user nightly and rank them online using the current session's context. This keeps the online path fast while retaining freshness where it matters. Batch inference done properly Because batch inference is simple, it is often built carelessly, and a careless batch job can do as much damage as a broken online service — sometimes more, since its outputs are consumed by many systems and persist until the next run. A few practices separate a reliable batch scoring job from a fragile one. Make every run idempotent. A run identified by its scoring date should produce the same output whether it runs once or three times, and a retry after a partial failure should not duplicate rows or leave half-written results. Writing to a partition keyed by run date, and replacing that partition atomically on completion, achieves this. Table formats with transactional writes make the atomic replacement straightforward. Validate outputs before publishing them. Before downstream systems can see a new set of predictions, the job should check them: the number of scored entities is close to the expected count; the share of nulls is zero or near it; the prediction distribution is within agreed bounds of the previous run's. If the checks fail, the job publishes nothing and the previous run's predictions remain in place, which is almost always better than publishing broken ones. Many batch incidents are cases where a broken upstream table produced a full set of plausible-looking but wrong scores that went straight to a marketing system and generated a campaign before anyone looked. Version the outputs. Each row of predictions should carry the model name and version that produced it and the run identifier. When a downstream team asks why a customer received a particular offer, the answer should be a query away. Keeping previous runs' outputs for a period, rather than overwriting them, also allows comparison between runs and quick restoration if a bad run slips through. Publish through a stable view. Consumers should read from a view or table name that points to the latest validated run, rather than from the job's working tables. Switching the view is the publish step, and switching it back is the rollback. Monitor freshness explicitly. A batch job that silently stops running leaves stale predictions in place indefinitely, and nothing about the stale predictions looks wrong. An alert on the age of the latest published run is simple and essential. None of this is sophisticated, and that is the point. Batch inference gives a team most of the benefits of machine learning with very little infrastructure, provided the job is engineered like any other production data pipeline. Packaging: making a model portable A model that runs only on the laptop where it was trained, with the library versions installed there, is not deployable. Packaging turns it into a self-describing artefact that can be loaded anywhere with the right runtime. The first layer of packaging is serialisation: saving the model's learned state in a format that can be reloaded. Each framework has its own — pickle and joblib for scikit-learn, native formats for XGBoost and LightGBM, SavedModel for TensorFlow, state dictionaries or TorchScript for PyTorch. Framework-neutral formats such as ONNX allow a model trained in one framework to run in a different, often faster, runtime; they are valuable when inference speed matters and the model's operations are supported, but not every model converts cleanly. Serialisation alone is not enough, because a model rarely runs in isolation. It depends on preprocessing — the scaler fitted on training data, the tokeniser, the category encoder — and on postprocessing — a calibration step, a threshold, a mapping from class indices to labels. If these live in separate code that must be kept in step with the model, they are a standing source of skew. The packaged artefact should include them, so that the thing loaded at serving time accepts raw features and returns business-meaningful outputs. MLflow's model format illustrates a practical approach. A logged model is a directory containing the serialised model, a descriptor file listing one or more "flavours" — ways it can be loaded — and environment specifications listing the exact Python and package versions it requires. Every MLflow model supports a generic "python_function" flavour, which exposes a single predict method taking a dataframe or array. A team can wrap arbitrary preprocessing, model and postprocessing logic in a custom class implementing that method, log the whole thing as one model, and every MLflow-aware serving tool will load and call it the same way. The model's signature, recorded when it is logged, declares the input and output schema, which serving systems use to validate requests. The second layer of packaging is the runtime environment, and here containers are the standard answer. A container image bundles the model, its code, its library dependencies and its system libraries into an immutable artefact, identified by a content digest, that runs identically on a developer's machine, a test cluster and production. MLflow can build such images directly from a registered model; many teams instead maintain a small set of standard base images per framework and copy the model artefact in at build time or download it from the registry at start-up. The second approach keeps images generic and small, and lets a new model version deploy without a new image, at the cost of a dependency on the registry at start-up. Whichever approach a team chooses, the deployed artefact must be traceable. The running container should know, and report, the registered model name and version it is serving, so that any prediction can be attributed to a specific model and, through the registry, to the data and code that produced it. Models at the edge A fourth pattern deserves brief mention: running the model on the device where predictions are needed — a phone, a vehicle, a factory controller, a camera — rather than in a data centre. Edge inference avoids network latency, works offline and keeps sensitive data on the device, which can matter for privacy. It also changes the operational picture considerably. Models must be compressed to fit limited memory and compute, typically by quantisation and conversion to a mobile runtime such as TensorFlow Lite, Core ML or ONNX Runtime Mobile. Updates are distributed through app releases or over-the-air mechanisms, so many model versions may be live at once on devices that update at different times, and rolling back is slow. Monitoring depends on telemetry that devices send back when connected, which may be sampled, delayed or restricted by privacy rules. The principles of this book still apply — versioned artefacts, stable interfaces, monitoring of inputs and outputs — but each is harder to implement when the serving fleet is in customers' pockets. Designing the prediction interface An online model is a service, and its interface deserves the same care as any other service's. The interface is a contract with callers, and it will outlive any particular model behind it. A good prediction interface accepts the minimum information the caller naturally has — typically an entity identifier and any request-time context — and does feature retrieval internally. If callers must assemble feature vectors themselves, every caller becomes a place where skew can arise, and every change to the model's features becomes a change to every client. Hiding features behind the service boundary lets the model evolve without its callers noticing. The response should carry more than a number. Including the model name and version makes every prediction auditable. Including a request identifier allows predictions to be joined to outcomes later for monitoring and retraining. Where useful, a confidence or calibrated probability lets callers make their own threshold decisions; explanations, such as the top contributing features, help where decisions must be justified to customers or regulators, though they add latency and should be optional. Two standards make interfaces more portable. The Open Inference Protocol, sometimes called the V2 inference protocol, defines a common REST and gRPC API for model servers, developed jointly by the KServe, NVIDIA Triton and Seldon communities; servers that implement it can be swapped without changing clients. And many teams wrap model calls in their own thin domain API — "score this transaction," "rank these items" — that hides the inference protocol entirely, so that the model server becomes an implementation detail. Latency, fallbacks and graceful degradation For online models the latency budget is a hard constraint, set by the calling application. A payments system may allow a fraud check fifty milliseconds; a search page may allow ranking a hundred. That budget must cover network hops, feature retrieval, preprocessing, inference and postprocessing, and it must hold at the high percentiles, not just on average. A service with a ten-millisecond median and a five-hundred-millisecond ninety-ninth percentile will time out on one request in a hundred, and at scale that is a great many requests. Feature retrieval is frequently the dominant cost rather than the model. A model that needs features from five sources, each with its own lookup, can easily spend most of its budget waiting on network calls. Consolidating features into a single online store lookup, as Chapter 3 described, is often the biggest latency improvement available. The model itself can be made faster by choosing a simpler model where the accuracy difference is small; by converting to an optimised runtime such as ONNX Runtime or TensorRT; by quantising weights to lower precision; or, for neural networks, by distillation into a smaller model trained to mimic a larger one. Batching multiple requests together, discussed in the next chapter, improves throughput on accelerators at some cost to individual latency. Every online model needs a defined behaviour for when it cannot answer in time or at all. The options are a cached prediction from a recent batch run, a simple rule-based fallback, a default that errs on the safe side, or an explicit "no prediction" that the caller handles. Which is right depends on the cost of each kind of error: a fraud check that cannot complete might allow small transactions and decline large ones; a recommendation that times out might show popular items. The fallback should be designed with the business owner, tested regularly by deliberately failing the model in a test environment, and monitored in production — a rising fallback rate is an early warning of trouble. Versioning the interface and the model Models change often; interfaces should change rarely. Separating the two allows each to evolve at its own pace. A new model version that accepts the same inputs and returns outputs of the same meaning can be deployed behind the existing interface without callers noticing — this is what the registry alias mechanism in Chapter 2 and the rollout patterns in Chapter 7 are for. A model change that alters the meaning of the output — a score recalibrated to a different scale, a classifier with a new class — is an interface change, even if the schema looks identical. Callers that apply thresholds to a score will behave differently if its distribution changes. Such changes must be versioned explicitly, communicated to every consumer, and usually run side by side during a migration period. This is one place where the undeclared-consumer problem of Chapter 1 becomes acute. If the team does not know who consumes its predictions, it cannot know who will be affected by a recalibration. Requiring callers to authenticate, and recording which clients call which model versions, turns undeclared consumers into declared ones and makes safe change possible. With a model packaged as a versioned, self-describing artefact, wrapped in a stable interface, and placed in a serving pattern that fits its use, the remaining question is where it runs and how it scales. For online serving at any real volume, the answer in most organisations is now Kubernetes. CHAPTER 7 Scalable Inference on Kubernetes Kubernetes has become the default platform on which organisations run online inference, and it is worth understanding why before discussing how. It was not designed for machine learning. It grew out of Google's experience with its internal cluster managers, Borg and Omega, which Burns and colleagues described in a 2016 retrospective, and it was released as open source in 2014 as a general system for running containerised services across a fleet of machines.1 What makes it suitable for inference is precisely that generality. A model server, once packaged in a container as Chapter 6 described, is just another service, and Kubernetes already solves most of the problems a service has: scheduling it onto machines with the right resources, restarting it when it crashes, routing traffic to healthy instances, scaling the number of instances with load, and replacing old versions with new ones without downtime. The machine-learning-specific difficulties lie in the details: models are large and slow to load, they often need accelerators, their resource profile differs sharply from typical web services, and their rollouts need statistical rather than merely operational validation. This chapter explains the core Kubernetes mechanisms in that light, then describes the model-serving layers built on top of them, with KServe as the main example. The building blocks A Kubernetes cluster is a set of machines, called nodes, managed by a control plane. Users describe the state they want in declarative manifests, and controllers in the control plane work continuously to make the cluster match. For inference, five kinds of object matter most. A Pod is the smallest deployable unit: one or more containers that share a network address and storage, scheduled together onto a node. A model server runs as a container in a pod, sometimes with helper containers alongside it — one that downloads the model from storage before the server starts, or one that forwards logs. A Deployment manages a set of identical pods, keeping the requested number running and replacing them in a controlled sequence when the specification changes. Changing the container image or the model version in a Deployment triggers a rolling update: new pods start, pass their readiness checks, and receive traffic while old pods are drained and removed. A Service gives a stable network name and address to a changing set of pods, load-balancing requests among those that are ready. Callers address the Service, not individual pods, so pods can come and go freely. Resource requests and limits tell the scheduler what each container needs. A request is the amount reserved for the container and used for placement; a limit is the maximum it may consume. For inference these matter more than for many services. A model that needs eight gigabytes of memory once loaded must request it, or the scheduler may place it on a node that cannot hold it, and it will be killed when it exceeds its limit. Accelerators are requested the same way, as extended resources — a pod that asks for one unit of nvidia.com/gpu will be placed only on a node whose device plugin advertises a free GPU. Probes tell Kubernetes about a container's health. A liveness probe detects a container that has hung, so it can be restarted. A readiness probe detects whether a container is ready to receive traffic. For model servers the readiness probe is critical: a large model can take tens of seconds or several minutes to download and load, and a pod that receives traffic before loading completes will fail every request. The readiness probe should check that the model is actually loaded and can produce a prediction, not merely that the process is listening. A startup probe can be added to give slow-loading models time before liveness checks begin. A minimal model service can be built entirely from these pieces: a Deployment running a container that loads a model from the registry, a Service in front of it, sensible resource requests and probes. Many teams start here, and it can carry a surprising amount of production load. Autoscaling Traffic to an online model varies — by time of day, by season, with marketing campaigns and news. Running enough replicas for peak traffic at all times wastes money; running too few for peak causes timeouts. Kubernetes provides several layers of autoscaling. The Horizontal Pod Autoscaler adjusts the number of replicas in a Deployment based on observed metrics. By default it uses CPU utilisation: if average utilisation across pods exceeds a target, it adds replicas; if it falls well below, it removes them. For CPU-bound models this works reasonably well. For many inference workloads, however, CPU is a poor signal. A GPU-backed model may be saturated while its CPU is idle; a model whose latency depends on batch size may degrade long before CPU climbs. The autoscaler can instead be driven by custom metrics — requests per second per replica, in-flight requests, queue depth, or GPU utilisation exported by the hardware vendor's monitoring agent — and these usually track the real constraint more closely. Event-driven autoscalers such as KEDA extend this to external signals, such as the length of a message queue feeding a streaming inference job. The Cluster Autoscaler, or newer node provisioners such as Karpenter, adds and removes nodes when pods cannot be scheduled for lack of capacity or when nodes sit underused. Without it, the Horizontal Pod Autoscaler can request replicas that have nowhere to run. Autoscaling inference has a characteristic problem: cold starts. A new replica of a large model must be scheduled, possibly on a node that must first be provisioned, then pull a container image that may be several gigabytes, then download model weights, then load them into memory or GPU memory, and only then pass its readiness check. This can take minutes. If traffic spikes faster than that, the existing replicas absorb the surge alone, and latency suffers. Mitigations include keeping a minimum number of replicas warm, scaling on leading indicators (such as rising request rate) rather than lagging ones (such as latency), caching model weights on nodes or in fast shared storage, keeping images small by separating the model from the image, and pre-provisioning node capacity before predictable peaks. Model-serving layers on Kubernetes Building every model service from raw Deployments and Services works, but it means each team reimplements model loading, request validation, metrics, scaling configuration and rollout logic. Several open-source projects provide a higher-level abstraction for serving models on Kubernetes. KServe, which began in 2019 as KFServing within the Kubeflow project and later became an independent project, is among the most widely used, and it makes the ideas concrete. Seldon Core, NVIDIA's Triton Inference Server, Ray Serve and BentoML occupy overlapping territory with different emphases. KServe introduces a custom resource called an InferenceService. Instead of writing a Deployment, a Service, autoscaling rules and routing configuration, a team declares what model it wants served and where to find it: apiVersion: serving.kserve.io/v1beta1 kind: InferenceService metadata: name: churn-classifier spec: predictor: minReplicas: 2 maxReplicas: 20 model: modelFormat: name: mlflow protocolVersion: v2 storageUri: s3://models/churn-classifier/15 resources: requests: cpu: "1" memory: 4Gi KServe's controller turns this into the underlying Kubernetes objects. It selects a serving runtime that understands the declared model format — in this case an MLflow model, served through a runtime that implements the Open Inference Protocol mentioned in Chapter 6 — and injects a storage initialiser that downloads the model from the given location before the server starts. It configures readiness checks against the model server, exposes a standard prediction endpoint, and sets up autoscaling. An InferenceService can also declare a transformer, a separate component that performs preprocessing or feature retrieval before the predictor, and an explainer, which returns explanations alongside predictions. KServe supports two deployment modes. In its serverless mode, it uses Knative to scale on request concurrency and can scale a model to zero replicas when it receives no traffic, which is economical for many rarely used models but reintroduces the cold-start problem on the first request. In its raw deployment mode, it creates ordinary Deployments and uses the standard Horizontal Pod Autoscaler, which is simpler to operate and avoids Knative's additional machinery. Teams with steady traffic often prefer the latter. For organisations running hundreds or thousands of small models — one per customer, per region or per product — dedicating a set of pods to each becomes wasteful. Multi-model serving addresses this by loading many models into a shared pool of servers and routing each request to a server holding the right model, loading and evicting models on demand. KServe's ModelMesh and NVIDIA Triton's multi-model capabilities are examples. The trade-off is added complexity and occasional latency when a request arrives for a model that must be loaded first. Accelerators and batching Deep learning models, and especially large language models, often require GPUs for acceptable latency. GPUs complicate serving in three ways. They are expensive and scarce, so utilisation matters. A GPU serving one small request at a time will sit mostly idle. Dynamic batching addresses this by collecting requests that arrive within a short window — a few milliseconds — and running them through the model together, which on parallel hardware costs little more than running one. Crankshaw and colleagues' Clipper system, presented in 2017, showed how adaptive batching under a latency objective can raise throughput substantially while keeping response times within bounds; Triton and other servers now provide configurable dynamic batching as a standard feature.2 The parameters — maximum batch size, maximum wait time — trade individual latency for throughput and must be tuned against the service's latency budget. They are coarse-grained. A small model may need only a fraction of a GPU's memory and compute, but Kubernetes by default allocates GPUs as whole units. Techniques for sharing exist — NVIDIA's Multi-Instance GPU partitions certain data-centre GPUs into isolated slices, each schedulable separately; time-slicing lets several pods share one GPU without isolation — and multi-model serving packs several models onto one device. Each has trade-offs in isolation and predictability. They make large models slow to start. Loading tens of gigabytes of weights into GPU memory takes time, which worsens cold starts and makes aggressive scale-to-zero impractical for large models. Serving large language models adds a further layer. Generating text proceeds token by token, each step depending on the last, so request durations vary enormously and naive batching wastes capacity on padded sequences. Specialised engines address this. The vLLM project, introduced by Kwon and colleagues in 2023, manages the memory used to cache intermediate attention state in fixed-size blocks, much as an operating system manages virtual memory pages, and combines this with continuous batching that adds new requests to a running batch as others finish.3 Engines of this kind can be run inside KServe or similar frameworks, and they have made large models far cheaper to serve. The operational principles of this chapter apply to them unchanged; the difference is in the scale of the resources involved. Reliability details that matter for models Several Kubernetes features that are optional for small stateless services become important for model servers, because model pods are expensive to start and slow to replace. Graceful shutdown. When a pod is removed — during a rollout, a scale-down or a node drain — Kubernetes sends its containers a termination signal and waits a configurable grace period before killing them. A model server should respond to the signal by failing its readiness probe, so that it stops receiving new requests, and then finishing the requests already in flight. If the server exits immediately, every rollout drops requests. For servers that batch requests, the grace period must be long enough to drain the current batch. Disruption budgets. A PodDisruptionBudget tells Kubernetes how many replicas of a service must remain available during voluntary disruptions such as node upgrades. Without one, a routine cluster maintenance operation can evict every replica of a model at once, and because each takes minutes to start elsewhere, the service is down for that time. A budget that requires, say, at least two-thirds of replicas to stay available forces maintenance to proceed gradually. Spreading replicas. Topology spread constraints or anti-affinity rules ask the scheduler to place replicas on different nodes and, where the cluster spans them, different availability zones. A model whose three replicas all sit on one node is one hardware failure away from an outage. Separating model loading from the container image. If model weights are baked into the container, every new model version requires a new image, and every new pod pulls the full image. If they are downloaded at start-up from object storage, images stay small and generic, but start-up depends on the storage system's availability and throughput. Many teams use the download approach with a local cache on each node, so that a pod restarting on the same node does not download again. KServe's storage initialiser implements the download pattern directly; more recent features allow model artefacts to be cached on nodes ahead of time. Autoscaling on the right metric. A Horizontal Pod Autoscaler driven by a request-based metric rather than CPU might look like this: apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: churn-classifier spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: churn-classifier minReplicas: 3 maxReplicas: 30 metrics: - type: Pods pods: metric: name: inference_requests_per_second target: type: AverageValue averageValue: "50" The target — here fifty requests per second per replica — should come from load testing, which finds the throughput at which a single replica's latency starts to exceed its budget, and set the target comfortably below that point. The custom metric must be exported by the model server and made available to Kubernetes through a metrics adapter; without that wiring, the autoscaler has nothing to act on. Safe rollouts on Kubernetes Chapter 5 described shadow, canary and A/B deployment as forms of online validation. Kubernetes and the tools around it provide the mechanisms to implement them. A plain Deployment's rolling update replaces pods gradually but has no notion of model quality: if the new pods pass their readiness probes, they take traffic. That protects against crashes, not against a model that is up but wrong. Model rollouts need traffic splitting controlled by policy. Canary releases route a small, configurable percentage of traffic to the new version while the rest continues to the old one. In KServe, this is a single field: updating an InferenceService with a new storage location and a canaryTrafficPercent value of, say, ten sends a tenth of requests to the new revision, and the percentage can be raised in steps or the canary rolled back by restoring the previous specification. Progressive delivery controllers such as Argo Rollouts and Flagger automate the sequence for ordinary Deployments, increasing the canary's share at intervals and querying a monitoring system at each step; if a metric — error rate, latency, or a model-specific metric such as the share of predictions above a threshold — breaches its limit, they roll back automatically. Shadow deployments require traffic mirroring: every request is sent to both versions, and only the primary's response is returned. Service meshes such as Istio support mirroring as a routing rule, and some serving frameworks implement it directly. The shadow's predictions must be logged with the request identifier so they can be compared with the primary's. Mirroring doubles the inference load during the shadow period, which must be provisioned for. A/B tests differ from canaries in that the split is designed for statistical comparison of outcomes rather than for operational safety. Assignment must be consistent per user, so that a given user always sees the same model during the test, which usually means routing on a hashed user identifier in the application or gateway rather than at random per request. Outcome measurement happens downstream, in whatever system records conversions, losses or complaints, and the decision to promote is made from that analysis. In every case the key is that rollout decisions are driven by metrics specific to the model, not only by infrastructure health. A new model version with perfect uptime and latency that assigns high fraud scores to twice as many transactions as its predecessor should halt its own rollout. Wiring prediction-distribution metrics into the progressive delivery controller, alongside error rate and latency, is what makes that possible. How much of this you need It is easy to read a chapter like this and conclude that serving a model requires a service mesh, a serverless layer, a GPU-sharing scheme and a progressive delivery controller. It does not. A single CPU-bound model serving a few hundred requests per second can run comfortably as an ordinary Deployment with three replicas, a Horizontal Pod Autoscaler, a good readiness probe and a manual canary process. Many teams never need more. The layers are worth adding when their absence starts to cost something specific: KServe or a similar framework when several teams are rebuilding the same serving scaffolding; custom-metric autoscaling when CPU-based scaling causes latency spikes; multi-model serving when idle model pods dominate the cluster bill; automated canary analysis when manual rollouts become a bottleneck or a source of incidents. As with feature stores, the principle is to adopt infrastructure in response to a demonstrated problem rather than in anticipation of one, while designing from the start — through stable interfaces, registry-driven model loading, and models packaged in standard formats — so that adopting it later is easy. What no serving infrastructure provides on its own is knowledge of whether the model is still right. Kubernetes will keep a wrong model running with perfect reliability. Closing that gap is the work of monitoring, to which the next chapter turns. CHAPTER 8 Monitoring, Observability and Incident Response A model in production has two kinds of health, and conventional operations tooling sees only one of them. Operational health is whether the service is up, responding quickly and not throwing errors. Predictive health is whether its outputs are still good. The first is necessary and easy to measure; Kubernetes, load balancers and standard monitoring stacks report it by default. The second is what the model exists for, and nothing reports it unless someone has built the means. This chapter treats monitoring as the closing link of the loop: the place where evidence about the model's behaviour in the world is gathered, interpreted and routed back into decisions about retraining, rollback and investigation. It draws on the statistical tools of Chapter 4 but is concerned with the system around them — what to log, what to measure, how to set objectives, and how a team responds when something breaks. Layers of monitoring It helps to think of model monitoring in four layers, each building on the one beneath. Infrastructure and service health is the foundation: request rate, error rate, latency percentiles, saturation of CPU, memory and GPU, pod restarts, and the health of dependencies such as the feature store and the model registry. These are the "golden signals" that the Google site reliability engineering literature recommends for any service — latency, traffic, errors and saturation — and they should be monitored for a model service exactly as for any other.1 They catch outages and capacity problems. They say nothing about whether the model is right. Data quality is the next layer: the validation checks of Chapter 4 applied continuously to serving inputs. Missing fields, values out of range, unknown categories, null rates, and feature freshness. When an upstream system breaks, this layer usually notices first. Distribution monitoring watches the statistical shape of inputs and outputs over time: feature distributions compared with a reference window, and, most valuable of all, the distribution of the model's predictions. A sudden change in the share of transactions scored as high risk, or in the average predicted demand, is a strong signal that something has changed in the model's inputs or the world. Outcome monitoring is the top layer: model quality measured against actual outcomes as labels arrive, and business metrics — revenue, loss, conversion, complaint rates — attributed to the model's decisions. This is the only layer that directly measures what the model is for, and it is often the slowest. A well-monitored model has signals at every layer, and the layers are designed to be read together. A rise in latency with stable predictions points to infrastructure. A shift in predictions with clean data validation and no upstream changes points to a changing world. A shift in predictions coinciding with a data quality alert points to a broken pipeline. A degradation in measured quality with stable inputs suggests concept drift. Each pattern points to a different response, and a team that sees only one layer cannot tell them apart. What to log Every layer above depends on logs, and the most common reason a team cannot diagnose a model problem is that the information it needs was never recorded. For an online model, each prediction should produce a record containing at least the request identifier; the timestamp; the model name and version; the entity identifier; the feature values actually used, as received from the feature store and after any preprocessing; the prediction, including scores or probabilities and not only the final decision; and whether a fallback was used. Where applicable, it should also record the decision taken on the basis of the prediction, and the probability with which an exploratory or randomised decision was made. This record serves several purposes at once. It is the raw material for distribution monitoring. It is the join key for outcome monitoring when labels arrive. It is, as Chapter 3 noted, a source of skew-free training data. And it is the audit trail when someone asks, months later, why a particular customer was declined. Logging every prediction can be expensive at high volume. Sampling — logging a fixed fraction of requests, or all requests for a subset of entities — reduces cost while preserving statistical usefulness, though a sampled log cannot answer questions about specific individual predictions. Regulatory requirements sometimes demand complete logs for certain decisions. Retention should be set deliberately, balancing the need for long reference windows and retraining history against storage cost and data protection obligations; logs containing personal data are subject to the same rules as any other personal data, including deletion requests. The prediction log is typically written to a message stream and then to analytical storage, where monitoring jobs compute statistics on a schedule. Real-time computation is rarely needed for distribution monitoring; hourly or daily aggregates suffice for most models and are far cheaper. Infrastructure metrics, by contrast, are collected continuously by systems such as Prometheus and viewed in dashboards. Open-source tools including Evidently, NannyML, whylogs and the monitoring features of various platforms compute standard drift and quality reports from such logs. NannyML in particular implements methods for estimating model performance before labels arrive, using the model's own confidence on the current data compared with its calibration on reference data — a useful technique for slowly labelled problems, provided the model is well calibrated and the shift is not one that breaks calibration itself. Attributing business outcomes The top layer of monitoring raises a problem of attribution. A fall in conversion rate, a rise in fraud losses or a jump in customer complaints may be caused by the model, or by any of a hundred other things happening in the business at the same time: a pricing change, a competitor's promotion, a website redesign, a seasonal pattern. A model team that claims credit for every improvement and denies responsibility for every decline will not be believed for long. The cleanest attribution comes from a persistent holdout: a small, randomly assigned share of traffic that continues to receive the previous model, or no model at all, so that the business metric can be compared between the two groups at any time. A holdout costs something — the users in it receive a worse experience, if the model is good — but it gives a continuous, causal measure of what the model contributes, and it reveals gradual decay that would otherwise be lost in the noise of other changes. Where a holdout is impractical, the next best evidence comes from careful comparison of the model's predictions with outcomes for matched cohorts over time, with seasonality accounted for. Either way, the model owner and the business owner should agree in advance how the model's contribution will be measured, so that the measure is not chosen after the results are known. Service level objectives for models Site reliability engineering gave operations teams a disciplined way to decide how reliable a service needs to be: the service level objective. An SLO states a target for a measurable indicator over a window — ninety-nine point nine percent of requests succeed within two hundred milliseconds, measured over thirty days — and the gap between the objective and perfection is an error budget the team may spend on change. When the budget is exhausted, releases slow down; when it is healthy, the team can move faster. The approach transfers to models, with additions. A model service should have conventional SLOs for availability and latency, set with its callers. It can also have objectives for data and prediction quality: feature freshness within an agreed age for a stated share of requests; fallback rate below a stated share; prediction distribution within agreed bounds of the reference; and, where labels permit, a quality metric above an agreed floor measured over each labelled cohort. Writing these objectives down forces a conversation between data scientists, engineers and business owners about what "working" means, and turns vague concern about drift into a concrete commitment that can be met or missed. The objectives should be few. A model with forty SLOs has none, because nobody will act on all of them. Three to six indicators, chosen because a breach would mean a real problem for users or the business, are usually enough, with everything else visible on dashboards for diagnosis but not tied to alerts. Alerting without fatigue The hardest part of monitoring is alerting. Every check that can fire will eventually fire, and each false alarm teaches the people receiving it to pay less attention to the next one. Model monitoring is particularly prone to this, because statistical drift checks fire on harmless changes, as Chapter 4 explained, and because the people responsible for the model are often not the same people who respond to operational pages. Several principles help. Alert on symptoms, not causes: page someone when the model's outputs or its objectives are affected, not when an individual feature's distribution moves; feature-level signals belong on dashboards for diagnosis. Make every alert actionable: each alert should come with a short runbook stating what it means, how to confirm it, and what to do — roll back, disable the model and use the fallback, contact the upstream data owner, or open a ticket for retraining. An alert with no defined response should not page anyone. Route by urgency: data pipeline failures that will corrupt tomorrow's batch predictions need attention today, not at three in the morning; a fraud model whose prediction distribution has collapsed needs attention now. Review alerts regularly: alerts that fired and needed no action should be tuned or removed, and incidents that were not caught by any alert should lead to a new one. Ownership must be explicit. Many organisations struggle with the question of who is on call for a model. The data scientists who built it understand its behaviour but may not be equipped to operate infrastructure at night; the platform engineers who run the serving system can restart pods but cannot judge whether a prediction shift is a problem. A workable arrangement is for the platform team to own operational health, with clear escalation to a named model owner for predictive health, and for the model owner to be responsible for writing the runbooks that let anyone execute the first response — usually a rollback or a switch to fallback — without deep model knowledge. Incident response for models When something does go wrong, the general practices of incident response apply: detect, mitigate, diagnose, fix and learn. Models add their own considerations at each step. Mitigation first. The priority in any incident is to stop the damage, and for models the fastest mitigations are usually rolling back to the previous version or switching to the fallback. The registry and serving practices described in earlier chapters make these fast: moving an alias back, or reverting an InferenceService, takes minutes. Teams should rehearse both, because an untested rollback path is likely to fail when needed — a previous model version may depend on a feature that has since been removed, or on a library no longer in the current image. Diagnosis follows the loop. Model incidents almost always trace to one of the handoffs described in Chapter 1, and the lineage built in Chapter 2 is what makes diagnosis tractable. Was a new model version deployed? Did the training data for that version differ from the previous one? Did an upstream pipeline change its schema, semantics or schedule? Did the feature store's materialisation fail? Did a new client start calling the model with a different population? Did something change in the world? Working through these questions in order, with the prediction log and the registry open, usually locates the cause. Fixes address the contract, not just the instance. If a model failed because an upstream team changed a field's units, the immediate fix is to correct the feature. The lasting fix is to add a validation check that would have caught the change, and to establish a data contract with the upstream team so that future changes are announced. If a retrained model was worse on a segment that offline evaluation did not cover, the lasting fix is a new slice in the evaluation gate. Blameless review. After any significant incident, a written review — what happened, when it was detected, how it was mitigated, what the root causes were, what will change — turns an expensive failure into an investment. The practice, again borrowed from site reliability engineering, depends on focusing on systems rather than individuals; a model incident is almost always the product of missing contracts and checks, not of one person's mistake. An incident, step by step A composite example shows how the layers and practices fit together. A consumer lender runs an online model that scores loan applications and returns a probability of default; applications above a threshold are declined automatically, those below are approved, and those in between go to human underwriters. On a Tuesday afternoon, the distribution monitor flags that the share of applications scored above the decline threshold has risen from about twelve percent in recent weeks to twenty-one percent over the past three hours. The alert pages the platform on-call engineer, whose runbook says: confirm the shift on the dashboard; check whether a model deployment or a feature store change occurred in the past twenty-four hours; if the shift exceeds a stated level and the cause is not immediately clear, switch the model to its previous version and notify the model owner. The engineer confirms the shift and checks the registry. No new model has been deployed in eleven days. The feature store's materialisation jobs all succeeded. Data validation on serving inputs shows no schema violations and no change in null rates. Following the runbook, and because the cost of wrongly declining a large number of creditworthy applicants is high, the engineer switches traffic to the fallback agreed with the credit risk team — routing applications in an expanded band to human underwriters — rather than rolling back to an older model, since nothing about the model has changed. The decline rate returns to normal within minutes, and the underwriting team is told to expect higher volume. The model owner, a data scientist, joins and opens the prediction log. Comparing feature distributions for the past three hours with the previous week, she sees that one feature — the applicant's reported monthly income — has shifted: a far larger share of applications report incomes in a narrow, low range. Slicing by application channel shows the shift is entirely from the mobile app. A message to the mobile team reveals that a new version of the app was released that morning with a redesigned form, in which the income field now asks for weekly rather than monthly income for applicants who select certain employment types. The values are well formed and pass validation; they are simply in different units. The immediate fix, agreed with the mobile team, is to convert weekly to monthly income in the application service before it reaches the model, deployed that evening. The fallback is removed the next morning after the distribution has stayed normal overnight. The review the following week leads to three changes: a data contract between the mobile team and the credit risk team covering the semantics, not just the type, of every field the model consumes; a validation rule checking the income distribution by channel, which would have caught the change within minutes of the release; and the addition of the model team to the mobile release checklist for any change to the application form. Every element of a well-run model incident is present: a distribution alert that caught what infrastructure monitoring could not, a runbook that let a non-specialist mitigate quickly, a fallback designed with the business, lineage and logs that made diagnosis fast, and a review that fixed the contract rather than only the instance. Feedback into the loop Monitoring is not the end of the lifecycle but the point at which it turns. Its outputs feed every earlier stage. Drift and performance signals trigger retraining, as Chapter 5 described. Prediction logs become training data. Incidents lead to new validation rules, new evaluation slices and new behavioural tests. Gaps in outcome data lead to changes in how decisions are randomised or how labels are collected. Monitoring that does not feed back in these ways is only observation. The connection works best when the same people, or closely connected teams, own both ends. A data scientist who reads the monitoring reports for their own model learns which features are fragile, which segments are poorly served and how quickly the world moves — knowledge that shapes the next model's design. When monitoring is owned entirely by a separate operations function, that learning is lost. The organisational choices that make this connection possible are the subject of the final chapter. CHAPTER 9 Teams, Platforms and Where to Start Every chapter so far has described a technical practice, and every one of them depends on an organisational condition. Lineage works only if nobody can deploy a model except through the pipeline. Feature stores reduce skew only if teams agree on shared definitions. Drift alerts help only if someone owns the response. Retraining gates protect only if their thresholds are not quietly loosened under deadline pressure. The hardest problems in MLOps are rarely about which tool to use. They are about who owns each handoff in the loop, and whether the organisation is arranged so that the contracts at those handoffs are kept. This chapter turns to those conditions: how teams divide the work, what a machine learning platform is and when one is worth building, how governance attaches to the lifecycle, and how a team that has none of the practices in this book should begin. Who does what The tension described in the introduction — between the exploratory culture of data science and the delivery culture of software engineering — is usually resolved through some division of roles. Amershi and colleagues' 2019 study of machine learning practice at Microsoft described a nine-stage workflow from requirements through data collection, cleaning, labelling, feature engineering, training, evaluation, deployment and monitoring, and found that the stages most distinctive to machine learning — those concerned with data — were where teams experienced the greatest difficulty and needed the most cross-disciplinary collaboration.1 Kreuzberger, Kühl and Hirschl's 2023 synthesis of MLOps literature and practitioner interviews identified a similar cast: data scientists, data engineers, software engineers, machine learning engineers, DevOps or platform engineers, and the business stakeholders who own the problem.2 The particular job titles matter less than three structural choices. The first is where the line falls between building models and running them. At one extreme, data scientists hand a model to a separate engineering team for production. This is the "throw it over the wall" arrangement, and it reliably produces the failures described throughout this book: the engineers reimplement features and introduce skew, the data scientists never see how their model behaves in production, and nobody owns predictive health. At the other extreme, data scientists own everything from notebook to on-call pager. This produces accountability but asks people to be expert in too many things, and it tends to produce idiosyncratic infrastructure that only its author understands. Most successful organisations sit between: data scientists or machine learning engineers own their models end to end — including their pipelines, their gates and their monitoring — while building on shared infrastructure owned by a platform team. The second is who owns the data. The models described in this book depend on data produced by operational systems owned by other teams, and Chapter 1 identified unstable data dependencies as the costliest form of technical debt. The organisational remedy is the data contract: an explicit, versioned agreement between a data producer and its consumers, stating the schema, semantics, freshness and quality of a dataset, and a process for changing it. A contract makes the producing team aware that a model depends on its data, gives the model team a counterparty to call when things change, and turns schema changes from silent breakages into managed migrations. Data contracts are an organisational practice before they are a technical one; tools can validate them, but the agreement itself has to be negotiated. The third is who owns the business decision. A model is almost always one component in a decision process — whether to approve, recommend, flag or order — and the business owner of that process should be involved in setting the model's objectives, its fallback behaviour and its acceptance criteria. When a fraud model's threshold trades off losses against customer friction, that trade-off is a business decision, not a modelling one. Teams that make it implicitly, inside the model, eventually discover that the business disagrees. Platforms, not projects When an organisation runs one or two models, each can reasonably have its own pipeline, its own serving setup and its own monitoring. As the number grows, this approach multiplies effort and inconsistency. Each team reinvents experiment tracking, writes its own deployment scripts, configures its own dashboards, and makes the same mistakes independently. A machine learning platform is the response: shared infrastructure and paved paths that make the practices in this book the default rather than a per-project achievement. A platform in this sense is not a single product. It is an assembled set of capabilities with a coherent experience for the teams that use it, typically including a tracking server and model registry, a feature store or shared feature library, a pipeline orchestrator with templates for common training patterns, a serving layer with standard rollout mechanisms, a monitoring service that ingests prediction logs and produces standard reports, and a self-service way to request compute, including accelerators. Uber's Michelangelo, cited in Chapter 3, was an early and influential example of such a platform built in-house; many organisations now assemble one from open-source components such as those named in this book, from a cloud provider's managed services, or from a commercial vendor. The most important property of a platform is not its feature list but the effort it takes to do the right thing. If logging a model with full lineage, registering it, and deploying it behind a canary takes one command and a configuration file, teams will do it. If it takes a week of reading documentation, they will find a shortcut. Platform teams should measure themselves by how quickly a new model can travel from a promising experiment to monitored production, and by how few of the failure modes in this book the paved path allows. Platforms have characteristic failure modes of their own. The first is building too early: designing a general system for a hundred models when the organisation has three, and imposing abstractions before anyone knows which ones fit. The second is building in isolation: a platform team that does not ship models itself, and does not sit with the teams that do, builds infrastructure that solves problems nobody has while ignoring the ones they do. The third is rigidity: a paved path that cannot accommodate the occasional model with unusual needs pushes those teams off the platform entirely. A useful discipline is to extract platform components from working projects rather than designing them up front — build the second model's pipeline by generalising the first's, and the fifth by generalising the second's. Build, buy or assemble Every organisation building a platform faces the question of how much to build itself. The options span a spectrum: a fully managed end-to-end service from a cloud provider or data platform vendor; an assembly of open-source components such as MLflow, Feast, an orchestrator and KServe, run on the organisation's own Kubernetes clusters; or custom-built systems for the parts where the organisation's needs are unusual. Managed services minimise operational effort and integrate their components with each other, at the cost of lock-in and of fitting the organisation's practices to the vendor's assumptions. Open-source assemblies offer flexibility and portability, at the cost of integration work — the components were not designed together, and making lineage flow from feature store to registry to serving to monitoring takes deliberate effort — and of operating each component. Custom systems fit exactly but are expensive to build and maintain, and are justified only where the organisation's scale or requirements are genuinely unusual. Most organisations end up with a hybrid, and the useful question is not which option to choose but which interfaces to standardise. If models are packaged in a standard format, served through a standard inference protocol, registered with standard metadata, and logged in a standard schema, then any individual component can be replaced later without disturbing the others. The interfaces are the long-lived part of the platform; the implementations behind them can change as the organisation's needs and the market evolve. Governance attached to the loop Machine learning systems increasingly operate under formal oversight. Financial institutions in the United States have for years been subject to supervisory guidance on model risk management — the Federal Reserve and Office of the Comptroller of the Currency issued their influential guidance, known as SR 11-7, in 2011 — which requires independent validation, documentation and ongoing monitoring of models used in material decisions.3 The European Union's AI Act, which entered into force in 2024 with obligations phased in over the following years, imposes requirements on systems it classifies as high-risk, including risk management, data governance, technical documentation, record-keeping, human oversight, and accuracy and robustness monitoring.4 Sector regulators in healthcare, insurance and employment impose their own. The striking thing about these requirements is how closely they map onto the practices this book has described for purely operational reasons. Technical documentation is the lineage, model cards and gate reports of Chapters 2 and 5. Data governance is the validation, contracts and datasheets for training data — Gebru and colleagues' proposal of a standard documentation format for datasets is a widely adopted model.5 Record-keeping is the prediction log of Chapter 8. Accuracy and robustness monitoring is Chapters 4 and 8. Human oversight is the gate review, the fallback design and the incident process. This alignment is an opportunity. Teams that build governance as a separate compliance exercise — documents written after the fact by people who did not build the model — produce paperwork that is expensive, quickly out of date and only loosely connected to what is running. Teams that generate governance artefacts from the pipeline itself — model cards populated from registry metadata, validation reports from gate results, audit trails from prediction logs — produce documentation that is always current and costs little to maintain. The loop, built well, is its own evidence. Fairness deserves particular mention because it interacts with every stage. A model can be fair at launch and become unfair as populations shift; a retraining run can improve overall accuracy while worsening outcomes for a protected group; a feedback loop can entrench a historical bias. The practical defences are those already described, applied with fairness in mind: slice-level evaluation on relevant groups in every retraining gate, behavioural tests for invariance to protected attributes where appropriate, and monitoring of outcome rates by group in production. These do not settle the hard normative questions about which notion of fairness applies — those require domain and legal judgement — but they ensure the questions are asked of every model version rather than once. Measuring the capability If the deliverable of machine learning work is the continuing ability to produce, deploy, observe and replace models, then that capability should be measured, not just the accuracy of individual models. Software engineering offers a precedent. The research programme summarised by Forsgren, Humble and Kim in Accelerate identified four measures of software delivery performance — deployment frequency, lead time for changes, change failure rate and time to restore service — and found that they distinguished high-performing organisations from others.6 Analogous measures for machine learning are straightforward to define and surprisingly revealing. Lead time from experiment to production is the time between a data scientist deciding that a new model is worth deploying and that model serving traffic under monitoring. In organisations without a paved path, this is often measured in months, dominated by handoffs, reimplementation and manual review. In organisations with a mature platform, it can be days. It is the single best indicator of how much the infrastructure is helping. Time to detect is the time between a model beginning to degrade and someone knowing about it. It is hard to measure directly, but it can be estimated after each incident by looking back through the logs to find when the problem started. A team that discovers problems through customer complaints rather than its own alerts has a detection problem, whatever its dashboards show. Time to restore is the time between detection and mitigation — usually a rollback or a switch to fallback. With a registry-driven deployment and rehearsed rollback, this should be minutes. Without them it can be days. Change failure rate is the share of model deployments that are subsequently rolled back or cause an incident. A very low rate may indicate excellent gates; it may also indicate that the team deploys so rarely that every deployment is a large, carefully reviewed event, which is its own problem. Alongside these, simple coverage measures show where the gaps are: the share of production models with recorded lineage, with input and prediction monitoring, with a defined owner, with a tested fallback, with an automated retraining pipeline. These are not sophisticated metrics, but tracking them across all of an organisation's models turns MLOps maturity from a matter of opinion into something that can be managed. Where to start A team that recognises its own situation in the failure stories of this book — models deployed by hand, no record of training data, no monitoring beyond uptime — faces a long list of possible improvements. Trying to adopt all of them at once is a reliable way to adopt none. The order matters, and a sensible sequence follows the contracts in the loop, starting with those whose absence is most costly. First, make the current state visible. Inventory every model in production: what it does, who owns it, who consumes its outputs, how it was trained, when it was last retrained, and how it is deployed. Many organisations are surprised by what this reveals — models nobody remembers deploying, consumers nobody knew about, critical decisions resting on a script on one person's machine. Second, establish lineage. Set up a tracking server and registry, and require that every model reaching production comes from a logged run with its code, data version and configuration recorded. This is cheap, it pays for itself at the first incident, and every later practice depends on it. Third, monitor what you serve. Log every prediction with its model version and inputs. Add data validation at the serving boundary and track the distribution of predictions over time. Even without labels, a dashboard of prediction distribution by day, with an alert on large changes, catches a large share of real problems. Fourth, automate the training pipeline. Turn the notebook into a pipeline that can be rerun from scratch, with data validation at the start and evaluation against the current production model at the end. Run it on a schedule chosen from a decay backtest, with a human approving promotion. Fifth, standardise deployment. Package models in a standard format, serve them behind a stable interface, load them from the registry by alias, and establish a rollback procedure that has been rehearsed. Add canaries for higher-risk changes. Sixth, address skew and scale. Once the basic loop is turning, examine whether training–serving skew is a problem — a test comparing logged serving features with their training-time equivalents will show it — and adopt shared feature definitions or a feature store if it is. Adopt higher-level serving frameworks, custom autoscaling and multi-model serving when the number and load of models justify them. Finally, automate promotion where it is safe. When the gates have been run and reviewed by humans long enough that the team trusts them — when every rejection and every approval over months has matched what a person would have decided — let routine, low-risk retraining runs promote themselves, keeping human review for larger changes. Each step is useful on its own, and none requires the next. A team that completes only the first three has transformed its ability to understand and protect its models. A team that completes all seven has built the closed loop this book has argued for — not as an act of tool adoption, but as a series of contracts, each one added because its absence had a cost. Conclusion: Designing for the Second Model Most machine learning work is organised around the first model. The project plan, the budget and the celebration are all aimed at the moment a model first serves real traffic. The argument of this book implies that this is the wrong moment to aim at. The first deployment proves that a model can be built. It is the second model, and the tenth, and the hundredth — each produced because the world moved and the last one stopped describing it — that determine whether the effort was worth making. A system that can produce only the first model has delivered a depreciating asset. A system that can produce the second as routinely as the first has delivered a capability. Designing for the second model changes decisions made on the first day. It means choosing features that can be computed identically in training and serving, even if a cleverer feature would score a little higher offline, because a feature that cannot be reproduced at serving time will eventually cause skew. It means recording the data version before the first training run, because the question "what was this trained on?" will be asked, usually at an inconvenient time. It means writing the evaluation gate before there is a second model to compare, so that the comparison exists when it is needed. It means logging predictions from the first request, because the reference distribution for drift detection, the training data for the next model, and the audit trail for the first incident all come from those logs. None of these choices is expensive on day one. All of them are expensive to retrofit. What follows from the loop Several conclusions follow from treating the lifecycle as a closed loop of contracts rather than a line from research to production, and they are worth stating plainly because they cut against common practice. The first is that the boundaries deserve more attention than the components. Teams naturally invest in the parts they own: the data scientist in the model, the platform engineer in the serving infrastructure, the data engineer in the pipelines. The failures described in this book happen between those parts — where features are handed from pipeline to model, where models are handed from training to serving, where predictions are handed to callers who interpret them, where monitoring hands signals back to retraining. An organisation that asks, for each boundary, "what is the contract, and what enforces it?" will find its most important gaps faster than one that audits each component separately. The second is that detection speed matters more than model quality, past a point. A model two points less accurate whose degradation is noticed within a day will outperform, over its life, a better model whose degradation goes unnoticed for a quarter. This does not mean model quality is unimportant; it means that once a model is good enough to be useful, the next unit of effort is usually better spent on monitoring and response than on further tuning. Few teams allocate their time this way. The third is that automation is earned, not adopted. Every automated step in the loop — retraining on a schedule, promotion through gates, scaling with traffic, rolling back on metric breaches — replaces a human judgement with a rule. That is safe only when the rule has been shown to make the same judgement a careful human would. The sequence recommended in the previous chapter — run the gates with a person approving, review their decisions, automate only when they have proved trustworthy — applies everywhere. Teams that automate first and validate later tend to discover the weaknesses of their rules in production. The fourth is that infrastructure should follow evidence of need. Feature stores, model-serving frameworks, GPU sharing schemes, progressive delivery controllers and multi-model serving each solve a real problem, and each carries a cost that is easy to underestimate. The right time to adopt one is when its absence is causing a measurable problem. The wrong time is when a reference architecture says a mature organisation has one. The contracts are mandatory from the start; the tools that enforce them at scale are not. What remains unsettled Several questions at the heart of MLOps lack settled answers, and a reader should be wary of anyone who claims otherwise. Drift detection remains more art than science. The statistical tests are well understood, but the mapping from a detected shift to its effect on model quality is not. A large shift in an unimportant feature may not matter; a small shift in an interaction between two features may matter a great deal. Methods that estimate performance without labels are promising but depend on assumptions that the very shifts they aim to detect may violate. For now, the practical answer is the layered approach of Chapters 4 and 8 — validation, prediction distribution, outcomes — with thresholds tuned by experience. The right cadence and form of retraining is similarly empirical. Backtests help, but they assume the future will drift the way the past did. Continual learning methods that update models incrementally on every new example are an active research area, and they may eventually replace periodic retraining for some applications, but they bring their own risks of instability and forgetting. The operational demands of large generative models are changing fast. The principles in this book apply to them — reproducibility, contracts at boundaries, gated promotion, monitoring of inputs and outputs — but the specifics differ. Evaluation is harder when outputs are open-ended text; drift may mean a change in what users ask rather than in a feature distribution; the model itself is often supplied by a third party who updates it on their own schedule. The discipline of treating such systems as decaying, observable loops transfers directly. Many of the tools are still being invented. And the organisational question — how to divide the work so that someone owns each boundary, without making anyone responsible for everything — has no universal answer. It depends on the size of the organisation, the number of models, the skills available and the risk of the decisions involved. What is clear is that the arrangement in which data scientists hand models to engineers and walk away does not work. What to do on Monday For a reader who has come this far, the most useful next step is small and concrete. Pick one model your organisation depends on. Find out, from records rather than memory, exactly what data and code produced the version now in production. Find out how you would know if it were wrong, and how long that would take. Find out what you would do, and whether the rollback has ever been tried. If any of these questions has no good answer, you have found where to start. Machine learning in production is not a harder version of building models. It is a different activity, closer to running a utility than to conducting an experiment, in which the deliverable is not a model but the continuing ability to produce, observe and replace models as the world changes. The organisations that do this well are not those with the most sophisticated algorithms or the most elaborate platforms. They are the ones that have accepted that every model will decay, and have built, one contract at a time, a system that notices and responds before anyone else has to. Notes Introduction 1. Andrei Paleyes, Raoul-Gabriel Urma and Neil D. Lawrence, "Challenges in Deploying Machine Learning: A Survey of Case Studies," ACM Computing Surveys 55, no. 6 (2022). 2. Shreya Shankar, Rolando Garcia, Joseph M. Hellerstein and Aditya G. Parameswaran, "Operationalizing Machine Learning: An Interview Study," arXiv:2209.09125 (2022). Chapter 1 1. D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-François Crespo and Dan Dennison, "Hidden Technical Debt in Machine Learning Systems," Advances in Neural Information Processing Systems 28 (NeurIPS 2015). 2. Google Cloud, "MLOps: Continuous Delivery and Automation Pipelines in Machine Learning," Cloud Architecture Center (first published 2020). Chapter 2 1. Matei Zaharia, Andrew Chen, Aaron Davidson, Ali Ghodsi, Sue Ann Hong, Andy Konwinski et al., "Accelerating the Machine Learning Lifecycle with MLflow," IEEE Data Engineering Bulletin 41, no. 4 (2018). 2. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji and Timnit Gebru, "Model Cards for Model Reporting," Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT* 2019). Chapter 3 1. Jeremy Hermann and Mike Del Balso, "Meet Michelangelo: Uber's Machine Learning Platform," Uber Engineering Blog (September 2017). Chapter 4 1. Neoklis Polyzotis, Martin Zinkevich, Sudip Roy, Eric Breck and Steven Whang, "Data Validation for Machine Learning," Proceedings of Machine Learning and Systems (MLSys 2019). 2. Zachary C. Lipton, Yu-Xiang Wang and Alexander J. Smola, "Detecting and Correcting for Label Shift with Black Box Predictors," Proceedings of the 35th International Conference on Machine Learning (ICML 2018). 3. João Gama, Indrė Žliobaitė, Albert Bifet, Mykola Pechenizkiy and Abdelhamid Bouchachia, "A Survey on Concept Drift Adaptation," ACM Computing Surveys 46, no. 4 (2014). 4. Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf and Alexander Smola, "A Kernel Two-Sample Test," Journal of Machine Learning Research 13 (2012). 5. Stephan Rabanser, Stephan Günnemann and Zachary C. Lipton, "Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift," Advances in Neural Information Processing Systems 32 (NeurIPS 2019). Chapter 5 1. Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib and D. Sculley, "The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction," Proceedings of the IEEE International Conference on Big Data (2017). 2. Denis Baylor, Eric Breck, Heng-Tze Cheng, Noah Fiedel, Chuan Yu Foo, Zakaria Haque et al., "TFX: A TensorFlow-Based Production-Scale Machine Learning Platform," Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2017). Chapter 7 1. Brendan Burns, Brian Grant, David Oppenheimer, Eric Brewer and John Wilkes, "Borg, Omega, and Kubernetes," ACM Queue 14, no. 1 (2016). 2. Daniel Crankshaw, Xin Wang, Giulio Zhou, Michael J. Franklin, Joseph E. Gonzalez and Ion Stoica, "Clipper: A Low-Latency Online Prediction Serving System," 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 2017). 3. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang and Ion Stoica, "Efficient Memory Management for Large Language Model Serving with PagedAttention," Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP 2023). Chapter 8 1. Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy, eds., Site Reliability Engineering: How Google Runs Production Systems (O'Reilly Media, 2016). Chapter 9 1. Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi and Thomas Zimmermann, "Software Engineering for Machine Learning: A Case Study," Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP 2019). 2. Dominik Kreuzberger, Niklas Kühl and Sebastian Hirschl, "Machine Learning Operations (MLOps): Overview, Definition, and Architecture," IEEE Access 11 (2023). 3. Board of Governors of the Federal Reserve System and Office of the Comptroller of the Currency, "Supervisory Guidance on Model Risk Management," SR Letter 11-7 (April 2011). 4. Regulation (EU) 2024/1689 of the European Parliament and of the Council laying down harmonised rules on artificial intelligence (Artificial Intelligence Act), Official Journal of the European Union (July 2024). 5. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III and Kate Crawford, "Datasheets for Datasets," Communications of the ACM 64, no. 12 (2021). 6. Nicole Forsgren, Jez Humble and Gene Kim, Accelerate: The Science of Lean Software and DevOps (IT Revolution Press, 2018). Further Reading Chip Huyen, Designing Machine Learning Systems (O'Reilly Media, 2022).** The most complete single-volume treatment of production machine learning, with particularly strong chapters on data distribution shift, monitoring and continual learning. A natural next step after this book. Mark Treveil and the Dataiku Team, Introducing MLOps: How to Scale Machine Learning in the Enterprise (O'Reilly Media, 2020).** A shorter, organisation-oriented introduction that is useful for managers and for teams making the case for MLOps investment. Valliappa Lakshmanan, Sara Robinson and Michael Munn, Machine Learning Design Patterns (O'Reilly Media, 2020).** A catalogue of reusable solutions to recurring problems, including several on reproducibility, serving and continued model evaluation that complement the chapters here. Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly Media, 2017).** Not about machine learning, but the clearest explanation available of the storage, streaming and batch-processing systems on which feature pipelines and prediction logs are built. Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy, eds., Site Reliability Engineering: How Google Runs Production Systems (O'Reilly Media, 2016).** The source of the service level objective, error budget and blameless postmortem practices adapted for models in Chapter 8. Freely available online. Brendan Burns, Joe Beda, Kelsey Hightower and Lachlan Evenson, Kubernetes: Up and Running, 3rd ed. (O'Reilly Media, 2022). A practical introduction to the Kubernetes objects and operating model on which Chapter 7 builds. D. Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NeurIPS 2015). Eight pages that remain the best short statement of why machine learning systems are hard to maintain. Worth rereading once a year. Eric Breck et al., "The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction" (IEEE Big Data 2017). A concrete checklist of tests for data, models, infrastructure and monitoring, easily adapted into a team's own production-readiness review. Stephan Rabanser, Stephan Günnemann and Zachary C. Lipton, "Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift" (NeurIPS 2019). A careful empirical comparison of drift detection methods, and a useful corrective to over-elaborate monitoring designs. Dominik Kreuzberger, Niklas Kühl and Sebastian Hirschl, "Machine Learning Operations (MLOps): Overview, Definition, and Architecture" (IEEE Access, 2023). A synthesis of the academic and practitioner literature on MLOps principles, components and roles, with an extensive bibliography. Hashtags: #MLOps #MachineLearningOperations #MachineLearningLifecycle #ProductionMachineLearning #MachineLearningEngineering #ModelDeployment #ModelMonitoring #ModelManagement #ModelVersioning #ModelRegistry #MLflow #FeatureStores #FeatureEngineering #TrainingServingSkew #DataValidation #DataDrift #ConceptDrift #DriftDetection #AutomatedRetraining #ContinuousTraining #CICDForML #MLOpsPipelines #ModelReproducibility #DataLineage #ExperimentTracking #Kubernetes #KServe #ModelServing #MLObservability #ScalableMLOps

Download the Book (PDF): Introduction Every large web application eventually reaches a point where the code is no longer the hard part. The hard part is the queue. Forty engineers in six teams share one repository, one build, one deployment pipeline and one release train. A team that wants to change the colour of a button on the account page waits behind a team migrating the checkout to a new payment provider, which waits behind a flaky end-to-end test owned by nobody. The application still works. It simply stops being possible for any single team to move at the speed its own work would allow. Backend engineering met this wall a decade earlier and answered it with microservices: split the system along business capabilities, give each piece to one team, let each team build, test and deploy on its own schedule, and connect the pieces through explicit contracts. The answer has costs, and the industry has spent years learning them, but the core move held up. A system shaped like the organisation that builds it is easier for that organisation to change. The browser did not get the same treatment for a long time. Teams split their servers into dozens of services and then put a single, monolithic single-page application in front of them, so that every backend team's work still had to be funnelled through one frontend codebase owned by one overloaded team. The ThoughtWorks Technology Radar gave the countermove a name in November 2016: micro-frontends, the practice of decomposing a web application's user interface into independently developed and deployed pieces, each owned end to end by a team that also owns the matching backend capability. Cam Jackson's long article on Martin Fowler's site in June 2019 set out the vocabulary most people still use. Since then the idea has gone from conference talk to production architecture at retailers, banks, streaming services, airlines and software vendors, and the tooling has matured from hand-rolled script loaders to a family of build-time and runtime systems with real engineering behind them. What this booklet argues The argument of this booklet is simple to state and harder to practise. Micro-frontends are an organisational tool that buys team autonomy by spending runtime complexity, and they pay off only when the organisation keeps the shared surface between pieces deliberately small: compose at clear boundaries, isolate by default, and share only what has been turned into an explicit, versioned contract. Everything that crosses a boundary between two micro-frontends, whether a JavaScript library, a CSS rule, a global variable, a piece of state or an assumption about the page, is a coupling point. Couplings are not forbidden; some, such as a single shared copy of React, are worth their cost. But each must be chosen on purpose, owned by someone and versioned like an API. Teams that treat sharing as free end up with a distributed monolith that is slower to change than the monolith they left. This framing settles a lot of arguments that otherwise go round in circles. Should every team be free to choose its own framework? Technically they can, and the later chapters show exactly how React, Vue and Angular components can live on one page. Whether they should depends on how much the organisation is willing to pay in download size, duplicated expertise and integration surface. Should micro-frontends share a global state store? Only if you want every deployment to become a coordinated release. Is Module Federation the right mechanism? It is a strong one, especially in its second generation, but it shares code at runtime, which is precisely the kind of coupling that demands the most discipline. Who this is for The booklet is written for engineers, technical leads and architects who already build web applications and now face the question of how to scale frontend work across several teams. It assumes you know what a bundler does, what a single-page application is and roughly how React, Vue or Angular render a component tree. It does not assume you have run micro-frontends in production. Where a term is specialised, it is defined where it first appears, and a short glossary at the end collects the most important ones. It is also written for the people who make the decision rather than the people who only implement it. A good share of the material concerns team structure, ownership, governance and the economics of the choice, because those factors decide whether the architecture succeeds far more often than any configuration flag does. How the booklet is organised Chapter 1 begins with the problem: why frontends become monoliths, what that costs, and how to tell whether splitting will help or merely move the pain. Chapter 2 deals with the decision that matters most and is easiest to get wrong, which is where to draw the boundaries between micro-frontends. Chapter 3 surveys the composition strategies, from build-time packages through server-side and edge assembly to client-side loading with iframes, routers, Web Components and federation, and explains what each trades away. Chapters 4 and 5 go deep on Module Federation. Chapter 4 explains how the original webpack 5 plugin works underneath: containers, remote entries, share scopes, singleton negotiation and the bootstrap pattern that trips up nearly every first implementation. Chapter 5 covers the second generation: the Module Federation 2.0 runtime and manifest, type sharing, Rspack as the faster webpack-compatible bundler, federation for Vite, Native Federation for Angular, and the browser-native alternative built on import maps. Chapter 6 tackles the problem that surprises teams most, which is CSS. Styles are global by default, and a page assembled from five independently built applications is an invitation to collisions. The chapter compares naming conventions, CSS Modules, CSS-in-JS, cascade layers, the @scope rule and Shadow DOM, and is honest about where each leaks. Chapter 7 shows how to combine React, Vue and Angular on one page, through lifecycle adapters and Web Component wrappers, and what that costs in practice. Chapter 8 addresses communication: how separately deployed pieces exchange information without becoming entangled, with the URL, browser events and narrow contracts preferred over shared stores. Chapter 9 covers delivery and operations, including independent deployment, version discovery, rollback, testing, observability and performance budgets. Chapter 10 turns to governance and migration: the platform team, the design system, the shell as a product, and the strangler-fig path out of an existing monolith. The conclusion draws out what follows from all of this, what remains unsettled and what to do differently on Monday. A note on currency The tooling in this field moves quickly, and several pieces of it changed substantially in the year before this booklet was written. Vite 8 shipped in March 2026 with Rolldown, a Rust bundler, replacing its former pairing of esbuild and Rollup. Module Federation reached its 2.0 milestone in April 2026, with tree shaking for shared dependencies and a reworked runtime. Rspack 2.0 followed days later. Native Federation continues to track Angular's release numbers. Browser support for multiple import maps arrived in Chromium and Safari during 2025, and the CSS @scope rule became available in all major engines when Firefox 146 shipped it in December 2025. Specific version numbers and support details are given where they matter and are accurate as of autumn 2026. The principles underneath them change far more slowly, and those are where the booklet spends most of its time. Chapter 1: Why Frontends Become Monoliths No one sets out to build a frontend monolith. It accumulates. A product starts as a single-page application written by four engineers who sit together and know every file. The structure is sensible for that team: one repository, one package.json, one router, one global stylesheet, one state store, one build that produces one bundle and one deployment that publishes it. Two years later the same structure is carrying thirty engineers, a dozen product areas and a release process that needs a spreadsheet. Nothing about the code is necessarily bad. The problem is that the shape of the codebase no longer matches the shape of the organisation working on it. Conway's Law applies to the browser too Melvin Conway observed in 1968 that organisations which design systems tend to produce designs that copy their own communication structures. The usual reading is descriptive: if three teams build a compiler, you get a three-pass compiler. The more useful reading for architects is prescriptive, sometimes called the inverse Conway manoeuvre: if you want a system with a particular structure, organise the teams that way, and if you have a particular team structure, a system that fights it will be expensive to change. Backend architecture took this lesson seriously during the 2010s. A company would organise around business capabilities, such as search, catalogue, pricing, checkout, accounts and recommendations, and give each capability to a team that owned its service from database to API. The frontend frequently stayed behind. The result was a characteristic shape that is still common in enterprises: many backend teams aligned to the business, and one or two frontend teams aligned to a technology. Every feature that touches the user needs work from a capability team and from the frontend team, so the frontend team becomes a shared dependency for the entire company. It is the most coordinated, most interrupted and most overloaded group in engineering. A second common shape tries to fix this by letting the capability teams contribute directly to the shared frontend codebase. That removes the human bottleneck but replaces it with a technical one. Now six teams commit to one repository and ship through one pipeline. Their changes interleave. A broken test from one team blocks everyone. A dependency upgrade, such as moving from one major version of the framework to the next, must be done for the whole application at once, which means it is done by whoever draws the short straw, slowly, while everyone else works around it. The deployment cadence converges on the speed of the slowest, most cautious team, because there is only one deployment. These are the costs micro-frontends address. It is worth naming them precisely because they are what the architecture should be judged against: • Coordination cost. The time teams spend waiting for, negotiating with or merging around each other. It grows faster than headcount, because the number of possible interactions between teams grows roughly with the square of their number. • Blast radius. How much of the product a single bad change can break. In a monolith, one uncaught exception during module initialisation can blank the whole page. • Release coupling. Whether a team can ship when its own work is ready, or only when the shared train leaves. • Upgrade coupling. Whether a framework, library or build tool can be upgraded in one area without upgrading every area. • Cognitive load. How much of the system an engineer must understand to change their part safely. Team Topologies, the 2019 book by Matthew Skelton and Manuel Pais, made cognitive load a first-class design constraint for this reason: a team can only own what it can hold in its head. What a micro-frontend actually is A micro-frontend is a part of a web application's user interface that a single team can develop, test and deploy independently of the other parts, and that is composed with the other parts into one experience for the user. The definition has three load-bearing words. Part means a slice of user-facing functionality, not a technical layer. A product details page, a checkout flow, a search results area, a recommendations rail or an account settings section can each be a micro-frontend. A "components" layer or an "API client" layer cannot, because no team can deliver user value by changing a layer alone. Independently means the team can merge and release without asking anyone else, and that a release of one micro-frontend does not require a release of any other. This is the property that makes the architecture worth having, and it is the property most often lost by accident. If deploying checkout requires redeploying the shell because they share a compiled-in version of a library, the two are not independent, whatever the repository layout says. Composed means something assembles the parts. That something might be a build step, a web server, an edge function or JavaScript running in the browser. Composition is where most of the engineering lives and where most of the later chapters spend their time. Almost every micro-frontend architecture also has a shell, sometimes called the host, container or app frame. The shell is the piece the browser loads first. It typically renders the global layout, such as the header, navigation and footer, decides which micro-frontends belong on the current page, loads them and gives each a place in the document to render into. A well-designed shell is thin. It does as little as possible, because everything it does is a dependency for every team. The costs you pay instead Micro-frontends do not remove complexity. They move it from the organisation into the runtime, and the runtime bill is real. Being clear about it up front prevents the most common disappointment, which is a team that splits its application, pays all of these costs and discovers it did not have the problem the split solves. The first cost is payload. If two micro-frontends each bundle their own copy of a framework, the user downloads it twice. Even with sharing, there is overhead: each independently built application has its own runtime glue, its own chunks and its own polyfills unless someone coordinates them. Sharing reduces the duplication but introduces version negotiation, which is the subject of Chapter 4. The second cost is consistency. Users do not care which team built which part of the page. They notice when the buttons in checkout are two pixels taller than the buttons in the cart, when the date formats differ between sections, or when one area has a loading spinner and another a skeleton screen. Independent teams drift apart visually and behaviourally unless a design system and some shared rules pull them together, and that design system is itself a shared dependency with its own versioning problem. The third cost is integration risk at runtime. In a monolith, the compiler and the test suite see everything at once. In a micro-frontend system, parts meet for the first time in the user's browser. A change to an event name, a CSS class or a global variable in one micro-frontend can break another, and no build will catch it. Contract testing, discussed in Chapter 9, reduces this risk but never eliminates it. The fourth cost is operational surface. Instead of one pipeline there are many. Instead of one set of performance metrics there are several, and the interesting failures are in the seams between them. Someone has to own the shell, the composition infrastructure, the shared libraries and the observability that ties them together. That someone is usually a platform team, and if the organisation does not fund one, the work lands on whichever team is most conscientious, which is not a sustainable plan. The fifth cost is local optimisation. Autonomy lets teams make good choices for their area. It also lets them make choices that are bad for the product as a whole: a heavier dependency, a different routing approach, a novel state library. Some governance is necessary, and the hard question in Chapter 10 is how to apply it without recreating the bottleneck the architecture was supposed to remove. When splitting pays and when it does not Given those costs, the decision should rest on evidence that the coordination problem is real and large. Several signals point toward splitting: • Several teams, typically four or more, regularly change the same frontend codebase, and they are aligned to distinct business areas rather than to technical layers. • Release frequency has fallen as headcount has grown, and the cause is waiting on shared pipelines, merge queues or release trains rather than on genuine product dependencies. • Framework or library upgrades are routinely postponed because they cannot be done incrementally. • The product already has natural seams, such as distinct sections of the site that users move between, with limited shared state across them. • The organisation must integrate user interfaces it does not fully control: acquired products, partner widgets, or applications built by separate business units on different stacks. Other signals point away from it: • One or two teams own the whole frontend. They have a coordination problem only with themselves, and a well-organised modular monolith, with clear internal package boundaries, lint rules that enforce them and fast incremental builds, will serve them better. • The product is a single dense screen where every part interacts with every other part, such as a design tool or a trading terminal. Splitting it produces micro-frontends that must constantly talk to each other, which is coupling in its most expensive form. • The real problem is a slow build or a flaky test suite. Those have direct fixes. Splitting an application to escape a slow build trades one slow build for many builds plus an integration problem. • There is no capacity to fund a platform function. Without someone owning the shell, the shared libraries and the delivery infrastructure, the system decays. It helps to consider the alternative seriously. A modular monolith keeps one deployable application but divides it into well-bounded packages, often in a monorepo managed with a tool such as Nx or Turborepo, with ownership files assigning each package to a team and dependency rules preventing packages from reaching into each other. It provides much of the code-ownership benefit of micro-frontends with none of the runtime costs. What it does not provide is independent deployment or independent upgrades. If the organisation's pain is mainly about who can change what, a modular monolith is often enough. If the pain is about when and how each team can ship, it is not. A worked example: the retailer with one checkout team Consider an online retailer whose web shop is a single React application of roughly four hundred thousand lines. Seven product teams contribute: discovery (search and listing pages), product (details pages and reviews), basket, checkout, account, content (marketing pages) and loyalty. A small web platform team owns the build and the deployment pipeline. The application deploys twice a week on a train. Each deployment requires a regression pass by a QA group, and any failure holds the whole train. The symptoms are familiar. The checkout team, which handles payments and is therefore cautious, wants every release reviewed carefully. The content team, which runs campaigns, wants to ship landing pages the day marketing approves them. Both share one train, so the content team waits for checkout's caution. An upgrade from one major version of React to the next has been in progress for nine months because the checkout and account areas depend on a forms library that has not been updated. The global stylesheet has accumulated so many overrides that engineers add !important defensively. This retailer is a strong candidate. Its teams are already aligned to business areas. Its pages are distinct and users move between them through navigation, so most micro-frontends will occupy whole pages or large regions rather than interleaving on one screen. The upgrade problem is exactly upgrade coupling. The release problem is exactly release coupling. And there is already a platform team that can own the shell. A sensible plan would not split everything at once. It would extract the content area first, because it has the strongest need for independent release and the weakest coupling to the rest of the application: landing pages need the header, footer and a few design system components, and almost nothing else. Next would come checkout, which gains the most from isolation of its blast radius and its upgrade schedule. The remaining areas would stay in the original application, which now plays the role of one large micro-frontend among several, until each has a reason to leave. Chapter 10 returns to this pattern under the name the strangler fig. The same retailer, with two frontend teams instead of seven, would be a poor candidate. The right move would be to modularise the codebase, fix the test suite and move from a train to continuous deployment. Micro-frontends would add a shell, a composition layer and a runtime integration problem to solve a coordination problem that two teams could solve by talking to each other. The shape of the rest of the problem If the decision is to split, four questions follow, and the rest of this booklet takes them in roughly the order a team meets them. Where do the boundaries go, so that teams can work independently without constant negotiation? How are the parts composed into one page, and by what mechanism are they loaded? How are the parts kept from interfering with each other, in code, in styles and in the frameworks they use? And how do separately deployed parts communicate, ship, get tested and get observed, so that independence does not turn into chaos? The order matters. Teams tend to start with the third question, because tools such as Module Federation are concrete and interesting, and the configuration can be working by the end of the afternoon. But a well-configured federation between badly drawn boundaries is worse than a monolith, because it adds runtime coupling to the organisational coupling the boundaries failed to remove. Boundaries come first. Chapter 2: Drawing the Boundaries The single most consequential decision in a micro-frontend architecture is where one micro-frontend ends and the next begins. Get it right and teams can work for weeks without needing each other. Get it wrong and every feature becomes a cross-team negotiation conducted through runtime events and shared libraries, which is more painful than the monolith it replaced. Tools can be swapped in a quarter. Boundaries tend to last for years, because they become team structures, repositories, pipelines and on-call rotations. Vertical slices, not horizontal layers The first principle is that a micro-frontend should be a vertical slice of the product: a piece of user-facing functionality that a team owns from the user interface down to the services and data behind it. The alternative, slicing horizontally by technical layer, reproduces the problem the architecture is meant to solve. A horizontal split might give one team the "design system micro-frontend", another the "data layer micro-frontend" and another the "pages micro-frontend". Every feature then needs changes in all three, made by three teams, released in a coordinated order. That is a distributed monolith: the deployment units are separate but the change units are not. The test of a good boundary is whether the most common changes, the ones product managers ask for every week, fall inside one micro-frontend. Vertical slicing borrows directly from domain-driven design, Eric Evans's 2003 approach to modelling software around the business. Its central idea for this purpose is the bounded context: a part of the business within which terms have a single, consistent meaning and a single model applies. In a retailer, "product" means something different to the catalogue team (a set of attributes and media), to the pricing team (a thing with a price history and promotions) and to the warehouse team (a stock-keeping unit with a location and quantity). Each meaning belongs to a context. Bounded contexts are natural candidates for team ownership and therefore for micro-frontend boundaries, because changes within a context tend to stay within it. In practice, three kinds of vertical boundary are common, and most systems use a combination. Page-level or route-level boundaries assign whole pages, or groups of routes, to a micro-frontend. Everything under /checkout belongs to the checkout team; everything under /account belongs to account. This is the simplest arrangement and the most robust. Each micro-frontend owns its whole screen apart from the shell's header and footer, so there is little opportunity for parts to interfere with each other. Navigation between areas is a hand-off from one micro-frontend to another, mediated by the URL. Most successful migrations begin here. Fragment-level boundaries place several micro-frontends on one page, each owning a region. A product details page might be owned by the product team but include a recommendations rail from the personalisation team, a "buy box" from the basket team and a reviews section from the community team. Fragments are attractive because many real pages are like this: they gather capabilities from several parts of the business. They are also where most integration trouble arises, because fragments share a viewport, a stylesheet cascade, a set of global objects and the user's attention. They need clear contracts for what each fragment receives and emits. Widget-level boundaries go smaller still: a single button, badge or form field owned by another team. A "save for later" heart icon, a loyalty points badge, a delivery estimate. Widgets are sometimes justified, particularly when the widget represents a capability whose logic changes often and must be consistent everywhere it appears. More often they are a sign that a boundary has been drawn too finely. If a widget needs data the host page already has, or if its appearance must match the surrounding page exactly, it may be better delivered as a library component than as an independently deployed micro-frontend. Heuristics for finding the seams No formula gives the right boundaries, but several heuristics point toward them consistently. Follow the teams you want, not only the teams you have. If the organisation is already aligned to business capabilities, the boundaries can mirror the teams. If it is not, drawing micro-frontend boundaries is an opportunity to decide what the teams should be. Drawing boundaries that cut across existing teams guarantees friction; drawing them to fit a team structure that is about to change wastes effort. Follow the user's journey. Users move through a product in stages: they discover, evaluate, decide, pay, then manage what they bought. Each stage tends to have its own screens, its own data and its own rate of change. Boundaries between stages are crossed by navigation, which is the cheapest possible integration. Follow the data ownership. A micro-frontend should, wherever possible, read and write data through services owned by the same team. If the checkout micro-frontend has to call the basket team's API, read the account team's addresses and update the loyalty team's points balance, its team will spend much of its time coordinating with three others. Some cross-team data access is inevitable; it should go through stable, versioned APIs rather than shared client-side state. Follow the rate of change. Parts of the interface that change weekly should not share a deployment with parts that change yearly. Marketing content, experiments and promotional surfaces change fastest. Account settings and legal pages change slowly. Separating them lets fast-moving areas move without risking slow ones. Avoid boundaries that need constant chatter. If two candidate micro-frontends would need to exchange events on most user interactions, they belong together. A filter panel and the results list it filters are a classic example: splitting them across teams produces an event protocol that is really a function call with extra steps. Prefer fewer, larger micro-frontends. The word "micro" misleads. A micro-frontend should be as large as one team can own comfortably, not as small as it can technically be. Each additional micro-frontend adds a pipeline, a deployable artefact, an entry in the shell's routing configuration, a set of shared-dependency negotiations and a potential failure mode. Organisations that succeed with this architecture usually have somewhere between three and fifteen micro-frontends, roughly one or two per team, not dozens per team. The shell and what it should own Every boundary decision implies a decision about the shell, because whatever no micro-frontend owns ends up there. The shell's responsibilities should be few and stable: • Loading the initial document and the minimum script needed to start. • Rendering global chrome: header, primary navigation, footer. Even these can be delegated, with the header itself delivered as a micro-frontend owned by a navigation team, but the shell decides where it goes. • Mapping URLs to micro-frontends, and mounting and unmounting them as the user navigates. • Providing a small number of cross-cutting services through narrow interfaces: the current user's identity and authentication token, feature flags, locale, analytics and error reporting. • Handling the failure of a micro-frontend gracefully, so that one broken area shows an error state rather than blanking the page. Everything else belongs in a micro-frontend. The temptation to put shared business logic in the shell is strong, because the shell is the one thing every team depends on. Resisting it is essential. A shell that knows about baskets, prices or loyalty points becomes a monolith in disguise, and every change to it requires the coordination the architecture was meant to avoid. A good test is to ask how often the shell needs to change. If the answer is "whenever a team ships a feature", the shell is doing too much. If it is "when we add a new micro-frontend or change the global layout", it is about right. Contracts at the boundary Once boundaries are drawn, each one needs a contract: an explicit statement of what crosses it. For a page-level micro-frontend, the contract may be as small as "it is mounted at these routes, receives a DOM element to render into and a small context object, and must clean up after itself when unmounted". For a fragment, it will include the inputs the fragment needs, such as a product identifier or a locale, the events it emits, such as "item added to basket", and any constraints on its rendering, such as the maximum height it may claim before content loads. Treat these contracts as APIs. Write them down, preferably as TypeScript types published in a small shared package that both sides depend on. Version them. Deprecate old fields before removing them. The difference between a micro-frontend system that scales and one that collapses under its own coordination is usually whether the boundary contracts were treated with the same seriousness as backend API contracts. A minimal contract for a mountable micro-frontend might look like this: export interface MountContext { basePath: string; // e.g. "/checkout" locale: string; // e.g. "de-DE" getAccessToken(): Promise; navigate(to: string): void; // ask the shell to change route flags: Readonly>; } export interface MicroFrontend { mount(el: HTMLElement, ctx: MountContext): void | Promise; unmount(el: HTMLElement): void | Promise; } The contract says nothing about frameworks. React, Vue, Angular, Svelte or plain DOM code can all satisfy it, which is exactly what makes multi-framework composition possible later. It gives the micro-frontend what it needs from the platform through functions rather than shared mutable objects, so the shell can change how it obtains a token or performs navigation without breaking anyone. And it makes the lifecycle explicit: anything a micro-frontend sets up in mount it must tear down in unmount, including event listeners, timers, subscriptions and anything it attached to window. A worked example: an insurance portal Consider an insurer that runs a customer portal where policyholders view their policies, make claims, pay premiums, update personal details and read documents. Five teams are involved: policies, claims, billing, customer profile and a documents team that also handles correspondence. The first proposal from an enthusiastic architect is to put every card on the dashboard into its own micro-frontend: a policies summary card, an open claims card, a next payment card, a profile completeness card and a recent documents card, each from the owning team, all on the landing page. This proposal deserves scrutiny. The dashboard would carry five micro-frontends, each with its own loading behaviour, error states and data fetching. Every visual change to the dashboard layout would require five teams to adjust their cards. The landing page, the most visited screen in the portal, would have the most complex composition and the most failure modes. A better design starts at the route level. Each team owns its section: /policies, /claims, /billing, /profile, /documents. Within its section, a team can build whatever screens it likes without coordinating. The dashboard is then treated as its own small product, owned by one team, which reads summary data from the other teams' APIs and renders it in a consistent style. If a card on the dashboard needs a capability that changes often and is genuinely owned elsewhere, such as the claims status tracker, that one card can be a fragment from the claims team, with a written contract about its size and inputs. The rest are plain components fed by APIs. The second design has fewer moving parts on the busiest page, keeps most changes within one team and still allows a fragment where it earns its place. It also shows the general rule: start coarse, at the route level, and split further only when a specific, recurring coordination problem justifies it. Hard cases and second thoughts Route-level boundaries settle most of a product, but a few surfaces resist clean ownership, and any boundary scheme invites the charge that it has merely spread a monolith across repositories. Both deserve a direct answer, as does the question of how to tell when a boundary has gone wrong. Pages that belong to everyone Route-level boundaries handle most of a product cleanly, but every product has a few surfaces that seem to belong to everyone: the home page, the global header, site-wide search and the navigation menu. These are where boundary disputes concentrate, and leaving them unresolved is one of the most common ways a micro-frontend system slides back into coordination. The global header is the clearest case. It typically contains the logo, primary navigation, a search box, a basket indicator, an account menu and perhaps a notifications bell. Each of those elements maps to a different capability. If the shell team owns the header outright, it becomes a bottleneck for every team that wants a change to its part of it. If each capability team contributes a fragment to the header, the most visible strip of the product becomes the most fragmented, with five loading states and five failure modes in forty pixels of height. The arrangement that works best in practice gives the header to one owner, often a navigation or platform team, and treats the elements other teams care about as small, well-specified slots. The basket team provides a badge component through a narrow contract: it receives nothing but a container and emits nothing but a click, and it fetches its own count. The account team provides the menu contents as data, a list of labels and URLs, rather than as code. The header owner controls layout, spacing, responsive behaviour and accessibility, which are the things that must be consistent, while the teams control the facts that change with their domains. The home page is a different problem. It is usually a merchandising or editorial surface that gathers content from across the business, and its layout changes with campaigns and seasons. Giving it to a content or marketing team, which assembles it from data supplied by other teams' APIs, tends to work better than composing it from many teams' fragments. A home page built from fragments forces every campaign change to be negotiated with several teams, and it puts the most performance-sensitive page on the site at the mercy of the slowest fragment. Search crosses boundaries in a third way. The search box appears everywhere, the results page is a destination, and the suggestions dropdown sits between them. A sensible split gives the results page and the suggestion service to the discovery team, and has the search box in the header do nothing more than navigate to the results route with the query in the URL, perhaps showing suggestions fetched from the discovery team's API. The header owner then needs no knowledge of how search works, and the discovery team can change ranking, facets and result layouts freely. The common pattern across all three is that cross-cutting surfaces should have a single owner responsible for their layout and experience, and that other teams should contribute data or tightly specified slots rather than open-ended regions. Shared ownership of a surface is, in practice, no ownership, and the surface degrades. An objection: are we just building a distributed monolith? A reasonable sceptic will say that all these contracts, slots and shared packages amount to a monolith spread across repositories. The objection deserves a direct answer, because it is sometimes true. The test is not whether the parts depend on each other. They always will, because they make up one product. The test is whether the dependencies are few, explicit and stable enough that most changes do not cross them. A system where the mount contract changes once a year, the event schemas change once a quarter and the shell changes when a new micro-frontend is added, while each team ships several times a week without coordinating, is not a distributed monolith, whatever its diagram looks like. A system where half of all features need a coordinated release is one, however many repositories it has. Measuring the proportion of releases that required another team's change, which Chapter 10 recommends, answers the sceptic with data rather than argument. Revisiting boundaries Boundaries are not permanent, but they are expensive to move, so it is worth knowing the signs that one is wrong. Frequent cross-team pull requests into the same micro-frontend suggest two teams' work has landed in one place. Coordinated releases, where two micro-frontends must ship together for a feature to work, suggest a boundary cuts through a feature. Growing event traffic between two micro-frontends suggests they belong together. And a shell that keeps acquiring logic suggests some capability has no natural owner and needs one. When a boundary must move, the cheapest way is usually to merge first and split again later along a better line, rather than to transfer pieces between running micro-frontends. Merging two micro-frontends is a code reorganisation within one team. Moving a piece between two teams' running systems requires both to change, deploy and coordinate, which is exactly the kind of work the architecture exists to minimise. Chapter 3: Composition Strategies Once the parts exist, something must assemble them into a page. There are only a handful of places where assembly can happen: at build time, on the server, at the network edge, or in the browser. Each place gives different guarantees about independence, performance and isolation, and each has failure modes that show up only in production. Most mature systems end up combining two or three of them, so the useful question is less "which one" than "which one for which part of the page". Composition before the browser Two strategies assemble the page before any of it reaches the user's browser: one at build time, one when the page is requested. They sit at opposite ends of the independence scale. Build-time composition The simplest approach publishes each micro-frontend as a package, typically to a private npm registry, and has the shell install them as dependencies. The shell's build compiles everything into one bundle. Each team works in its own repository, with its own tests and its own versioning, and the shell pins a version of each package. Build-time composition has real virtues. The bundler sees all the code at once, so it can deduplicate shared libraries, tree-shake unused exports and produce an optimal bundle. Type checking works across boundaries. There is no runtime loading, so there is no runtime loading failure. Local development is ordinary. It also gives up the main thing micro-frontends are for. A team that publishes a new version of its package cannot ship it to users; it must wait for the shell to bump the dependency, rebuild and redeploy. Release coupling is back, now with an extra step. In addition, every package must agree on the versions of shared dependencies, because they end up in one bundle, so upgrade coupling returns too. For this reason, many practitioners do not count build-time composition as a micro-frontend architecture at all. It is better understood as a modular monolith with distributed source control. It remains the right choice for genuinely shared libraries, such as the design system's components, which should be versioned and consumed as packages rather than loaded at runtime. Server-side and edge composition The oldest runtime approach assembles HTML on the server. Each micro-frontend runs as a service that returns an HTML fragment. A composition layer requests the fragments needed for a page and stitches them into one document before sending it to the browser. The mechanisms predate the term micro-frontends by decades. Server Side Includes, supported by Apache and nginx, let a page template contain directives such as that the web server replaces with the response from another URL. Edge Side Includes, a markup language submitted to the W3C in 2001 by Akamai and others, does the same at a CDN or caching proxy using tags like , and is supported in varying degrees by Akamai, Fastly and the Varnish cache. More recent systems do the same job in application code. Zalando's Project Mosaic, published in 2016, included Tailor, a Node.js library that streams fragments into a layout as each becomes ready. Podium, from the Norwegian marketplace Finn.no, is a Node.js library for composing server-rendered fragments, which it calls podlets, into layouts. Edge compute platforms such as Cloudflare Workers, Fastly Compute and Vercel's edge functions let teams write the stitching logic as small programs that run close to the user. A simple nginx configuration shows the idea: location / { ssi on; proxy_pass http://shell-service; } location /fragments/reviews { proxy_pass http://reviews-service/fragment; } The shell service returns a layout containing an include directive for /fragments/reviews, and nginx fetches and inserts the reviews fragment before responding. Server-side composition produces fast first renders, because the browser receives complete HTML. It is friendly to search engines and to users on slow devices. Each fragment can be cached independently with its own lifetime, so a rarely changing footer can be cached for hours while a stock indicator is fetched every request. Fragments can come from services written in any language. Its weaknesses are on the client. Server-side composition delivers HTML, but most modern interfaces also need JavaScript for interactivity. Each fragment's scripts must still be loaded and initialised in the browser, and if fragments use client-side frameworks, the page must hydrate several independently built applications, which brings back the client-side problems of shared dependencies and isolation. Server-side composition also makes the slowest fragment a risk to the whole page: a robust implementation needs timeouts, fallbacks and streaming, so that a slow recommendations service does not hold the entire response. Tailor's design, which streams the layout and fills fragments as they arrive, was a direct response to this. Client-side composition Client-side composition assembles the page in the browser. The shell loads, decides which micro-frontends are needed, fetches their code and mounts them into the document. This is the most common approach for applications that behave like single-page applications, and it comes in several varieties. Iframes An iframe embeds one complete document inside another. It is the strongest isolation the browser offers without opening a new window: separate JavaScript global scope, separate CSS, separate DOM, and, if served from a different origin, the protection of the same-origin policy. A crash, a memory leak or a stylesheet war inside an iframe cannot reach the host page. Communication goes through window.postMessage, which forces an explicit, serialisable message protocol, a constraint that turns out to be healthy. Iframes have a bad reputation, partly earned. They do not size themselves to their content, so the host must be told the height through messages, and the page can jump as content changes. Deep linking, browser history and focus management need careful handling. Each iframe loads its own copy of every library. Modals and overlays cannot extend beyond the iframe's rectangle, which breaks common patterns such as a dropdown that overflows its container. Accessibility suffers if the frame's title and focus order are neglected. Yet iframes remain the right answer in specific cases: embedding third-party or untrusted content, integrating legacy applications that cannot be modified, and enterprise portals where many internal tools must coexist and isolation matters more than seamless appearance. SAP's open-source Luigi framework, for instance, composes micro-frontends in iframes for exactly these administrative-portal scenarios. Application routers: single-spa and its relatives single-spa, first released in 2015 by Joel Denning, was the first widely used framework built specifically for client-side micro-frontends. Its model is small and influential. Each micro-frontend registers as an application with three lifecycle functions, bootstrap, mount and unmount, and an activity function that says when it should be active, usually based on the URL. single-spa listens to route changes and calls the lifecycle functions so that the right applications are mounted at the right time. import { registerApplication, start } from 'single-spa'; registerApplication({ name: '@acme/checkout', app: () => import('@acme/checkout'), activeWhen: ['/checkout'], }); start(); single-spa says nothing about how the code is fetched; the app function can use dynamic import() with import maps, SystemJS, Module Federation or anything else that returns the lifecycle object. Helper libraries, such as single-spa-react, single-spa-vue and single-spa-angular, wrap a framework's root component in the three lifecycle functions. The stable release line as of autumn 2026 is version 6, released in late 2023, with a version 7 that has been in beta since 2024. The Chinese ecosystem built heavily on the same ideas: qiankun, from Alibaba's Ant Group, extends single-spa with HTML entry loading and a JavaScript sandbox that limits what micro-frontends can do to global variables. The router approach is excellent for page-level boundaries. It handles the hand-over between applications on navigation cleanly and gives a well-understood lifecycle. It is less natural for many fragments on one page, though single-spa's "parcels" address this. Web Components Web Components are a set of browser standards for defining custom HTML elements. A micro-frontend can be exposed as a custom element, such as , and the host page uses it like any other tag. The browser handles the lifecycle: the element's connectedCallback runs when it is inserted into the document and disconnectedCallback when it is removed, which maps neatly onto mount and unmount. Inputs travel as attributes or properties, outputs as DOM events. Optionally, the element can use Shadow DOM to encapsulate its markup and styles from the surrounding page, a topic Chapter 6 treats in depth. Web Components are attractive as a neutral integration contract. Every framework can render a custom element and listen to its events, and every major framework can produce one, as Chapter 7 shows. They are not a loading mechanism; something still has to fetch the script that defines the element. They are an interface, and a very good one for fragments. Module Federation and import maps The last family shares code between separately built applications at runtime. With Module Federation, introduced in webpack 5 in October 2020 and since implemented for Rspack, Vite and others, each application can expose modules and consume modules exposed by others, with the runtime negotiating which copy of a shared library to use. With import maps, a browser standard, the page declares a mapping from bare module names, such as react or @acme/checkout, to URLs, and native ES module imports resolve through it. Both make it possible to load a micro-frontend's code on demand and to share libraries between micro-frontends without bundling them together. Chapters 4 and 5 cover them in detail. How each strategy fails Composition strategies are usually compared by how they behave when everything works. In production, the more revealing comparison is how each behaves when one part does not. A micro-frontend may be slow, may throw an exception during start-up, may be deployed with a broken asset path, or may simply be unreachable because a CDN region is having a bad afternoon. With build-time composition, a broken part is usually caught by the shell's build or test suite, which is the strategy's main safety advantage. When something does slip through, it fails with the whole application: an exception during module initialisation in one package can stop the entire bundle from starting. There is no runtime boundary to contain it. With server-side or edge composition, the composition layer can contain failures if it is written to do so. Each fragment request needs a timeout, after which the layer inserts a fallback, such as an empty placeholder, a cached older version of the fragment, or a link to the full page. Without timeouts, one slow fragment holds the whole response, and the page's time to first byte becomes the maximum of all fragments' response times rather than something closer to the slowest critical one. ESI processors and libraries like Tailor support fallbacks and timeouts explicitly; hand-rolled stitching code often forgets them. With iframes, failure is naturally contained: a broken frame shows a blank rectangle or its own error page, and the host continues. The host should still detect the failure, for example by expecting a ready message within a set time, so it can show something more useful than an empty box. With client-side routers, Web Components and federation, containment depends entirely on the shell and the micro-frontends being written defensively. Every load of remote code is a network request that can fail, and every mount is a call into another team's code that can throw. The shell must wrap each in error handling that renders a fallback for that region and reports the failure with the micro-frontend's name and version. Unhandled promise rejections from a failed dynamic import, or an uncaught exception in a custom element's connectedCallback, can otherwise break navigation or leave the page half rendered. single-spa, for example, moves an application that throws during a lifecycle into a broken status and keeps the rest running, but only the shell can decide what the user sees in its place. There is also a performance failure mode particular to client-side composition: the loading waterfall. The browser downloads the shell, which runs and then requests the discovery document, which arrives and then tells the shell which remote entry to request, which arrives and then requests the chunks for the exposed module, which arrive and then request their own lazy chunks. Each step waits for the previous one. On a fast connection this costs a few hundred milliseconds; on a mobile network it can cost seconds. The cures are to flatten the chain: embed the discovery data in the initial HTML, emit or hints for the remote entries and critical chunks the current route needs, which federation manifests make possible, and render the first view on the server where performance matters most. Choosing among strategies The strategies differ along a few dimensions that matter in practice, summarised in Table 1. The independence column asks whether a team can release without anyone else rebuilding. The isolation column asks how well the browser keeps the parts from interfering with each other. The cost column notes the main price. Table 1. How the main composition strategies compare. Strategy Independent release Runtime isolation Initial render Main cost Build-time packages No None Fast Release and upgrade coupling Server or edge includes Yes None on client Fastest Client hydration and slow-fragment risk Iframes Yes Strong Slow Layout, history and duplication Router with lifecycles Yes By convention Moderate Global scope shared Web Components Yes Optional via Shadow DOM Moderate Needs a loader; styling edge cases Module Federation or import maps Yes By convention Moderate Version negotiation at runtime Two patterns come out of experience. First, page-level boundaries pair naturally with a client-side router or with server-side routing that sends each path to a different application. Second, fragments pair naturally with Web Components as the interface and federation or import maps as the loader, optionally rendered on the server first for performance. A worked example: combining strategies A hypothetical travel company's booking site illustrates a realistic combination. The marketing pages, destination guides and search entry page must render fast and rank well in search engines. The booking funnel, from flight selection to payment, is highly interactive. A partner hotels widget comes from a third party. The company composes its landing and content pages at the edge. An edge function fetches the page layout from the content team's service and includes the header and footer fragments owned by the navigation team, each cached separately. These pages ship very little JavaScript. When the user starts a search, the browser navigates to /book, where a client-side shell takes over. The shell uses single-spa to mount the flights, seats, extras and payment applications as the user moves through the funnel, each owned by a different team and each loaded through Module Federation so that React and the design system are downloaded once. The partner hotels widget, which the company does not control, is embedded in an iframe on the confirmation page, communicating with the host through a small postMessage protocol that passes the destination and dates in and a selected hotel out. Each strategy is used where its strengths matter and its costs are tolerable. The rest of this booklet is mostly concerned with the client-side portion of such a system, because that is where the hardest engineering problems of shared code, shared styles and mixed frameworks arise. Chapter 4: Module Federation in Webpack Module Federation is the mechanism that made runtime code sharing between independently built JavaScript applications practical. Zack Jackson proposed it, and it shipped as part of webpack 5 in October 2020. Before it, teams loading micro-frontends at runtime either bundled every dependency into every micro-frontend, paying for duplicate copies of React or Angular, or relied on global variables and script tags, which made version management fragile and manual. Federation turned sharing into something the bundler understood. Understanding how it works underneath is the difference between configuring it confidently and copying configuration until the errors stop. The core idea: containers In ordinary webpack builds, every module is known at build time. The bundler resolves every import, assigns each module an identifier and emits a runtime that can find modules by identifier within the bundles it produced. Nothing outside the build can participate. Module Federation adds the notion of a container. A build configured with the ModuleFederationPlugin can emit a special entry file, conventionally named remoteEntry.js, which defines a container object. The container exposes two functions. get(name) returns a promise of a factory for one of the modules the build has chosen to expose. init(shareScope) lets the container take part in a shared pool of dependencies. An application that publishes a container is called a remote. An application that loads containers and imports modules from them is a host. The same build can be both. When the host's code executes import('checkout/CartSummary'), the webpack runtime recognises checkout as a remote, ensures the remote's remoteEntry.js has been loaded, initialises its container with the shared scope, calls get('./CartSummary'), waits for the chunks behind that module to download, and finally calls the factory to obtain the module's exports. To the host's code it looks like any other lazy import. Underneath, two separately compiled applications have negotiated a module exchange in the user's browser. Configuring a remote and a host A remote declares a name, a filename for its entry, the modules it exposes and the dependencies it is willing to share: // webpack.config.js in the checkout application const { ModuleFederationPlugin } = require('webpack').container; module.exports = { output: { publicPath: 'auto', uniqueName: 'checkout' }, plugins: [ new ModuleFederationPlugin({ name: 'checkout', filename: 'remoteEntry.js', exposes: { './CartSummary': './src/CartSummary.tsx', './mount': './src/mount.ts', }, shared: { react: { singleton: true, requiredVersion: '^19.0.0' }, 'react-dom': { singleton: true, requiredVersion: '^19.0.0' }, }, }), ], }; The keys in exposes are the public names; the values are internal paths. Only exposed modules are reachable from outside, which makes exposes the remote's public API. Treat it as such: renaming an exposed path is a breaking change for every host. Setting publicPath to 'auto' tells webpack to infer the base URL for chunks from the location of the script that loaded them, so the remote's chunks are fetched from wherever its remoteEntry.js was served. Forgetting this is a common source of 404s when a remote's chunks are requested relative to the host's origin instead of the remote's. The host declares where to find its remotes and the same shared dependencies: // webpack.config.js in the shell new ModuleFederationPlugin({ name: 'shell', remotes: { checkout: 'checkout@https://cdn.acme.example/checkout/remoteEntry.js', }, shared: { react: { singleton: true, requiredVersion: '^19.0.0' }, 'react-dom': { singleton: true, requiredVersion: '^19.0.0' }, }, }); The remote string has the form globalName@url. In this static form, the remote's URL is baked into the host's build, which is fine for a demonstration and a liability in production: changing the location of a remote requires rebuilding the host. Production systems almost always load remotes dynamically, from a manifest or discovery service, using the runtime API covered in Chapter 5. How sharing works: the share scope The shared section is where most of federation's power and most of its difficulty live. Each build that participates contributes its copies of shared dependencies to a share scope, a registry kept in memory in the browser and keyed by package name and version. When a module needs a shared dependency, the runtime looks in the share scope and picks a version according to the rules each consumer declared. Several options control the rules: • requiredVersion is a semver range the consumer accepts. If not set, webpack reads it from the consumer's package.json. • singleton: true says there must be only one copy of this package in the page. The runtime picks the highest available version, and if that version does not satisfy a consumer's requiredVersion, it logs a warning and uses it anyway. • strictVersion: true, combined with singleton, turns that warning into a runtime error, so a consumer refuses to run against an incompatible version rather than silently misbehaving. • eager: true puts the shared module directly into the initial chunk instead of loading it asynchronously. This avoids an extra request but defeats the negotiation for that package, because the eager copy is present before anyone can choose otherwise. • import: false says this build will not provide its own copy at all and relies on someone else to supply it. Libraries that keep global state or depend on a single instance must be singletons. React is the canonical case: two copies of React in one tree produce the infamous "invalid hook call" error, because hooks rely on a dispatcher held in module state, and a component rendered by one copy cannot use hooks from another. The same applies to anything that uses React context across a boundary, to routers, to state libraries such as Redux when a store is shared, to Angular's core packages and to styled-components, which tracks injected styles in module state. Libraries without global state, such as date utilities or lodash, do not need to be singletons. Sharing them saves bytes when versions line up and falls back gracefully to separate copies when they do not. Many teams also choose not to share such libraries at all, accepting some duplication in exchange for fewer negotiations. The bootstrap pattern and eager consumption The first error nearly every team meets with federation reads something like: "Shared module is not available for eager consumption". It arises because webpack must initialise the share scope before any code uses a shared module, and initialisation is asynchronous: remotes may need to be fetched and their versions considered. If the application's entry file imports React synchronously, the import is evaluated before negotiation has finished. The standard fix is to make the entry file a one-line asynchronous boundary: // src/index.js import('./bootstrap'); // src/bootstrap.js import { createRoot } from 'react-dom/client'; import App from './App'; createRoot(document.getElementById('root')).render(); The dynamic import of bootstrap gives webpack a point at which to initialise the share scope, load what it needs and only then evaluate the application. Marking React as eager also removes the error, but at the cost of always shipping that build's copy of React in the initial chunk, which undermines sharing. The bootstrap pattern is almost always the better choice. Version negotiation in practice The share scope's behaviour is easiest to understand through a concrete scenario. Suppose the shell is built against React 19.1, the checkout remote against React 19.2 and an older account remote against React 18.3, all declaring React as a singleton with requiredVersion taken from their own package.json. When the page loads, the shell and both remotes register their copies of React in the share scope. Because React is a singleton, the runtime selects one version, the highest registered, 19.2. The shell's range of ^19.1.0 is satisfied. Checkout's range is satisfied. The account remote's range of ^18.3.0 is not. Without strictVersion, the runtime logs a warning and gives the account remote React 19.2. Whether that works depends on whether the account code uses any APIs removed or changed in 19, and the answer may only become clear when a user reaches a particular screen. With strictVersion, the account remote fails to load, cleanly and immediately, and the shell can show an error state. Neither outcome is good, and that is the lesson. Federation can negotiate versions, but it cannot make incompatible code compatible. The organisation must decide on a policy for singletons, typically that all micro-frontends stay within one major version of each singleton framework, and must give teams a migration window when a new major version arrives. A common approach is for the platform team to own an "allowed versions" document or package, with CI checks that fail a build whose singleton ranges fall outside it. Upgrades then become a scheduled, platform-coordinated event for singletons and a team-level decision for everything else. Some coupling remains, but it is explicit, limited to a short list of packages and managed by someone. It also explains why a page can end up loading a version of a library that nobody tested with. Highest-version-wins means that the newest remote on the page effectively decides the singleton version for everyone. A remote that ships a minor upgrade to React therefore upgrades every other micro-frontend on the page at runtime, without their teams building or testing anything. Minor and patch releases rarely break, which is why the model works most of the time, but it is worth stating plainly: under federation, one team's dependency bump is every team's dependency bump. Expose a mount function, not components The examples above expose a React component, CartSummary, alongside a mount function. In practice the two represent different philosophies, and the choice between them shapes how coupled the teams become. Exposing a framework component means the host renders it inside its own component tree. That is convenient: props flow in naturally, context from the host is available, and the component participates in the host's rendering, suspense and error boundaries. It also means the host and remote must share the same framework, the same major version and very often the same versions of the router, the state library and the design system, because the component expects to find them in its surrounding tree. A remote component can read any context the host provides, which invites hidden dependencies: a component that quietly relies on the host's theme provider or query client works in one host and breaks in another. Exposing a mount function means the host gives the remote a DOM element and a context object, and the remote renders its own independent tree inside it. The remote brings its own providers, its own router instance scoped to its base path and its own error boundaries. The host knows nothing about the remote's framework, so the remote can upgrade, or even change frameworks, without the host noticing. Communication is limited to what the contract carries, which makes it explicit. The cost of the mount approach is that the two trees are separate. A React host cannot pass a React context into a remote's tree mounted with its own root, and the remote's rendering does not participate in the host's suspense boundaries. For page-level and large fragment micro-frontends, that separation is exactly what is wanted. For small widgets tightly integrated into a host's layout, exposing a component can be justified, provided both sides accept the shared-framework commitment and keep the component's dependencies on host context explicit in its props. A rule of thumb that has served many teams: expose one mount function per micro-frontend as its public interface, and expose components only for widgets designed to be embedded, owned and documented as such. Never expose a team's internal components merely because federation makes it possible. Once another team imports them, they are public APIs whether or not anyone intended it. Dynamic remotes before the 2.0 runtime The static remotes configuration shown earlier ties a host's build to its remotes' URLs. Teams on the first-generation plugin worked around this in two ways, both still found in older codebases. The first used a promise-based remote: instead of a name@url string, the remote value was a string of JavaScript that the plugin embedded in the host, which at runtime fetched the remote's URL from configuration, injected a script tag for the entry file and resolved to the container object. It worked, but the code lived inside a string in the webpack configuration, invisible to linters and type checkers. The second bypassed the remotes option entirely and used webpack's low-level runtime functions directly. The host loaded the entry script itself, called __webpack_init_sharing__('default') to initialise its own share scope, called the container's init with __webpack_share_scopes__.default, and then called container.get('./mount') to obtain the module factory. This made any remote loadable at runtime from any URL, and it is still how many internal "dynamic federation" utilities are implemented. Both approaches are superseded by the 2.0 runtime's registerRemotes and loadRemote, which do the same work with a supported, documented interface. Understanding them remains useful because they show what the runtime does on every remote load: fetch an entry, initialise sharing, negotiate versions, retrieve a factory, evaluate a module. Common failure modes Teams running federation in production report the same handful of problems. Remote unavailable. If a remote's entry file cannot be fetched, because of a network failure, a CDN outage or a bad deployment, the dynamic import rejects. Without handling, the error propagates and may break the page. Every remote import should be wrapped in an error boundary or equivalent, with a fallback state. In React, this means a Suspense boundary for loading and an error boundary for failure around each federated component. Stale entry files. If remoteEntry.js is cached aggressively by the browser or CDN while its chunks are not, or the reverse, a host can load an entry that points to chunks which no longer exist. The entry file should be served with short cache lifetimes or a cache-busting URL, while the content-hashed chunks it references can be cached for a year. Duplicate globals. Two remotes built with the same output.uniqueName, or with none, can collide on webpack's global chunk-loading variable. Each build needs a unique name. CSS order dependence. Federation loads JavaScript, but if exposed modules import CSS, the order in which styles are injected depends on load order, which depends on user navigation. A rule that wins when the user arrives via the home page can lose when they arrive via a deep link. Chapter 6 deals with this. Silent singleton mismatch. As described above, a singleton version mismatch without strictVersion produces only a console warning. Production monitoring should capture those warnings, because they are early indicators of real bugs. Type drift. Federated imports such as checkout/CartSummary are invisible to TypeScript by default. Teams either write declaration files by hand, which go stale, or adopt the type generation that Module Federation 2.0 provides, described in the next chapter. Federation as a contract, not a convenience The most important thing to understand about Module Federation is what kind of coupling it creates. An exposed module is a function call across a team boundary, resolved at runtime. Its signature is an API. Its shared dependencies are a joint commitment about the runtime environment. Changing either can break another team's code in production with no build failure anywhere. Used with discipline, federation is an excellent tool. The exposed surface is small: ideally one mount function per micro-frontend, conforming to a contract like the one in Chapter 2, plus perhaps a few widgets. Shared dependencies are few and governed: the framework, its router if shared, the design system runtime, and little else. Failure is handled at every remote boundary. Used without discipline, federation becomes a way for teams to import each other's internal components freely, which recreates the monolith's coupling with none of its compile-time safety. The configuration is the same in both cases. The difference is entirely in what the organisation chooses to expose and share. Chapter 5: The Second Generation of Federation The webpack 5 plugin proved that runtime module sharing could work at scale, and it also exposed what was missing. Remote locations were baked into builds. Types did not cross the boundary. Debugging the share scope meant reading webpack's runtime in the browser's developer tools. And the whole mechanism was tied to one bundler, at a time when the JavaScript ecosystem was moving toward faster, Rust-based and native-ESM tools. Between 2023 and 2026 a second generation of federation emerged to address these gaps: a bundler-independent runtime and manifest format, a fast webpack-compatible bundler, federation for Vite, a standards-based variant for Angular, and a growing case for doing without a federation runtime altogether by using the browser's own import maps. Module Federation 2.0: runtime, manifest and types The Module Federation project, now maintained as an open-source organisation with significant engineering from ByteDance's web infrastructure group alongside Zack Jackson, began publishing its second generation in 2024 under the package @module-federation/enhanced and reached its 2.0 release milestone in April 2026. Three ideas define it. The first is a standalone runtime. In the original plugin, federation logic was generated into each build by webpack. In 2.0 it lives in a separate package, @module-federation/runtime, that any bundler integration can use and that applications can call directly. This makes it possible to register remotes at runtime from configuration fetched from a server, rather than hard-coding URLs at build time: import { createInstance } from '@module-federation/enhanced/runtime'; const mf = createInstance({ name: 'shell', remotes: [ { name: 'checkout', entry: 'https://cdn.acme.example/checkout/mf-manifest.json' }, ], }); const { mount } = await mf.loadRemote<{ mount: MountFn }>('checkout/mount'); Applications built with a federation plugin can instead obtain the default instance with getInstance() and call registerRemotes on it. Either way, the location of each remote is now data, which is what makes independent deployment workable: a team publishes a new version and updates a record in a discovery service, and the shell picks it up without rebuilding. The runtime also accepts plugins, objects that hook into its lifecycle. Hooks such as beforeRequest let a plugin rewrite a remote's URL, for instance to route a tester's browser to a preview deployment. Hooks around shared module loading let a plugin control version selection. An error hook, errorLoadRemote, lets a plugin supply a fallback module when a remote fails to load, which centralises the failure handling that otherwise has to be repeated at every import site. The second idea is the manifest. Alongside or instead of remoteEntry.js, a 2.0 build emits mf-manifest.json, a machine-readable description of the remote: its name, its exposed modules, the chunks and stylesheets each one needs, its shared dependencies with versions, and the location of its type definitions. Because the manifest lists the assets each exposed module requires, the runtime and server-side renderers can preload them in parallel instead of discovering them one request at a time. Tooling can read the manifest to show which versions of which remotes a page uses, and to detect mismatched shared dependencies before deployment. The third idea is type sharing. With the dts option enabled, a remote's build generates TypeScript declarations for its exposed modules and publishes them with its other assets. A host's build downloads them, so that import('checkout/mount') is type-checked against the version actually deployed. During development the types refresh as the remote changes. This closes one of the most frequent sources of runtime breakage in the first generation, where an exposed component's props changed and the host only found out in production. The 2.0 release added further features aimed at larger installations. Shared dependencies can now be tree-shaken, so a shared library contributes only the exports some consumer actually uses rather than its whole surface. Server-side rendering is supported through integration with the Modern.js framework, along with data-fetching helpers for both server and client rendering. A side-effect scanner can examine a remote for changes it makes to global variables, event listeners and styles, which are exactly the cross-boundary effects that cause hard-to-trace bugs. A Chrome developer tools extension shows the live share scope, the loaded remotes and their versions, and allows a developer to point one remote at a local development server while the rest of the page uses production. The project positions these features as incremental: a team on the first-generation plugin can adopt the enhanced plugin and enable manifest, types and runtime plugins one at a time. Beyond webpack: Rspack, Vite and Angular The 2.0 runtime made federation independent of webpack in principle. Three ecosystems turned that into practice in different ways: one by reimplementing webpack in Rust, one by porting the protocol to a different bundler, and one by rebuilding the idea on browser standards. Rspack: the webpack-compatible fast path Rspack is a JavaScript bundler written in Rust, created at ByteDance and released as open source in 2023, with its 1.0 release in August 2024 and 2.0 in April 2026. Its defining design decision is compatibility with webpack's configuration format and most of its plugin and loader ecosystem, so that an existing webpack project can often move to Rspack by changing its imports and a few options rather than rewriting its build. For micro-frontends this matters because many enterprise frontends are large webpack projects whose build times have become a tax on every team. Module Federation support is built into Rspack and uses the same 2.0 runtime and manifest as the webpack integration. A remote built with Rspack and a host built with webpack can federate with each other, which makes migration incremental: teams can move one micro-frontend at a time. Rspack 2.0 brought tree shaking of shared dependencies to its federation implementation, published its core packages as pure ES modules and set Node.js 20 as its baseline. The Rspack team reports production builds of its own benchmark roughly ten percent faster than in 1.7, with persistent caching roughly halving rebuild times. Larger gains, often several times faster, are commonly reported by teams moving from webpack, though they depend heavily on the project and on how much of the build is spent in JavaScript loaders that Rspack cannot accelerate. Rsbuild, a higher-level build tool layered on Rspack in the way that Vite is layered on its bundler, provides a federation plugin with sensible defaults, and the Modern.js framework builds on both. Nx, the monorepo tool, offers generators for federated React and Angular applications that can target either webpack or Rspack. Federation for Vite Vite, created by Evan You and now developed by a core team largely employed by his company VoidZero, became the most popular build tool for new frontend projects during the early 2020s. Its development server serves native ES modules directly to the browser and its production build historically used Rollup. Federation did not fit naturally, because webpack's federation runtime assumes webpack's module system. The first widely used solution was @originjs/vite-plugin-federation, a community plugin that implemented a federation-like protocol for Vite. It worked within Vite applications and could interoperate with webpack remotes in some configurations, but had limits, notably around the development server. The Module Federation organisation later took over a newer plugin, @module-federation/vite, which implements the 2.0 runtime and manifest for Vite so that Vite applications can federate with webpack and Rspack applications through the same protocol. Its configuration mirrors the webpack plugin, with name, filename, exposes, remotes and shared. Its documentation notes that it manages code splitting itself and rejects manual chunking configuration, because the federation bootstrap depends on a particular chunk order. Vite 8, released in March 2026, replaced both esbuild and Rollup with Rolldown, a single Rust bundler compatible with Rollup's plugin interface. The Vite team cites production builds several times faster in real projects, such as Linear's falling from 46 seconds to 6. Rolldown's stated goals include first-class federation support, which would bring Vite's federation closer to the integrated experience Rspack offers. As of autumn 2026, teams choosing Vite for federated micro-frontends should verify the plugin's behaviour against their specific needs, particularly hot module replacement across remotes and server-side rendering, which have historically been the least mature areas. Native Federation for Angular Angular took a different path. From version 17 the Angular CLI's default build system uses esbuild instead of webpack, which left Angular teams who relied on webpack Module Federation with a choice between staying on an older build system and losing federation. Manfred Steyer and the Angular Architects team created Native Federation to resolve this. It keeps Module Federation's mental model, with remotes, exposed modules and shared dependencies, but implements it on web standards: shared packages and exposed modules are compiled to ordinary ES modules, and the browser resolves them through an import map that the Native Federation runtime generates at start-up. A remote's configuration lives in federation.config.js: const { withNativeFederation, shareAll } = require('@angular-architects/native-federation/config'); module.exports = withNativeFederation({ name: 'orders', exposes: { './routes': './src/app/orders.routes.ts', }, shared: { ...shareAll({ singleton: true, strictVersion: true, requiredVersion: 'auto' }), }, }); Each remote publishes a remoteEntry.json describing its exposed modules and shared packages. The host reads a manifest mapping remote names to those files, initialises federation before bootstrapping Angular, and loads remote modules on demand, typically as lazy routes: // main.ts in the shell import { initFederation } from '@angular-architects/native-federation'; initFederation('federation.manifest.json') .then(() => import('./bootstrap')) .catch(err => console.error(err)); // a route in the shell { path: 'orders', loadChildren: () => loadRemoteModule('orders', './routes').then(m => m.ORDER_ROUTES), } The same bootstrap pattern seen in Chapter 4 appears here for the same reason: shared dependencies must be resolved before any application code runs. Native Federation's version numbers follow Angular's, so the release line matches the Angular major version in use. Recent releases have added automatic chunking of shared dependencies, a scan that skips compiling packages the code never imports, server-side rendering and incremental hydration, and automatic reloading of the shell in development when a remote rebuilds. The core is framework-agnostic and published separately under @softarc/native-federation, so the approach can be used outside Angular, though Angular is where most of its users are. Import maps without a federation runtime Native Federation points toward a broader observation: once the browser can resolve bare module names to URLs by itself, a great deal of what a federation runtime does can be done by a static file. An import map is a JSON object in a Any module that executes import { mount } from '@acme/checkout' receives the file named in the map. Shared libraries are shared because everyone imports the same URL. Deploying a new version of checkout means publishing its files and updating one line of the map, which the shell can generate on the server from a deployment registry. This is the arrangement single-spa's documentation has long recommended, originally with the SystemJS loader standing in for native support. Import maps became available in Chrome and Edge in 2021, in Firefox in late 2022 and in Safari in 2023, so they are now usable everywhere that matters for most enterprises. A significant limitation was that a page could have only one import map, and it had to appear before any module loaded, which forced the shell to know every micro-frontend's dependencies up front. Support for multiple import maps, which allows maps to be added after start-up provided they do not redefine names already resolved, shipped in Chrome 133 and Safari 18.4 in 2025; as of autumn 2026 Firefox has it only behind a setting. The es-module-shims library by Guy Bedford polyfills this and other import map features where they are missing. What import maps do not provide is version negotiation. A map says exactly one URL per name; it does not choose the highest compatible version among those offered by several applications. That decision moves to whoever generates the map, usually a server-side step at deployment or request time, which many teams find easier to reason about than negotiation in the browser. The map also offers scopes, which let a specific micro-frontend resolve a name to a different URL, for example during a migration where one area still needs an older library. Choosing a mechanism The options differ in how tightly they bind teams to a build tool and in where version decisions are made. Table 2 summarises them. Table 2. Runtime code-sharing mechanisms compared. Mechanism Build tools Where versions are chosen Type sharing Best fit MF 2.0 on webpack webpack 5 Browser share scope Built in Existing webpack estates MF 2.0 on Rspack Rspack, Rsbuild Browser share scope Built in Large builds needing speed MF 2.0 on Vite Vite plugin Browser share scope Built in Vite-based teams Native Federation esbuild, Angular CLI Generated import map Via packages Angular on esbuild Plain import maps Any ESM output Server generating the map Via packages Simple, standards-first setups Three practical rules follow. Use one mechanism across the organisation where possible; mixing them is possible through the shared 2.0 protocol, but each additional mechanism is another thing for the platform team to support. Prefer the mechanism that fits the dominant build tool rather than changing build tools to fit a mechanism. And keep the shared surface small regardless of the mechanism, because none of them removes the obligation to agree on singleton versions and to treat exposed modules as public APIs. A worked example: moving a federated estate to the second generation Consider an organisation with a shell and eight remotes, all built with webpack 5 and the original federation plugin. Remote URLs are baked into the shell's configuration, so each new remote or relocation requires a shell release. Types for federated imports are hand-written declaration files, frequently stale. Production builds take between four and nine minutes per application, and the platform team wants to move to Rspack. A sensible sequence changes one variable at a time. First, the platform team replaces webpack.container.ModuleFederationPlugin with the plugin from @module-federation/enhanced in every application, leaving all other configuration as it was. The enhanced plugin accepts the same options, so the change is mechanical, and the applications continue to interoperate with each other during the rollout because both generations speak the same container protocol. This step adds the manifest to every remote's output without anyone yet relying on it. Second, the shell moves from static remotes to runtime registration. A small discovery service returns the manifest URL for each remote, and the shell calls registerRemotes at start-up. From this point, remotes can deploy and relocate without a shell release, which is the largest single gain in team independence in the whole migration. The team adds a runtime plugin implementing errorLoadRemote to return a standard fallback module for any remote that fails, replacing eight slightly different hand-written fallbacks. Third, teams enable dts one remote at a time. Each host deletes its hand-written declarations for that remote and consumes the generated types. The first build after the switch typically reveals several mismatches that had been silently wrong in production, which is a useful early return on the effort. Fourth, applications move from webpack to Rspack one at a time, starting with the remote whose build is slowest. Because Rspack's federation uses the same runtime and manifest, a remote built with Rspack federates with a host built with webpack without changes to either side's federation configuration. Loaders and plugins with no Rspack equivalent are the main source of friction, and a short audit of each application's build configuration before migrating it avoids surprises. At no point in this sequence does the system stop working, and at no point must all teams change at once. That property, incremental adoption with interoperability at every step, is the most important practical feature of the second generation and the reason organisations with large first-generation estates can adopt it at all. Chapter 6: Isolating CSS JavaScript isolation gets most of the attention in micro-frontend discussions, but CSS causes more day-to-day breakage. JavaScript at least has modules: a variable declared inside one micro-frontend's module is invisible to the others unless it is deliberately attached to window. CSS has no such default. Every rule in every stylesheet loaded on the page applies to every element that matches its selector, regardless of which team wrote the rule or which team rendered the element. A page assembled from five independently built applications is five teams writing into one global namespace without seeing each other's work. How collisions happen Collisions take a few recurring forms, and naming them makes the defences easier to evaluate. Selector collisions. Two teams use the same class name for different things. The checkout team styles .card with a border and padding; the recommendations team styles .card as a product tile with a shadow. On a page that loads both, every card gets both sets of rules, and which property wins depends on specificity and source order. Leaking element and global selectors. A team writes button { border-radius: 0; } or h2 { margin-top: 2rem; } in its stylesheet, intending to affect only its own area. It affects every button and heading on the page. Resets and base styles are the worst offenders: two micro-frontends that each include their own CSS reset will fight over every element. Order dependence. When two rules have equal specificity, the later one wins. In a client-composed page, the order in which stylesheets are inserted depends on the order in which micro-frontends load, which depends on how the user arrived. A style that looks right on a fresh page load can be wrong after client-side navigation, or the other way round. These bugs are notoriously hard to reproduce. Leftover styles. When a micro-frontend is unmounted, its injected

Download the Book (PDF): Introduction Most advice written for university students assumes a body that shows up. It assumes that if you sleep reasonably, eat something green now and then, and keep a calendar, you will be able to attend the lecture, finish the problem set, and sit the exam on the day it is scheduled. The advice is not wrong; it is simply written for someone else. If you live with a chronic condition, whether that is Crohn's disease, type 1 diabetes, lupus, migraine, epilepsy, postural orthostatic tachycardia syndrome, myalgic encephalomyelitis, long COVID, a connective tissue disorder, or any of the hundreds of other conditions that come and go on their own schedule, you already know that the body does not always show up. Sometimes it arrives late. Sometimes it arrives for the morning and leaves at noon. Sometimes it does not arrive for a week. This book is for students in that position, and for the parents, advisers, and friends who want to understand it. It is a practical book, but it rests on a single argument, and it is worth stating plainly at the outset because everything else follows from it. Managing a chronic illness at college is a design problem, not a willpower problem. The students who do well are rarely the ones who push hardest through bad days. They are the ones who build a system in advance: a workload paced to what their body can reliably deliver across a whole semester, and a set of agreements with their institution, negotiated calmly before they are needed, that allow the semester to bend when the illness flares instead of breaking. A flare-up that arrives inside a well-designed system is an inconvenience. The same flare-up arriving in a semester built for a healthy body, with no accommodations on file and no professor aware of anything, becomes a crisis that the student must manage alone, while sick, at the worst possible moment. Why willpower fails It is worth dwelling on why the willpower approach is so tempting and why it so reliably goes wrong. Many students with chronic conditions have spent years proving that they can keep up. They were the child who did homework in a hospital bed, the teenager who hid symptoms from coaches, the applicant who wrote the admissions essay about resilience. Pushing through has been rewarded. It feels like character. The trouble is that pushing through spends a resource the student cannot see and cannot easily replenish. For many conditions, effort beyond a certain point does not simply produce tiredness that a night's sleep repairs. It produces a delayed payback: a flare, a crash, a week of lost function that arrives a day or two after the effort that caused it. The student who pulls an all-nighter to meet a deadline on Tuesday may lose Thursday, Friday, and the following Monday. Measured over the semester, the push was not heroic. It was expensive borrowing at a punishing rate of interest. Willpower also fails because it is private. A student who is quietly coping has not told anyone, which means nobody can help when coping stops working. The professor who would gladly have granted a two-day extension in week three learns about the illness in week twelve, when the student emails at midnight with four missing assignments and a hospital discharge letter. By then the options are narrower and the conversation is harder for everyone. What this book covers, and what it does not The chapters that follow move from understanding to arrangement to action. The first chapter examines why the ordinary structure of university life, with its bunched deadlines, irregular hours, and assumption of constant availability, is particularly hard on bodies that fluctuate. The next three chapters deal with communication: the legal and institutional framework that gives students with chronic illnesses the right to reasonable accommodations, the conversation with the disability services office where those accommodations are set up, and the separate, often more delicate conversation with individual professors. The following two chapters turn to pacing: first as a daily practice of managing energy, then as a way of designing an entire semester, from how many credits to take to where to put the hardest course. The seventh chapter is a protocol for flare-ups, covering what to do before, during, and after one, including the formal options of incompletes and medical withdrawal. The last chapter widens the frame to the parts of college life outside the classroom that make or break a semester: housing, food, healthcare continuity, friendships, and the question of identity that many students with chronic illness wrestle with quietly. Throughout, the book uses illustrative scenarios. These are composites, drawn to show common patterns, and they are clearly presented as such. None of them describes a real person. Some limits need stating. The book is educational. It is not medical advice, and nothing in it substitutes for guidance from the clinicians who know your particular condition and history. Pacing strategies that suit one condition may be wrong for another; the right level of activity for someone recovering from a Crohn's flare is not the same as for someone with ME/CFS or poorly controlled asthma. Where the book discusses treatment-related decisions, it points you back to your own clinicians, and it means it. The legal and administrative details also vary. Much of the discussion of rights draws on the United States, where the Americans with Disabilities Act and Section 504 of the Rehabilitation Act govern how colleges must treat disabled students, and the book says so when it does. Other countries have their own frameworks, such as the Equality Act 2010 in Great Britain, and individual institutions differ widely in their procedures, their forms, and their culture. The principles, however, travel well. Almost every university in the English-speaking world now has some office responsible for disability and accessibility, some process for requesting adjustments, and some route for students who become too unwell to continue a term. Learning how to use those routes well is a transferable skill. A word about the word "disability" Many students with chronic illnesses do not think of themselves as disabled. The word can feel like it belongs to someone else, or like an admission of defeat, or simply inaccurate for a condition that is invisible most days. This book does not ask you to adopt any identity you do not want. It does ask you to understand that, for administrative and legal purposes, the offices that can help you are usually called disability services, accessibility services, or something similar, and that the definition they work with is broad. In the United States, the ADA Amendments Act of 2008 explicitly broadened the definition of disability, including conditions that are episodic or in remission if they would substantially limit a major life activity when active. A condition that flares unpredictably is precisely the kind of condition the system is designed to cover. You are also far from alone. The US National Center for Education Statistics reported that in the 2015-16 academic year, 19 percent of undergraduates said they had a disability. Not all of those students have chronic physical illnesses, and many students with chronic illnesses never report them, but the number is a useful corrective to the feeling that you are the only person in the lecture hall whose body is negotiating with them. Disability services offices are not dealing with rare exceptions. They are dealing with a substantial minority of the student body, and their procedures exist because the need is ordinary. How to use this book The book is short enough to read straight through, and doing so once is worthwhile, because the parts depend on one another. Pacing without accommodations leaves you exposed when a flare comes anyway. Accommodations without pacing turn into a cycle of extensions that pile up faster than you can clear them. Communication with professors works best when it builds on a formal accommodation already in place. After that first reading, the chapters are designed to be returned to at particular moments. Read the chapters on disability services and professors in the weeks before a semester starts, when you have time to make calls and gather documents. Read the chapters on pacing and semester design when you are choosing courses. Keep the flare-up chapter somewhere you can find it quickly, and consider reading its protocol section on a good day and writing your own version, so that on a bad day you are following instructions rather than making decisions. That last point captures the spirit of the whole book. On a bad day, your judgement, your energy, and your ability to write a clear email are all impaired. The time to decide what you will do on a bad day is a good day. Everything that follows is an attempt to help you use your good days to protect your bad ones. Chapter 1: A System Built for Steady Bodies Universities do not announce the assumptions they make about their students' bodies. Nobody writes them into the course catalogue. But the assumptions are there, embedded in the way the academic year is organized, and a student with a chronic illness will run into them repeatedly. Understanding them clearly is the first step towards working around them, because many of the difficulties that feel like personal failures are in fact collisions between a fluctuating body and a structure designed for a steady one. The hidden assumptions of the academic calendar Consider how a typical semester is shaped. Courses start together and end together. Midterm examinations cluster in the same two or three weeks because every department is following the same calendar logic. Final papers and examinations pile into the last fortnight. Laboratory sessions, seminars, and tutorials happen at fixed times each week, and attendance is often counted towards the grade. Group projects require students to meet at times that suit four or five schedules at once, which usually means evenings. Campus is large, and getting from one building to another in ten minutes assumes a certain walking pace. Every one of these features assumes that a student's capacity is roughly constant from week to week, and that it can be pushed higher for short bursts when deadlines demand. For a student without a chronic condition, that assumption is approximately true. A healthy student can run a sleep deficit through exam week and recover over the break. Their productive hours on a Tuesday in October look much like their productive hours on a Tuesday in November. For a student with a fluctuating condition, neither part of the assumption holds. Capacity varies, sometimes predictably and often not. And the ability to surge, to do more than usual for a short period, is either limited or comes with a cost that arrives later. The semester's structure concentrates demand into peaks precisely when the fluctuating student can least afford them. Consider a second-year student with inflammatory bowel disease. In a stable month she manages her classes, her part-time job, and a reasonable social life. But stress is one of the things that can make her symptoms worse, and the calendar has arranged for three midterms, a lab report, and a presentation to fall within nine days. The stress of that fortnight does not cause her disease, but it does not help it either, and if a flare begins in the middle of the cluster she faces the peak of the semester's demands at the trough of her capacity. Nothing about this is her fault. It is a predictable consequence of the calendar's shape meeting her body's shape. The point is not that universities are cruel. They are organized around the median student, and at scale that organization is efficient. The point is that the structure has a shape, and that a student with a chronic illness needs to see that shape clearly in order to plan around it rather than being ambushed by it. The transition problem For many students, the difficulty is sharpened by the timing of the move to university. Arriving at college means, very often, arriving somewhere new at the exact moment that several supporting structures fall away together. At home, a parent may have managed prescriptions, noticed early signs of a flare, booked appointments, and handled communication with school. A paediatric specialist may have known the student for years. The school itself, particularly in the United States, may have operated under a formal plan that obliged teachers to make adjustments, with a team of adults responsible for making it work. The daily routine was fixed by the school timetable, meals were regular, and sleep, however imperfect, followed a predictable rhythm. At university, much of this disappears in the same week. The student is now responsible for their own medication supply and refills, often through a different pharmacy. They may need to transfer to adult healthcare services, which in many conditions means a new specialist, a new clinic, and new expectations that the patient will manage their own care. The support plan from school does not transfer automatically; the student must start a new process with a new office. The daily structure dissolves: classes may start at eight on Monday and at two on Tuesday, meals depend on dining hall hours, and the social life of a residence hall tends to push bedtimes later. This clustering matters because each change on its own is manageable, but together they can destabilize a condition that had been well controlled. Irregular sleep, changed diet, new stresses, missed medication doses in the chaos of moving, and a gap between leaving one doctor and establishing care with another: any of these can provoke symptoms, and first-year students often face all of them at once. Many students experience their worst health of the year in the first semester, and many interpret it as evidence that they cannot cope with university, when in fact they are coping with an unusually hostile transition. The practical lesson is to treat the first semester as a transition semester and plan it with more margin than you think you need. That might mean a lighter course load, a deliberate effort to establish care with a local clinician before term starts, a stockpile of medication to cover the gap while prescriptions are transferred, and an early appointment with disability services. Students who plan the transition carefully frequently find that the second semester is much easier, not because they have become tougher but because the ground has stopped moving under them. Boom and bust If there is one pattern that almost every student with a fluctuating condition recognizes, it is the cycle that clinicians and patients often call boom and bust. It works like this. On a good day, the student feels able to do a great deal, and there is always a great deal to be done: work that was put off during the last bad spell, social plans that were cancelled, reading that has piled up. So the student does as much as possible. The good day becomes a very full day, perhaps extended late into the night because the energy is there and the backlog is real. Then the bill arrives. For some conditions, most notably myalgic encephalomyelitis, also called chronic fatigue syndrome, this has a specific clinical name, post-exertional malaise, and a specific shape: a worsening of symptoms that may be delayed by a day or more after the exertion that triggered it and may last days or longer. In other conditions the mechanism differs but the experience is similar. Overexertion leads to pain, fatigue, a migraine, a flare of inflammation, or simply a collapse in function. The student spends the next several days recovering, during which the backlog grows again. When they feel better, the backlog is even bigger, the pressure to catch up is even stronger, and the cycle repeats. What makes boom and bust so difficult to escape is that each individual decision within it is reasonable. It really is sensible to get things done when you feel well. The problem becomes visible only across the whole cycle, where the total output is lower and the suffering higher than they would be if the student did a steadier, more moderate amount every day. Averaged over a month, the student who booms and busts often achieves less than the student who deliberately does less on good days, because the busts consume so much time. The academic calendar actively encourages boom and bust. Deadlines concentrate demand, so the student is under constant pressure to surge. Grades reward work produced, not sustainability, and the cost of a surge rarely shows up in the same assignment that benefited from it. A student may earn an excellent mark on an essay written in one frantic weekend and then lose marks on the next three assignments because the weekend triggered a two-week flare. Nobody connects the two, including, often, the student. Breaking the cycle is the central task of pacing, which later chapters treat in detail. Here the important step is recognition. If your semesters feel like a series of sprints followed by collapses, the problem is probably not insufficient effort. It is too much effort at the wrong times, driven by a structure that rewards surges and a habit of catching up that feels virtuous but is self-defeating. Designing for the worst realistic week The most useful single change a student with a chronic illness can make is to stop planning around their average week and start planning around their worst realistic week. Most people, when they plan a semester, picture a typical week and check that it looks manageable. Five courses, a few hours of reading each, a job, some exercise, a bit of social life: fine. But a chronic condition means that the typical week is not the week that determines whether the semester survives. The semester survives or fails in the bad weeks, and a plan that works only in average weeks will fail in exactly the circumstances where failure is most costly. A worst realistic week is not the absolute worst case. It is not the week you were hospitalized two years ago, unless hospitalization is a regular feature of your condition. It is the kind of bad week you can reasonably expect to have several times a semester: a flare that costs you two or three days entirely and leaves the rest of the week at half capacity, say, or a week of broken sleep and pain that makes concentration unreliable. You probably know from experience roughly what your bad weeks look like and roughly how often they come. Planning for that week means asking a set of specific questions before the semester begins. If I lose three days in the week before midterms, which assignments will slip, and what happens to them? If I cannot attend class for a week, how will I get the material? If I cannot walk to the far side of campus, which of my classes can I still reach? If I miss an exam, what is the procedure for sitting it later, and have I already arranged the right to use it? If I have to stop working my part-time job for a fortnight, can I still pay rent? Each of these questions points to something that can be arranged in advance. Recorded lectures or a note-taking arrangement, flexible deadlines, alternative exam dates, a course schedule that clusters classes in reachable buildings, a financial buffer. None of them costs much to set up during a good week in August, and all of them are difficult or impossible to set up during a bad week in November. This is what it means to say that chronic illness at college is a design problem. The goal is not to prevent bad weeks, although good medical care and sensible pacing may reduce them. The goal is to build a semester that can absorb bad weeks without collapsing. Engineers call this designing for tolerance: a bridge is not built to carry the average load but to carry the heaviest load it can reasonably expect, with a margin on top. A semester for a student with a chronic illness should be built the same way. Looking well, and what the system can bend The problem of looking well One further feature of university life deserves attention, because it shapes every conversation a student with a chronic illness will have. Most chronic conditions are invisible most of the time. A student with lupus, endometriosis, epilepsy, or diabetes usually looks exactly like everyone else in the room. Even on bad days, the evidence may be hard to see: fatigue does not show, pain is often masked, and many students become skilled at appearing fine because appearing fine has been socially necessary for years. Invisibility creates two problems. The first is credibility. A professor who saw the student looking well and lively in Tuesday's seminar may be puzzled, even quietly suspicious, when the same student asks for an extension on Thursday. Fluctuation is not intuitive to people who have not lived with it. The common mental model of illness is binary, sick or well, with sickness visible and temporary. A condition that is present all the time but varies in severity, and that can be invisible even at its worst, does not fit that model, and people unfamiliar with it may fill the gap with doubt. The second problem is internal. Students with invisible conditions often doubt themselves. On a good day, the memory of the bad days fades, and the student wonders whether they were really that bad, whether asking for help is an exaggeration, whether other people would simply have coped. This self-doubt leads students to under-request accommodations, to delay registering with disability services, and to present themselves in conversations with professors as far less affected than they are. It is one of the most common reasons that a manageable semester turns into an unmanageable one. Both problems have the same partial remedy, which is documentation and formal process. When a disability services office has reviewed the evidence and issued an accommodation letter, the professor does not need to judge the student's credibility; the question has already been settled by the office whose job it is to settle it. And the student does not need to justify each request from scratch, re-litigating their own illness in their head every time they need a few extra days. The formal process exists partly to remove the burden of being believed from the individual interaction, and that is one of the strongest reasons for using it even when a student's condition is well controlled. What the system can and cannot bend It helps, before going further, to be clear about what kinds of flexibility universities can realistically offer, because not every part of the structure is equally negotiable. Some things bend easily. The timing of individual assignments often has room in it, and a professor who knows in advance that a student may need a few extra days is usually able to grant them without difficulty. The format of participation can often be adjusted; a student who cannot attend every seminar may be able to contribute through written responses. Testing conditions are routinely modified: extra time, a separate room, permission to take breaks, to eat, or to use medical equipment during an examination. The physical environment can often be changed too, through the assignment of a classroom that is easier to reach or priority housing near the student's classes. Other things bend less. A course whose core learning happens in a hands-on laboratory may not be completable without attending the labs, although the schedule for them might shift. A clinical placement for nursing or medicine may have attendance requirements tied to professional licensing. A course that assesses spoken participation in real time cannot always substitute a written alternative without changing what is being taught. Universities in the United States are not required to make changes that would fundamentally alter the nature of a course or programme, and the same principle, in different words, appears in most national frameworks. Understanding this distinction saves a great deal of frustration. Students sometimes approach disability services hoping for a single arrangement that will remove all the pressure of the semester, and are disappointed to find that accommodations adjust the conditions of academic work rather than removing the work. Others never approach the office because they assume that the essential requirements of their course cannot be touched, and so miss the many ways in which the conditions around those requirements could be made easier. The realistic picture is that a semester can be made substantially more forgiving, but that the student still has to do the learning. Accommodations buy time, flexibility, and access. Pacing determines whether the student can use that time well. Both are necessary, and neither works well without the other. The chapters that follow take each in turn, beginning with the framework of rights and offices that determines what accommodations are available and how to ask for them. Before moving on, it is worth doing a short exercise, preferably on paper. Write down, as honestly as you can, what your last few bad weeks looked like: how many days they cost you, what you could and could not do during them, and what happened to your academic work as a result. Then look at your upcoming semester's calendar and mark the weeks where demand is highest. The places where your likely bad weeks overlap with the calendar's peaks are where your planning effort should go first. That page will be useful in every chapter that follows. Chapter 2: Your Rights and the Office That Holds Them Students with chronic illnesses often approach disability services with a vague sense that they might be entitled to something, combined with an equally vague fear that asking will be treated as a request for special treatment. Both feelings dissolve once the framework is understood. Accommodations are not favours. In most developed countries they are a legal obligation on the institution, created because lawmakers recognized that an education system built for one kind of body excludes people with other kinds unless it adjusts. Knowing the shape of that obligation lets a student ask for what they need with confidence and without apology. This chapter concentrates on the United States, where the framework is detailed and where the transition from school to college involves a particularly sharp change in who is responsible for what. It then sketches the equivalent structures in several other countries. Wherever you study, the specific laws will differ from those described here, and the procedures at your institution will have their own quirks. The disability services office at your university is the authoritative source on how things work there, and its website is worth reading in full before your first appointment. The legal foundations in the United States Two federal laws govern how American colleges must treat students with disabilities. The first is Section 504 of the Rehabilitation Act of 1973, which prohibits discrimination on the basis of disability in any programme that receives federal financial assistance. Since almost every college in the country accepts federal student aid, almost every college is covered. The second is the Americans with Disabilities Act of 1990, whose Title II covers public institutions such as state universities and community colleges, and whose Title III covers most private ones. In practice, the two laws impose broadly similar obligations on higher education, and the Office for Civil Rights within the US Department of Education is the federal body that enforces them in the education context. The central obligation is that a college must not exclude a qualified student with a disability from its programmes, and must provide reasonable modifications and auxiliary aids so that the student has equal access. Several words in that sentence carry weight. The student must be qualified, meaning that they meet the academic and technical standards for admission and participation in the programme, with or without accommodations. The law does not require a college to admit someone who does not meet its standards. It requires the college not to use disability as a reason to exclude someone who does. The student must have a disability as the law defines it: a physical or mental impairment that substantially limits one or more major life activities. The definition was deliberately broadened by the ADA Amendments Act of 2008, after a series of court decisions had narrowed it. The 2008 amendments made several changes that matter a great deal to people with chronic illness. They specified that the operation of major bodily functions, including functions of the immune system, digestive system, neurological system, and endocrine system, counts as a major life activity. They directed that an impairment that is episodic or in remission counts as a disability if it would substantially limit a major life activity when active. And they instructed that the question of whether a condition is substantially limiting should generally be decided without regard to the ameliorating effects of mitigating measures such as medication. A student whose epilepsy is well controlled by medication, or whose Crohn's disease is currently in remission, does not lose protection because treatment is working. Accommodations must be reasonable. An institution is not required to lower essential academic standards, to fundamentally alter the nature of a course or programme, or to take on an undue financial or administrative burden. What counts as essential is a question the institution is expected to answer through genuine deliberation, not by assertion, and courts have generally expected colleges to show that they considered alternatives seriously before refusing an accommodation on those grounds. But the limit is real. The purpose of an accommodation is to remove barriers to demonstrating learning, not to remove the learning itself. What changes between school and college Students who received support at school in the United States often assume that it will continue at college. It does not, at least not automatically, and the reason lies in a change of legal framework. Before college, many students with chronic illnesses are served under a Section 504 plan, and some under an individualized education programme created under the Individuals with Disabilities Education Act. These frameworks place significant responsibility on the school. The school district has a duty to identify students who may need support, to evaluate them, and to develop and implement a plan with a team that includes parents. Once the student graduates from high school, the Individuals with Disabilities Education Act no longer applies, and the obligations of Section 504 and the ADA in higher education look quite different. The comparison in Table 1 sets out the main shifts. The most important is that responsibility moves from the institution to the student. A college has no obligation to find you. It is your job to disclose your condition to the disability services office, provide documentation, and request accommodations. Until you do, the college is under no duty to adjust anything, and professors are not permitted to provide disability accommodations on their own authority in many institutions, even if they are sympathetic. Table 1. How disability support changes between US high school and college. Aspect High school (Section 504 or IDEA) College (Section 504 and ADA) Who starts the process School must identify and evaluate Student must disclose and request Who pays for evaluation School, at no cost to family Usually the student Parents' role Legal participants in the plan No automatic role once student is an adult Goal of support Access and, under IDEA, educational benefit Equal access, not modified standards Existing plan Governs the school's obligations Useful as evidence but not binding Who informs teachers School staff Student delivers letters or requests notification A second shift is in the role of parents. In school, parents are legal participants in planning. In college, the student is generally treated as an adult, and in the United States the Family Educational Rights and Privacy Act gives rights over education records to the student once they are eighteen or attending a postsecondary institution. Disability services offices will normally deal with the student directly and will not discuss the student's case with parents without the student's written consent. Many offices are happy for a parent to attend an initial meeting if the student wishes, but the student is the one making requests. A third shift is in the scope of support. School plans can include modifications to what is taught and how it is graded. College accommodations generally cannot change the academic standard itself. A student may receive extra time on an examination, but the examination will ask the same questions and be marked the same way. A student may receive extensions, but the assignment will be held to the same expectations. None of this means that the school plan is useless. A recent 504 plan or individualized education programme, together with the medical evidence behind it, is often valuable supporting documentation when requesting college accommodations. The US Department of Education's Office for Civil Rights has published guidance for students making this transition, which is listed in the further reading at the end of this book and is worth reading in the year before starting college. Documentation and confidentiality Colleges are permitted to ask for documentation that establishes the existence of a disability and explains how it affects the student's functioning. Requirements vary widely. Some offices ask for a detailed form completed by a treating clinician. Others accept a letter on letterhead that states the diagnosis, describes its functional impact, and comments on its likely course. Many now place significant weight on the student's own account in an intake interview, treating clinical documentation as one source of evidence among several. The Association on Higher Education and Disability, the main professional body for disability services staff in the United States, has published guidance encouraging this more flexible approach, which it frames as reducing unnecessary barriers for students. The most useful documentation for a chronic illness focuses on function rather than diagnosis. A letter that says only "this patient has lupus" tells the disability office almost nothing about what adjustments are needed. A letter that explains that the condition produces unpredictable flares lasting several days, during which the patient experiences joint pain that limits writing and walking and fatigue that limits concentration, and that these flares are expected several times per semester, gives the office something to work with. If you are asking a clinician for a letter, it is entirely appropriate to explain what the office needs and to ask them to address functional limitations, their frequency and duration, and any treatment demands such as regular infusions or appointments that will affect attendance. Many offices publish guidelines for documentation on their websites, and bringing a copy to your clinician saves time. Confidentiality is a frequent source of anxiety, and generally a groundless one. Disability services offices typically keep medical documentation in their own files, separate from the student's academic record. The accommodation letters that go to professors normally state which accommodations the student is entitled to, not the diagnosis or the reasons. A professor learns that a student may have flexible deadlines of a certain length and may take examinations with extra time in a separate room; the professor does not learn that the student has Crohn's disease unless the student chooses to say so. The decision about how much more to tell each professor is the student's, and later chapters discuss how to make it. Accommodation status also does not normally appear on transcripts or diplomas. A future employer or graduate programme looking at your record will not see that you used accommodations. Other countries and particular situations Beyond the United States The details differ elsewhere, but the core structure is widely shared: a legal duty on institutions to make reasonable adjustments, a specialist office within the institution that administers them, and a process that begins with the student disclosing. In Great Britain, the Equality Act 2010 requires higher education providers to make reasonable adjustments for disabled students, and the duty is anticipatory, meaning that institutions are expected to think in advance about the barriers disabled students may face rather than waiting for each student to raise them. In England, eligible students can also apply for Disabled Students' Allowance, which funds support such as specialist equipment, assistive software, and certain kinds of non-medical help. The arrangements for the allowance differ in Scotland, Wales, and Northern Ireland, and they have changed over time, so current details should be checked with the relevant funding body. Northern Ireland is covered by separate disability discrimination legislation rather than the Equality Act. In Australia, the Disability Discrimination Act 1992 and the Disability Standards for Education 2005 set out the obligations of education providers, including the duty to consult with students and make reasonable adjustments. In Canada, obligations flow mainly from provincial and territorial human rights legislation, which imposes a duty to accommodate to the point of undue hardship, and universities in each province have developed their own accessibility offices and procedures accordingly. In all these systems the student will find a familiar pattern: register with an office, provide evidence, have an assessment conversation, receive a plan, and share it with the teaching staff who need to implement it. The names change, from disability services to accessibility services to student support or wellbeing, but the task is recognizably the same. Students in particular situations Several groups of students face variations on the standard process that are worth knowing about in advance. Graduate and professional students are covered by the same legal obligations as undergraduates, but their academic lives are often organized differently. A doctoral student may have few formal classes and a great deal of research, teaching, or laboratory work, and the flexibility they need may concern a supervisor's expectations, a qualifying examination timeline, or the pace of a funded studentship rather than assignment deadlines. Many disability offices are more experienced with undergraduate coursework than with research degrees, so a graduate student may need to explain the structure of their programme carefully and ask explicitly how accommodations apply to things like progress milestones. Where a graduate student is also employed by the university, as a teaching assistant for example, adjustments to the employment role may be handled by a different office, typically human resources, under employment rather than education provisions. Students in clinical, laboratory, or field programmes, such as nursing, medicine, engineering, or the sciences, often find that some requirements are tied to safety or professional standards and are therefore harder to adjust. This makes early conversation more important, not less. The right time to discover which requirements are fixed and which can bend is before a placement begins, not during it. International students should be aware that their immigration status may interact with their academic choices. In the United States, for example, students on F-1 visas are generally required to maintain a full course of study, but the regulations allow a reduced course load for a documented medical condition, with limits on how long it can last. Any change to course load, a leave of absence, or a withdrawal can affect visa status, so international students should speak to their institution's international student office before making such changes, alongside disability services. Other countries have their own rules, and the same advice applies: check before you act. Students studying online or part-time are entitled to the same protections, and online study can suit some chronic conditions very well. But online courses have their own accessibility demands, including timed quizzes that may need extended windows, synchronous sessions at fixed times, and platforms that may not work well with assistive technology. The same process of registering and requesting applies. When things go wrong Most accommodation arrangements work without dispute, but it is useful to know in advance what happens when they do not, because knowing that a route exists makes it easier to raise problems early and calmly. Problems typically take one of three forms. The disability office may decline to approve an accommodation the student believes is necessary. A professor may refuse to implement an approved accommodation, or implement it grudgingly or incompletely. Or an accommodation may be approved and implemented but turn out to be inadequate, because the condition has changed or because the original plan did not anticipate a particular kind of course. For the first and third problems, the right starting point is almost always a further conversation with the disability office, ideally supported by additional evidence or a clearer explanation of the difficulty. Offices revise plans regularly, and a request framed as "this is what happened last semester and this is why the current arrangement did not cover it" is usually received well. For the second problem, the student should contact the disability office promptly rather than trying to negotiate alone. Implementing approved accommodations is part of the institution's legal obligation, and disability offices usually have a procedure for contacting faculty who are not complying. Most such problems are misunderstandings that a short intervention resolves. If internal routes fail, American institutions receiving federal funds are required to have a designated coordinator responsible for compliance with Section 504, and a grievance procedure for disability complaints. Beyond the institution, students can file a complaint with the Office for Civil Rights, which has time limits for filing that should be checked on its website. Other countries have their own equivalents, such as independent adjudicators for student complaints or human rights commissions. Formal complaints are rarely necessary, and a student should not begin a relationship with disability services expecting conflict. But knowing that the obligations are real, and enforceable, changes the tone of a request. You are not asking for kindness. You are asking the institution to do what it is required to do, and to do it well. With the framework in view, the next step is the conversation itself: what to bring to disability services, what to ask for, and how to turn an initial appointment into an accommodation plan that will actually work on your worst realistic week. Chapter 3: The Conversation with Disability Services The first meeting with a disability services office is, for many students, the most important conversation of their university career, and one of the least prepared for. Students arrive with a folder of medical records, a nervous sense that they must prove they are sick enough, and no clear idea of what they want. They leave with a standard package of accommodations that may or may not match the way their condition actually behaves. A little preparation changes that outcome substantially. The staff in these offices are, in the great majority of cases, on the student's side; their job is to design arrangements that work, and they can do it much better when the student arrives knowing what the arrangements need to do. Before the appointment The single most common mistake is registering too late. Students put it off because their condition is stable, because they hope they will not need help, because they do not want to be the kind of student who needs help, or simply because the process looks bureaucratic and they are busy. Then a flare arrives in week seven and they discover that registration involves forms, documentation, an intake appointment that may be booked several weeks out, and, once approved, letters that must reach professors before the accommodations take effect. Many institutions do not apply accommodations retroactively. A student who registers in week eight after missing a midterm in week seven may find that the accommodation to reschedule examinations applies only to the next one. The right time to register is before the first semester begins, or as soon as possible after a diagnosis if it comes later. Many offices invite incoming students to begin registration in the summer. Registering when you are well costs very little, since an accommodation you have but do not use is harmless, and it means the arrangements are in place the moment you need them. Think of it as insurance: the premium is an hour or two of paperwork, and the cover begins before the accident. Preparation for the appointment has three parts. The first is documentation, discussed in the previous chapter. Check your institution's documentation guidelines on its website and, if you need a new letter from a clinician, request it several weeks before the appointment, bringing a copy of the guidelines so the letter addresses what the office needs. If you have records from school accommodations, bring those too. The second is a written account of how your condition affects you, in your own words. This is the part most students neglect, and it is often the most useful. Take the notes from the exercise at the end of the first chapter and turn them into a page or so of plain description. How often do bad spells happen? How long do they last? What exactly can you not do during them: attend class, walk distances, sit for long periods, read, write by hand, concentrate for more than twenty minutes? Are there regular treatment demands, such as infusions every eight weeks or specialist appointments every month, that will conflict with classes? Are there triggers the university environment might affect, such as heat, cold, stairs, irregular meals, or late-night examinations? Are there warning signs that a flare is starting? The third is a list of the specific problems you anticipate this semester, drawn from comparing your likely bad weeks with the academic calendar. This list turns an abstract conversation about your condition into a practical conversation about your courses. "I have a laboratory course with mandatory weekly sessions and I typically lose two or three days at a time when I flare" is a problem an adviser can solve. "I have lupus" is a starting point for a conversation that still has to find its way to the problem. The intake conversation Intake meetings vary from office to office, but most follow a recognizable shape. The adviser will ask about your condition, how it affects you, what support you have had in the past, and what you think you will need. They will review documentation, sometimes during the meeting and sometimes afterwards. They may propose a set of accommodations, discuss them with you, and explain how to activate them. In the United States, this is often described as an interactive process: the student and the office work together to identify the barriers and the adjustments that will remove them. The phrase is worth taking seriously. It means that you are a participant, not a petitioner. You are expected to contribute your knowledge of your own condition, and the adviser is expected to contribute their knowledge of the institution and of what has worked for other students. The language you use matters. Advisers respond best to functional descriptions: what happens, how often, and what it prevents. A student who says "I get really tired sometimes" invites a vague response. A student who says "About three times a semester I have a flare in which I cannot leave my room for two to four days, and for about a week afterwards I can attend class but cannot manage more than two or three hours of focused work a day" has given the adviser a specification. They can now reason about attendance flexibility, deadline extensions of a suitable length, and whether the student's course load is realistic. Some students find it helpful to describe their last bad week in detail, day by day. Others find it easier to describe the pattern in general terms. Either works if it is concrete. It is also important to describe the invisible costs. If you can technically attend a morning class during a flare but it will take you three days to recover from doing so, say that. If you can sit an examination in pain but your performance will not reflect what you know, say that. Advisers can only work with what they hear. It is equally important to be honest about good periods. Students sometimes fear that admitting to long stable spells will weaken their case, and so they present their condition as uniformly bad. This backfires. It makes the accommodations harder to design, because a plan for someone who is always ill looks different from a plan for someone who fluctuates, and it can create credibility problems later when the student is visibly well for weeks at a time. The strength of a fluctuating condition's case lies in its unpredictability, and that is best conveyed by describing the good periods and the bad ones truthfully. Two illustrative scenarios show the difference preparation makes. Consider a first-year student with migraine who arrives at intake saying that she gets headaches and sometimes misses class. The adviser, with little to go on, offers standard examination accommodations. In her first semester, she has a migraine on the morning of a midterm and discovers that extra time does not help if she cannot sit the examination at all; she needed the right to reschedule, which she never asked for. Now consider the same student arriving with a written summary: migraines two to four times a month, each lasting one to two days, triggered by bright screens and irregular sleep, with nausea and light sensitivity that make examinations impossible during an attack. The adviser can now propose rescheduling rights for examinations, reduced-glare conditions, a flexible deadline arrangement, and possibly a schedule that avoids early morning examinations after late evening classes. The condition is the same; the plan is completely different. When the diagnosis is uncertain Not every student arrives with a clear diagnosis. Some conditions take years to identify; students with endometriosis, connective tissue disorders, autonomic conditions such as postural orthostatic tachycardia syndrome, and a range of autoimmune diseases often spend a long time between the onset of symptoms and a confirmed name for them. Others develop symptoms during university, after an infection, for example, and are still undergoing investigation. Still others have conditions whose diagnosis is established but whose recognition among clinicians is uneven, which can make obtaining supportive documentation harder. Students in this position often assume they cannot register until they have a diagnosis. That is frequently not true. Many offices can put provisional or temporary accommodations in place on the basis of the evidence available, such as a letter from a clinician confirming that the student is under investigation for a condition that is currently causing specified functional limitations. The functional description matters more than the label. An office can reason about a student who cannot reliably stand for long periods, who experiences unpredictable episodes of severe fatigue, or who needs frequent medical appointments, whether or not the underlying cause has been named. If your situation is uncertain, contact the office early, explain that you are awaiting diagnosis, and ask what evidence they can accept in the meantime. Keep a simple symptom diary, since a record of what happened on which days is useful both to the office and to the clinicians investigating your condition. And accept that the plan may change once more is known. A provisional arrangement that gets you through a semester of investigations is worth far more than a perfect arrangement that arrives after the semester has already gone wrong. A related situation arises when a student's condition is newly diagnosed mid-semester, or when a long-standing condition changes character. The same principle applies: contact the office promptly, explain what has changed, and ask for an interim arrangement while documentation is gathered. Institutions generally cannot undo what happened before they were told, but they can act quickly once they know. Accommodations for fluctuating conditions Standard accommodation packages grew up largely around learning disabilities and sensory impairments, where needs are relatively stable: a student with dyslexia needs extra time on every examination. Chronic illnesses are different, because the need is episodic. A student may need nothing for six weeks and then need a great deal for ten days. Accommodations for fluctuating conditions therefore tend to focus on flexibility: the right to vary attendance, deadlines, and examination timing when the condition demands it. Table 2 summarizes accommodations commonly used by students with chronic illnesses, what each does, and its limits. Not every institution offers every one, the names vary, and your adviser may suggest others. The table is a starting point for conversation, not a menu from which to order. Table 2. Common accommodations for students with chronic illness. Accommodation What it does Typical limits Flexible attendance Allows absences beyond the course policy during flares Must not undermine essential in-person elements Deadline flexibility Allows extensions of a set length, requested per assignment Usually excludes final deadlines; must be requested promptly Examination rescheduling Allows a missed examination to be sat later Often requires notice on the day and a set make-up window Examination conditions Extra time, breaks, separate room, food, medical equipment Applies to how exams are sat, not whether Note-taking or recordings Provides access to missed material Recording may need consent in some classes Housing or access Room near classes, private bathroom, parking, elevator access Depends on availability; deadlines are early Priority registration Earlier course enrolment to build a workable timetable Does not guarantee places in specific sections Flexible attendance and deadline flexibility deserve special attention because they are the backbone of most chronic illness plans, and because they are frequently misunderstood. Neither is a blanket exemption. A flexible attendance accommodation usually means that the student may exceed the course's normal absence limit when their condition requires it, without penalty for the absences themselves, while still being responsible for the material missed. It does not usually mean the student can miss half the semester; at some point, a professor may reasonably conclude that the student has missed so much that the essential elements of the course cannot be met, and a conversation about an incomplete or a withdrawal becomes appropriate. Deadline flexibility usually means a defined extension, often a few days, available for assignments when the condition flares, requested through a set procedure. It generally does not cover final examinations or the last deadlines of the semester, since grades must be submitted. The practical value of these accommodations depends heavily on how precisely they are defined. An accommodation that says "reasonable flexibility with deadlines" places each request back into the professor's judgement and returns the student to the problem of needing to be believed. An accommodation that says "extensions of up to three days on assignments, requested by email before the deadline where possible and within twenty-four hours afterwards where not" is predictable for everyone. Many disability offices now use agreements of this kind, sometimes called flexibility agreements or attendance and deadline modification agreements, in which the student and each professor agree at the start of term exactly how the accommodation will work in that course: how many absences, how long an extension, how to notify, and what happens with missed in-class work. If your office uses them, complete them early in each semester, while you are well. Examination accommodations for chronic illness often go beyond extra time. A student with diabetes may need to eat, check glucose, and have medication available. A student with inflammatory bowel disease may need a seat near a bathroom and the right to leave the room without the clock stopping against them. A student with a pain condition may need to stand, stretch, or take stop-the-clock breaks. A student whose condition is worse at particular times of day may benefit from examinations scheduled accordingly where the institution can manage it. These are all routine requests, and none of them alters what the examination tests. Making the plan work in practice An approved plan is not the end of the process. Several further steps determine whether it actually functions when you need it. First, understand exactly how activation works at your institution. In some places, the office sends letters to professors automatically each term. In others, the student must request letters for each course through an online system, and in others the student must hand the letter to each professor personally. Accommodations generally take effect only once the professor has been notified, so a delay here can leave you uncovered. Most institutions require this to be renewed every term. Second, find out the notification rules for each accommodation. How much notice does the testing centre need to schedule an examination with extra time? How do you tell a professor that you are invoking an extension, and by when? What evidence, if any, is required when you miss an examination because of a flare? Write the answers down somewhere you can find them on a bad day. Third, identify a named contact. Most offices assign each student an adviser or case manager. Know who that person is and how to reach them, and save the office's email address and phone number in your phone. If a flare becomes serious, you will want to be able to send one email to one person who can help coordinate. Fourth, review the plan after the first semester. It is very rare for an initial plan to be perfect. Something will not have worked: an extension period that proved too short, an accommodation that no professor seemed to understand, a need that nobody anticipated. Book a review appointment early in the following term, bring a short account of what happened, and ask for changes. Offices expect this, and plans that have been refined over two or three semesters are usually far better than first attempts. Finally, keep your own records. Save copies of your accommodation letters, emails with professors about extensions or absences, and any correspondence with the office. This is not about preparing for conflict. It is about memory: in a busy semester, it becomes hard to remember which extension was agreed for which assignment, and a clear record avoids misunderstandings for everyone. A single folder, physical or digital, is enough. The approved plan gives you rights. The next chapter is about turning those rights into working relationships with the people who will implement them, which is a different kind of conversation. Chapter 4: Talking to Professors An accommodation letter tells a professor what the institution has agreed to. It does not tell the professor who you are, how your condition behaves, or what you will need from them in practice. That second layer of communication is up to you, and it has a large effect on how smoothly the semester runs. Professors are, for the most part, reasonable people who want their students to succeed and who dislike surprises. A student who has introduced themselves early, explained briefly how things are likely to go, and then communicated promptly and concisely when problems arise will usually find professors cooperative. A student who has said nothing and then vanishes for two weeks will often find the same professor confused and defensive, not out of hostility but because they have been left to guess. Deciding what to disclose The first decision is how much to tell each professor beyond the letter itself. There is no correct answer, and the choice may reasonably differ from course to course. At one end, a student may share nothing beyond the letter. This is entirely within their rights. The letter tells the professor what they need to implement, and the student's medical information belongs to the student. Some students prefer this approach because their condition is personal, because they have experienced stigma, or because they want to be seen in the classroom simply as students. It works well in large lecture courses where the professor's involvement is limited and the accommodations are mostly handled administratively. At the other end, a student may name their condition and explain in some detail how it affects them. This can build understanding and trust, and it helps a professor make sense of patterns they might otherwise find puzzling, such as a student who is energetic and engaged one week and absent the next. It works well in small courses, seminars, laboratory classes, and research settings where the professor and student work closely. Most students settle somewhere in between, and the most useful middle ground is functional disclosure: describing what happens without necessarily naming the condition. "I have a chronic condition that flares unpredictably. Most of the time you won't notice anything, but a few times a semester I may be unable to attend for several days, and I'll use the extension arrangement in my letter." This gives the professor everything needed to understand the accommodation in practice, without exposing the student to assumptions or questions about a specific diagnosis. A few considerations help with the decision. How closely will you work with this professor? The closer the relationship, the more useful fuller disclosure tends to be. How likely is it that your condition will be visible in their class, through absences, through needing to leave the room, or through symptoms they might notice? If it is likely, some explanation in advance prevents misreadings. Are there safety considerations? A student with epilepsy in a laboratory, or with diabetes on a field trip, may need at least one responsible person to know what to do in an emergency, and in those contexts fuller disclosure may be sensible. And how do you feel about it? Disclosure is not a test of courage. Choosing privacy is legitimate, and choosing openness is legitimate, and neither is more mature than the other. One firm piece of advice applies whatever you choose: you are never obliged to justify the accommodation to a professor or to prove your condition to them. That question was settled by the disability office. If a professor asks for medical evidence, or questions whether the accommodation is really needed, the correct response is a polite redirection to the office: "The disability services office has reviewed my documentation; if you have questions about the accommodation, they're the best people to ask." This protects you and, frankly, protects the professor too, since most institutions do not want faculty collecting students' medical information. The early conversation The best time to talk to a professor is in the first two weeks of the semester, before anything has gone wrong. A short conversation during office hours, or a brief email followed by a conversation if needed, sets up the semester on good terms. The aim of this conversation is modest. You are not asking for anything new; the letter covers that. You are introducing yourself, confirming that the professor has received the letter, and making sure you both understand how the accommodations will work in this particular course. Every course is different. The extension arrangement that works easily in an essay-based history course may need more thought in a mathematics course with weekly problem sets whose solutions are posted the day after the deadline, or in a studio course with in-class critiques. The early conversation is where these details get worked out. A useful structure has four parts. Introduce yourself and mention the letter. Describe briefly, at whatever level of disclosure you have chosen, what the professor might see. Ask how the accommodations will work with the specific features of their course. And agree how you will communicate if you need to use them. An illustrative email might read as follows. "Dear Professor Okafor, I'm in your Tuesday and Thursday section of Introduction to Organic Chemistry. You should have received my accommodation letter from Disability Services. I have a chronic health condition that is usually well managed but occasionally flares for a few days at a time, so there may be weeks when I need to use the attendance and extension arrangements. I'd like to check how these would work with the weekly lab reports and the in-class quizzes, since I know lab time is limited. Would it be possible to meet briefly during your office hours next week? Thank you, Jordan." Note what this email does. It is short. It states the essentials. It shows that the student has thought about the specific course. It asks a practical question and proposes a practical next step. It does not apologize, over-explain, or plead. It treats the accommodation as the ordinary administrative matter it is. In the meeting itself, the questions worth settling are specific. If I miss a lab, is there a make-up session, and how do I book it? If I miss a quiz, is it rescheduled or dropped from the grade calculation? How long an extension is practical for problem sets, given when solutions are released? Who should I contact if I can't reach you, a teaching assistant or a course coordinator? Are there any parts of the course where attendance is essential, so that I can plan my energy around those dates? Write down the answers. If you agreed something important, a short follow-up email confirming it is useful: "Thanks for meeting today. Just to confirm, if I miss a lab I'll email you and the lab coordinator within twenty-four hours and book a make-up slot in the following week." For students whose office uses formal flexibility agreements, this conversation is where the agreement gets filled in. Treat it seriously. The numbers you agree now, how many absences, how many days of extension, what counts as notice, will govern the rest of the semester. Communicating during a flare The real test of the relationship comes when you need to use the accommodations. The principles here are simple, though they are hard to follow when you are unwell, which is why it helps to prepare in advance. Communicate early. The moment you realize a deadline or class is at risk, send a message. Not after the deadline has passed, not once you know exactly how long you will be ill, not once you feel well enough to explain properly. A two-line message sent on the first day of a flare is worth far more than a thoughtful message sent a week later. Professors can work with uncertainty; what they cannot work with is silence. Communicate briefly. You do not need to describe your symptoms or justify your absence. You have an accommodation, and you are using it. A message such as the following is enough. "Dear Professor Okafor, I'm having a flare of my health condition and won't be able to attend class this week. I'd like to use my extension arrangement for Friday's lab report and will submit it by Monday. I'll catch up on the lecture material from the recordings. Thanks, Jordan." That is the whole message. It gives the professor what they need: what is happening, what you are doing about it, and when they can expect the work. Prepare templates in advance. Draft three or four message templates during a good week and save them somewhere easy to reach: one for missing class, one for invoking an extension, one for missing an examination, and one for an extended absence where you are not yet sure when you will return. On a bad day, you will be able to fill in a few details and send, rather than trying to compose from scratch with a foggy head. Some students also keep a list of every professor's email address and the relevant teaching assistant's, so that one message can be sent to everyone at once. Update when things change. If the flare lasts longer than you expected, send another short message before the new deadline you gave, not after it. If you realize that the absence is becoming long enough to put the course at risk, say so plainly and copy your disability services adviser: "This flare is lasting longer than usual and I'm concerned I'm falling too far behind. I've copied my disability adviser so we can think about options." That single sentence often prompts a professor to suggest solutions you had not thought of. Close the loop. When you return, a brief message acknowledging the help and confirming your plan to catch up is courteous and useful. It also marks the end of the episode clearly, so that nobody is left wondering where things stand. Notes, evidence, and verification A common anxiety concerns proof. Will you need a doctor's note every time you miss a class? For most students with an established accommodation, the answer should be no, and it is worth confirming this with your disability office. The point of a flexible attendance or deadline accommodation is that the office has already verified that your condition causes unpredictable absences; requiring fresh medical evidence for each one would defeat the purpose, and would also be impractical, since clinicians rarely see a patient during a two-day flare managed at home. Some institutions do require documentation for missed final examinations or for absences beyond a certain length, and some professors ask for evidence out of habit because their course policy requires it of other students. If a professor asks for a note for an absence covered by your accommodation, the polite response is to point them to the office: "My accommodation covers absences related to my condition, so I don't usually provide individual notes, but my disability adviser can confirm that if it's helpful." There are cases where evidence matters: an emergency department visit, a hospital admission, a period of absence long enough to require formal action such as an incomplete. In those cases, ask for discharge paperwork or a brief letter at the time, because obtaining it later is harder. Keep copies in your records folder. When the relationship is difficult Most professors respond well to this kind of communication. A minority do not, and it is worth knowing how to handle the difficult cases calmly. Some professors are unfamiliar with fluctuating conditions and find the pattern hard to understand. They may make remarks that suggest doubt, such as noting that you seemed fine last week, or asking whether you are sure you need the extension. The most effective response is usually brief, factual, and unruffled: "My condition fluctuates, so I can be well for weeks and then have a flare. The extension is part of my accommodation." Avoid over-explaining, which can make the exchange feel like a negotiation. If the doubt persists, involve your disability adviser, who can explain the accommodation to the professor directly. Some professors have rigid course policies, such as fixed attendance penalties or a strict no-late-work rule, and may be reluctant to depart from them. It is worth remembering that an approved accommodation modifies these policies for you; that is its purpose. If a professor tells you that their policy applies regardless of your accommodation, do not argue the point yourself. Contact the disability office promptly, explain what happened, and let them resolve it. This is precisely the kind of situation offices exist to handle. Group work presents a separate difficulty, because your classmates have not seen your letter and have no reason to understand your absences. A group member who disappears for a week without explanation will be resented, however good the reason. Here some disclosure to the group is usually wise, at a level you are comfortable with: "I have a health condition that sometimes flares. If I go quiet for a few days, it's that, and I'll catch up. Can we split the work so my parts aren't on the critical path right before deadlines?" Taking on tasks that can be done flexibly, such as research or editing, rather than tasks that others depend on at fixed times, can help. If a group project becomes unworkable, talk to the professor early; alternative arrangements are sometimes possible. Teaching assistants and laboratory staff are often the people you actually deal with week to week, and they may not have seen your accommodation letter at all. It is worth asking the professor at the start of term whether teaching assistants will be informed, and if not, whether you can tell them the essentials yourself. A teaching assistant who knows that you may need to leave a laboratory session quickly, or that you may miss a section and need the handout, can make a great deal of difference. Finally, some relationships simply do not work. A professor may be persistently unhelpful despite the office's intervention, or the course may prove incompatible with your condition in ways nobody anticipated. In those cases, it may be better to withdraw from the course, perhaps taking it later with a different instructor or in a different format, than to fight through a semester that is damaging your health. That is not failure. It is the same kind of design decision this book advocates throughout: arranging the conditions of your study so that your condition and your coursework can coexist. A semester in practice It helps to see how these pieces fit together over time. Consider an illustrative third-year student with type 1 diabetes and a history of difficult-to-control blood glucose during periods of stress and irregular sleep. In the first week of term, he requests his accommodation letters and emails each of his four professors a short note: he has a condition that is usually well managed, he may occasionally need to eat or check his glucose in class, and he may rarely miss a session or need a short extension after a bad night. He meets two of them, the laboratory instructor and the seminar leader, because those courses have in-person requirements. The laboratory instructor, reassured to know in advance, tells him where the nearest place to keep glucose supplies is and says the teaching assistants will be told. In week six, a hypoglycaemic episode overnight leaves him exhausted and unwell the next day. He sends his prepared template to the two professors whose classes he will miss, uses the extension for a problem set due that afternoon, and submits it two days later. Nobody asks for anything more. In week eleven, an infection sends his glucose badly out of range for several days and he is briefly seen in hospital. This time he sends a longer message to all four professors, copying his disability adviser, explaining that he is unwell and will be in touch within a few days. When he returns, his adviser helps him prioritize the backlog, one professor agrees to drop a missed quiz from his grade calculation, and he asks the hospital for discharge paperwork in case it is needed for the final examination period. Nothing about this semester was easy. But at no point did he have to explain his condition from scratch, argue for his accommodations, or decide under pressure what to do. The arrangements made in the first week carried him through both episodes. Communication, though, is only half of the system. Accommodations and good relationships create room for your condition. What you do with that room is a question of pacing, which is the subject of the next two chapters. Chapter 5: Pacing: Energy as a Budget Accommodations change what the university expects of you. Pacing changes what you expect of yourself. It is the more intimate of the two tasks, and in many ways the harder, because it asks a student to override instincts that feel like virtues: the urge to finish what you started, to make the most of a good day, to keep up with everyone else. Yet pacing is the practice that most directly determines how many bad weeks a semester contains and how long each one lasts. A well-accommodated student who does not pace will still spend the semester lurching between surges and crashes, using extensions to recover from flares that the surges helped provoke. A student who paces well can often make their accommodations last, needing them less often and recovering faster when they do. What pacing is Pacing, in the sense used by clinicians who work with chronic pain and fatigue, means managing activity so that it stays within the limits your condition currently allows, spreading it evenly rather than concentrating it, and taking rest before exhaustion forces it on you. The idea has a long history in rehabilitation and in patient self-management programmes, and it appears in guidance for a range of conditions. The UK's National Institute for Health and Care Excellence, in its 2021 guideline on myalgic encephalomyelitis and chronic fatigue syndrome, placed energy management at the centre of its recommendations, describing an approach in which people learn to use their available energy while staying within their limits, and it advised against graded exercise programmes based on fixed increments. For other conditions, including many chronic pain conditions and inflammatory diseases, clinicians may encourage carefully increasing activity over time, and appropriate exercise can be an important part of treatment. What counts as good pacing is therefore not the same for everyone, and decisions about physical activity in particular belong in a conversation with your own clinicians. The common thread, across conditions, is the rejection of boom and bust. Whatever the right overall level of activity is for you, it is better delivered steadily than in surges. That principle applies as much to reading and writing and socializing as it does to walking or exercise. The most widely known way of describing this among patients is the spoon theory, set out by Christine Miserandino in a 2003 essay in which she explained her life with lupus to a friend by handing her a bunch of spoons, each representing a unit of energy, and taking one away for every ordinary task of a day: getting up, showering, dressing, travelling. The friend ran out before evening. The metaphor spread widely because it captures two truths that healthy people rarely grasp: that energy for someone with a chronic illness is a limited stock rather than an effectively bottomless supply, and that ordinary tasks others do without thinking draw on that stock. Many students with chronic illnesses describe themselves as spoonies, and the vocabulary can be a useful shorthand with friends. For the purposes of planning, though, it helps to go a step further and think about energy as a budget, with income, expenditure, and the consequences of overdraft. Finding your baseline A budget begins with knowing your income. In pacing terms, this means identifying your baseline: the amount of activity you can do on most days, including moderately bad ones, without triggering a worsening of symptoms in the following days. The baseline is almost always lower than students expect, and lower than what they can do on a good day. That is the point. A baseline set at the level of your best days will be exceeded on every ordinary day, and a baseline that is regularly exceeded is not a baseline at all. The goal is a level you can sustain through most of the semester, with room left over. Finding it takes some observation. A practical method is to keep a simple diary for two or three weeks. Each day, note roughly how much time you spent on different kinds of activity: attending class, studying, physical activity including walking, socializing, and anything else significant. Note your symptoms, on a simple scale, both that day and in the days that follow. Note sleep. Many students use a notebook or a spreadsheet; some use apps designed for symptom tracking. The tool matters less than the habit. After a few weeks, patterns tend to appear. You may find that more than about four hours of focused study in a day reliably leads to a worse day after. You may find that days with a long walk across campus and an evening social event are followed by a crash, while days with either alone are not. You may notice that your symptoms are worse after late nights, or after days without proper meals, or in the days following an examination. These patterns are your personal data, and they are far more useful than generic advice. A reasonable starting baseline is somewhat below what the diary suggests you can manage on an average day. Some clinicians suggest aiming for a level you could manage even on a moderately bad day. That can feel frustratingly low, but a baseline you reliably meet is a foundation for building up, where a baseline you regularly exceed leads straight back to boom and bust. If your condition allows it, and with appropriate clinical guidance, the baseline can be increased gradually once it has been stable for a while. If your condition is one in which exertion carries particular risk, the aim may instead be to stay within your limits and let improvements, if they come, come on their own. The kinds of energy students spend One reason students underestimate their expenditure is that they count only physical effort. But college draws on several kinds of energy, and for many chronic conditions each has its own cost. It helps to think of student life as drawing on at least five accounts. Physical energy goes on walking between buildings, climbing stairs, standing in laboratories, and the easily forgotten costs of carrying a heavy bag or waiting in a queue; clustering classes, using lifts, and planning rest stops on long routes all reduce it. Cognitive energy goes on lectures, reading, problem sets, and examinations, and also on screen time and the constant switching between tasks that modern study involves; short blocks of work and a rule of one genuinely hard task per day help protect it. Emotional energy goes on presentations, conflicts, homesickness, and the quiet work of hiding symptoms; recovery time after stressful events is the main defence. Sensory energy goes on loud lecture halls, bright screens, and crowded dining rooms, and can be conserved with quiet study spaces and dimmed displays. Social energy goes on parties, group meetings, and clubs, with their late nights and the repeated effort of explaining oneself; choosing events deliberately and deciding in advance when to leave keeps it in check. The relative weight of each account varies a great deal between conditions and between people, which is one more reason to rely on your own diary rather than on general rules. Cognitive energy deserves particular attention because it is the currency of academic work, and because many students with chronic illness experience cognitive symptoms as part of their condition: the difficulty with concentration and memory that patients often call brain fog. Cognitive exertion can be as costly as physical exertion for some conditions. A student who carefully avoids overexerting physically but then spends eight hours in the library before an exam may be surprised by the crash that follows, but the crash is no mystery. Eight hours of intense study is a heavy expenditure. Emotional energy is also routinely underestimated. A difficult phone call, an anxious presentation, or a conflict with a flatmate can drain a student as effectively as a long walk. So can the ongoing work of masking symptoms: the effort of appearing fine in class, at a party, or in a seminar when you are in pain or exhausted. Many students with invisible conditions do this so constantly that they no longer notice it, and are puzzled why a day of apparently light activity leaves them wrecked. Counting masking as an expenditure helps explain those days and suggests a remedy, which is building in some time each day in a place where you do not have to perform wellness for anyone. Pacing techniques that work at college With a baseline and an awareness of the kinds of expenditure, several practical techniques help keep activity within limits. None is complicated. All require practice. Break work into blocks and rest between them. Rather than studying for three hours straight, study for forty minutes, rest for fifteen, and repeat. The right block length depends on you; some students with significant fatigue find twenty minutes is the useful maximum. The rest should be genuine rest, which for many conditions means lying down with eyes closed, or at least doing something undemanding, not switching to scrolling a phone, which is itself cognitively and visually demanding. Rest before you need to. This is the counterintuitive heart of pacing. The time to stop is not when you are exhausted but before, while you still feel able to continue. Stopping at the point of exhaustion means you have already overspent. Many students find it useful to set a timer that ends each block regardless of how they feel, because in the middle of a task, especially an absorbing one, the signals of approaching exhaustion are easy to ignore. Spread demanding tasks across the week. If you have three hard tasks this week, do one a day rather than all three on Monday because Monday is a good day. This is where pacing collides most directly with the academic calendar, since deadlines push towards clustering. The answer is to start earlier, so that the work for a Friday deadline is spread across the preceding week rather than compressed into Thursday. Starting early is usually presented as a matter of good discipline; for a student with a chronic illness it is a matter of medical self-management. Alternate types of demand. A day that mixes physical, cognitive, and social activity in moderate amounts is often more manageable than a day that is heavy in only one. After a long lecture, a short walk might be restful; after a long walk, sitting with a book might be. Some students plan their days as a sequence of changes of activity rather than a sequence of tasks. Plan the week with slack. Leave at least one period each week, ideally more, with nothing scheduled. That slack is where recovery happens on normal weeks and where catching up happens on bad ones. A week that is full on Monday has no room for anything to go wrong. Know your warning signs. Most students with fluctuating conditions can learn to recognize the early signals of overexertion or an approaching flare: a particular kind of tiredness, a change in pain, sleep that stops being refreshing, irritability, a subtle cognitive fog. When those signs appear, the pacing response is to reduce activity immediately, before the flare fully arrives. This is exactly when students are most tempted to push on, because they sense they are about to lose time and want to get as much done as possible first. Pushing on at that point very often makes the flare worse and longer. Sleep, meals, and medication as infrastructure Pacing usually focuses on activity, but a budget has income as well as spending, and for most chronic conditions the reliable sources of income are unglamorous: sleep, regular meals, and consistent treatment. These are also the three things college life disrupts most. Sleep in student housing tends to drift later and become irregular, especially in the first year, and many conditions respond badly to irregular sleep. Migraine, epilepsy, and many pain and fatigue conditions are commonly associated with sleep disruption as a trigger or aggravating factor, and diabetes control can be affected by irregular routines. A consistent sleep and wake time, protected even at weekends where possible, is one of the highest-return pacing measures available. That may mean choosing classes that do not begin at dawn if mornings are hard for you, or negotiating quiet hours with flatmates, or, if housing is a persistent problem, asking disability services about a single room. Regular meals matter for similar reasons, and matter especially for conditions affected by diet or blood glucose. Dining halls have fixed hours that may not match a student's timetable, and students who are exhausted often skip meals because getting food is itself an effort. Keeping simple food in your room for bad days, planning when you will eat on long days, and, where relevant, asking about dietary or housing accommodations can close this gap. Medication adherence is the third piece. Missed doses are common in the disruption of term, particularly during travel, late nights, and the weeks when prescriptions are being transferred. Phone reminders, a weekly pill organizer, and a system for reordering before supplies run low are dull but effective. If your treatment involves regular infusions, injections, or appointments, put them into your calendar for the whole semester at the start, and treat them as fixed commitments around which classes and study are arranged rather than the other way round. None of these is a substitute for activity pacing, but together they raise the level of energy available to be paced. A student who sleeps regularly, eats predictably, and takes treatment consistently usually has a higher and more stable baseline than the same student living on the chaotic rhythms that university life tends to produce. The good-day problem Every student who tries pacing runs into the same obstacle. On a good day, the diary says to stop, the timer says to stop, the plan says to stop, but the student feels fine, has a pile of work, and cannot see why they should waste a good day resting. This is the moment that determines whether pacing works. The argument for stopping is not that you feel unwell now. It is that doing more now will, on the evidence of your own diary, cost more later. The good day is precisely when you are most likely to overspend, because the warning signals are quiet. Using good days to bank some of the week's work is sensible; using them to do three days' work in one usually is not. A useful compromise is to let good days go somewhat above baseline, by a modest and fixed amount, and to put that extra into tasks that reduce future pressure, such as starting the next assignment early, rather than into catching up in a frantic burst. Another is to treat the energy you do not spend on a good day as a reserve that protects the next few days. Neither makes the feeling of wasted potential go away entirely. But students who persist with pacing through the first semester of trying it often report that their bad weeks become less frequent and less severe, and that their total output across the semester rises even though their output on good days falls. That shift is the whole case for pacing, and it can only be demonstrated by trying it long enough for the diary to show it. Consider an illustrative student with a long-term post-viral fatigue condition who began pacing in her second year. For the first month she found it maddening: she was stopping work in the middle of productive afternoons, turning down invitations, and going to bed at a fixed hour while her flatmates stayed up. Her marks did not improve. But her diary showed that she had one crash that month instead of the three she typically had, and each crash was shorter. By the end of the semester she had used fewer extensions than in any previous term. She was still doing less on her best days than she wanted to. She was also doing something every day, which she had not managed before. Pacing handles the day and the week. The next chapter scales the same logic up to the whole semester, where the decisions are about how many courses to take, which ones, and how to arrange them so that the semester's demands stay within a sustainable budget. Chapter 6: Designing a Semester Daily pacing can only work within the space the semester allows. A student who has signed up for a load that exceeds their budget by a wide margin will fail at pacing no matter how disciplined they are, because there is simply more work than their sustainable daily capacity can complete. The largest pacing decisions are therefore made before the semester starts, at the moment of registration, when a student chooses how many courses to take, which ones, at what times, and in what combination. These decisions are made once and then lived with for fifteen weeks. They deserve more care than they usually get. How much to take on The standard full-time load at most institutions is designed around the average healthy student, and it carries with it an assumption about how many hours per week that student will spend on coursework. For a student with a chronic illness, the relevant question is not whether they can manage a standard load in a good week, but whether they can manage it across the whole semester including their expected bad weeks, while also leaving time for healthcare, rest, and the extra administrative work that chronic illness entails. That last point is often forgotten. Living with a chronic condition is time-consuming even when it is stable. Appointments, pharmacy trips, insurance calls, treatment sessions, symptom management, and the rest can add up to several hours a week, sometimes many more. A student who plans their semester as if those hours did not exist is planning a semester with a built-in deficit. A practical method is to estimate your realistic weekly capacity for academic work, using your pacing diary, then subtract the time your condition itself will consume, then compare the result with what the proposed course load will demand. Estimate the demand generously: most institutions suggest a rule of thumb for hours of study expected per credit, and that figure is a useful starting point, though courses in the same department can vary widely. Then apply the worst-realistic-week test from the first chapter: if you lose two or three weeks of full capacity over the semester, where will that work go? If the answer is that it will pile onto the remaining weeks and push them over your baseline, the load is too heavy. Many students who carry out this exercise honestly discover that a standard full load exceeds their sustainable capacity. That discovery is uncomfortable, because it seems to mean graduating late, paying more, or falling behind friends. It is worth weighing that discomfort against the alternative. A student who takes a full load and cannot sustain it often ends up with a semester of repeated flares, low grades, incomplete courses, or withdrawal, which costs more time and money than a lighter load would have, and costs health as well. A student who takes a lighter load that they can actually complete makes steady progress. Over a degree, steady progress at a sustainable rate frequently arrives sooner than a series of overloaded semesters punctuated by crashes. There are several ways to lighten a semester, and they differ in their consequences. Which ones exist at your institution, and exactly how they work, needs to be checked locally. In particular, American students should be aware that full-time or half-time enrolment status can affect eligibility for federal financial aid, some scholarships, some health insurance arrangements, and campus housing, and international students' visa status can depend on enrolment, as discussed in the second chapter. Before reducing your load, speak to the financial aid office, and to the international student office if you hold a student visa. The most straightforward option is part-time enrolment: fewer courses each term, often at a lower cost per term. Its drawbacks are administrative rather than academic, since part-time status can affect aid, housing eligibility, insurance, and visas. A second option keeps the number of credits the same but changes their mix, balancing demanding courses with lighter ones so that the total weekly demand falls; this depends on how much freedom your degree requirements and prerequisite sequence allow. A third spreads credits across the year through summer or intersession courses. A fourth is to plan formally for an extended time to degree, agreeing with your academic adviser from the outset that you will take an extra term or two, which makes each semester lighter; here the thing to check is whether aid programmes or scholarships limit the total number of terms they will fund. A fifth option deserves special mention because many students do not know it exists. Some institutions allow students with documented disabilities to be treated as full-time while carrying fewer credits than the usual full-time minimum, as an accommodation. Where it is available, this can preserve access to housing and some benefits tied to full-time status. However, it does not automatically change the rules of outside bodies such as government aid programmes or private scholarship providers, which may apply their own definitions. Ask disability services whether it is available and ask financial aid exactly what it would mean for you. Summer and intersession courses need a caution. They spread credits across the calendar, which can lighten the main semesters. But they are usually compressed, covering a full course in a few weeks, which concentrates demand in exactly the way that pacing tries to avoid. For some students, a single summer course at a compressed pace works well; for others, it is a recipe for a flare. Try one before relying on them. The cost of going slower The arithmetic of course load is usually easier than the feelings attached to it. Students who take a lighter load often find themselves out of step with their friends, graduating a term or a year later, explaining to relatives why they are still at university, and quietly measuring themselves against peers who seem to move through their degrees without effort. For students who have spent years insisting that their illness would not hold them back, a reduced load can feel like the illness winning. It is worth confronting that feeling directly, because left unexamined it drives students back into loads they cannot sustain. Three points help. The first is that the comparison is false. Your peers are not running the same race with a lighter pack; they are running a different race. Comparing your pace with theirs makes about as much sense as comparing your medication schedule with theirs. The meaningful comparison is between the different ways you yourself might get through your degree, and on that comparison, a sustainable pace very often wins. The second is that the time cost is usually smaller than it feels. Many students with chronic illnesses who take lighter loads graduate only a term or two later than the standard timeline, and some make up the difference through summer study. Set against a career that may last forty years, a few extra months at university is a small price for arriving at the end with your health, your grades, and your transcript intact. The third concerns other people's expectations. Parents in particular may have strong views, often rooted in worries about cost or in a desire to see their child treated as unaffected by illness. A calm conversation, before registration, that explains the reasoning, including the realistic alternative of an overloaded semester that ends in withdrawal, tends to go better than a decision presented after the fact. If family members are paying fees, involving them in the planning is both fair and practical. If they remain unconvinced, remember that you are the one who lives in your body and sits in the examinations, and that the decision is, in the end, yours. Choosing and combining courses Beyond the total load, the particular combination of courses matters a great deal. Two semesters with the same number of credits can have very different demands on a student with a chronic illness. Balance heavy and light courses. Most programmes contain some courses known to be demanding, such as intensive laboratory courses, large projects, or courses with heavy weekly problem sets, and some known to be lighter. Asking older students, advisers, and departmental offices about workload is legitimate and useful. A semester with two demanding courses and two lighter ones is usually more sustainable than one with four demanding courses and a lighter one the following semester. Balance assessment types. Some courses are assessed mainly through examinations, some through continuous coursework, some through a large final project. Each has a different risk profile. Examination-heavy courses concentrate risk into a few days: a flare on the day of a final can be very disruptive, though examination rescheduling accommodations help. Continuous assessment spreads risk but requires steady output every week, which is hard during a flare but easy to recover from with extensions. Large final projects allow work to be spread across the semester if started early, but can become crushing if a flare arrives late. Knowing which pattern suits your condition, and mixing them so that no single week or type of failure can sink the whole semester, is part of design. Look at when deadlines fall. Syllabi from previous years, where available, show roughly when major assessments occur. If three of your prospective courses all have major projects due in week twelve, that week is a risk point; you may choose to swap one course, or at least to plan early starts for the projects. Think about attendance requirements. Courses with mandatory attendance, laboratory sessions, or in-class participation grades are harder to manage during flares than courses where attendance is less central. That does not mean avoiding them, since many are essential and valuable. It does mean not stacking too many of them in the same semester. Consider format. Some courses are available online, hybrid, or with recorded lectures, and these can be easier during flares. Some students find online courses suit their condition extremely well. Others find that the self-direction required, and the absence of a regular structure, makes pacing harder. Knowing which you are is useful; trying one online course before committing to several is a sensible experiment. Sequence prerequisites carefully. In programmes with long prerequisite chains, a single failed or withdrawn course can delay a whole year of progress. Planning your degree with an adviser at the start, and identifying which courses are critical path, lets you protect those courses by putting them in lighter semesters and taking flexible electives in heavier ones. Arranging the timetable The weekly timetable is where the semester's design meets the body's daily rhythms, and small choices here can make a large difference. Time of day matters. Many conditions have daily patterns: some people are stiff, fatigued, or unwell in the morning and better later; others fade as the day goes on. Where you have a choice of sections, choosing times that match your better hours can transform your ability to attend and engage. Priority registration, which many disability offices can arrange, is valuable precisely because it gives students the chance to secure sections that fit their needs before they fill. Location matters. On large campuses, back-to-back classes at opposite ends can require a fast walk that is simply not possible for some students, or not possible without significant energy cost. Clustering classes in nearby buildings, or leaving longer gaps between distant ones, reduces physical demand. If your mobility is limited, disability services may be able to arrange for classes to be moved to accessible rooms, provide transport options that some campuses offer, or help with parking. Spacing matters. Some students prefer to concentrate classes on a few days, leaving others free for rest and independent study. Others find that long days of back-to-back classes exceed their capacity and prefer a lighter daily load spread across five days. Neither is correct in general; your pacing diary will suggest which suits you. What matters is that the arrangement is deliberate rather than an accident of which sections had space. Build in breaks. A timetable with a gap after a long class, or at midday, gives room to rest, eat, take medication, or simply recover before the next demand. Where possible, identify a place near your classes where you can rest during these gaps, such as a quiet room, a library corner, or a room that disability services can help you access. Some institutions have designated rest spaces; it is worth asking. Protect treatment time. If you have regular appointments or treatments, schedule classes around them rather than hoping to fit them in. An infusion every few weeks that clashes with a mandatory laboratory is a problem better solved at registration than halfway through term. A worked example It is worth seeing how these principles combine in a single plan. Consider an illustrative second-year student majoring in biology who has Crohn's disease. His condition has been moderately active over the past year, with two flares that each cost him about a week. He receives an infusion every eight weeks at a hospital forty minutes away, and those days are effectively lost. His mornings are often difficult. The standard load in his programme is five courses. He has been advised to take organic chemistry, a genetics course with a weekly laboratory, a statistics course, and two electives. He begins by estimating capacity. His diary suggests he can manage about five hours of focused academic work on an average day, perhaps less on bad days, and he loses roughly a week and a half per semester to flares and infusions, with some further reduced days before and after. A five-course load with two laboratory-based science courses would, on a rough calculation, exceed his sustainable capacity even in average weeks, and leave nothing for bad ones. He meets with his academic adviser and disability services. Disability services confirms that his institution allows a reduced load as an accommodation and still treats him as full-time; financial aid confirms that his particular aid and scholarship would not be affected, though they advise him to check again each year. He decides to take four courses: organic chemistry, genetics, statistics, and one elective chosen partly because it is assessed through continuous essays rather than a final examination. He moves the second elective to a summer online course, which he will take slowly as the only commitment during that period. He then turns to the timetable. Using priority registration, he chooses sections that start no earlier than ten in the morning. He schedules his genetics laboratory on a day far from his infusion dates, after checking with the department that make-up sessions are available if needed. He clusters his classes in the science buildings on the same side of campus and leaves a two-hour gap in the middle of the day on his longest day, when he can rest, eat, and use a bathroom he knows is accessible. Finally, he marks the semester's high-risk weeks: the organic chemistry midterms, the genetics laboratory report deadlines, and the week before finals. He schedules his infusions, where his specialist can accommodate it, to fall outside those weeks. He plans to start every major assignment at least a week before it is due. None of this guarantees a smooth semester. He may still flare during midterms. But the semester now has a margin. If he loses a week, his load is light enough that the backlog can be absorbed into the following weeks without pushing him far above his baseline, and his accommodations allow him to reschedule an examination or take an extension without argument. A flare is no longer a threat to the whole semester; it is a problem with a known solution. This kind of design is not a one-off. It is worth repeating before each semester, using what you learned in the last one. Did the lighter load work? Were the morning classes as difficult as you feared? Did continuous assessment suit you better than examinations? Each semester's experience is data for the next, and students who treat semester planning as an ongoing experiment usually find that their plans improve steadily over the course of a degree. Even the best design will sometimes be overwhelmed. The next chapter concerns what happens when it is, and how to handle a flare-up so that it does as little damage as possible. Chapter 7: When the Flare Comes Every student with a fluctuating condition will, sooner or later, face a flare in the middle of a semester. Good pacing may make flares less frequent, and good medical care may make them milder, but neither can eliminate them. What a student can control is how a flare is handled: whether it is met with a prepared response that limits its academic damage and allows the body to recover, or with improvisation, silence, and a desperate attempt to keep up that makes it longer and worse. This chapter sets out a protocol, in four phases. The first is preparation, done on a good day. The second is the first two days of a flare, when the key decisions are made. The third covers what to do when a flare lasts longer than accommodations can cover. The fourth is return and review. The protocol is general; you will need to adapt it to your condition and your institution, and the medical side of it should always be shaped by your own clinicians. Preparation: writing a flare plan The most useful thing you can do about flares is to write down, while well, what you will do when one comes. A flare impairs exactly the capacities needed to manage it: judgement, energy, memory, and the ability to write a clear message. A plan written in advance turns a set of hard decisions into a checklist. A good flare plan fits on a page or two and covers several things. Medical steps come first. Many people with chronic conditions have, or can ask their clinicians for, a written action plan: what to do when symptoms begin to worsen, which medications to adjust or take, and which signs mean you should contact your care team or seek urgent help. If you have such a plan, keep a copy with your flare plan. If you do not, ask for one at your next appointment. Include the contact details for your specialist team, your campus health service, and the nearest urgent care and emergency facilities to where you live. Know the emergency signs. For some conditions, a flare can become dangerous: severe dehydration, very high or very low blood glucose, prolonged seizures, signs of serious infection in someone on immune-suppressing medication, or other red flags that your clinicians will have described. In a medical emergency, call emergency services immediately. In the United States that means calling 911; other countries use different numbers, and you should know the one where you study. Do not let concern about disrupting your semester, or embarrassment, delay you. Academic arrangements can always be sorted out later. Next come the people. Who needs to know when you flare, and who will help? For many students, this includes at least one friend or flatmate who knows about your condition, knows what a flare looks like, and has agreed to check on you, bring food, or help you get to medical care if needed. This matters especially for students living alone or away from family. It is a larger request than it may seem, so make it explicitly and gratefully, and be clear about what you would and would not want them to do. Your plan should also list your disability services adviser, your academic adviser, and, if relevant, an emergency contact who can be called if you are unable to communicate. Then the practical supplies. Keep a small stock of what you need to get through a few days without leaving your room: food that requires no preparation, drinks, medications and any equipment, and whatever comfort items help. Knowing that these are there removes one layer of worry when you are unwell. Finally, the academic steps. Include the message templates described in the fourth chapter, a list of every professor's and teaching assistant's email address, the procedure for invoking each of your accommodations, including any deadlines for notifying the testing centre of a missed examination, and the contact details for disability services. Some students keep a one-line summary for each course: how to report an absence, how long extensions are, and what happens if an exam is missed. Keep the plan where you can reach it without effort: printed on the wall, saved on your phone, or both. Review it at the start of each semester and update the course details. The first forty-eight hours When a flare begins, the first two days matter most. Decisions taken then, or not taken, shape how the next weeks go. Put health first. The first task is to follow your medical action plan, contact your care team if the plan calls for it, and rest. Everything academic comes second. This sounds obvious, but many students in the first hours of a flare instinctively try to finish the assignment due tomorrow, attend the class they were planning to attend, or sit the exam they have studied for. Sometimes one of these is genuinely the right call; more often, pushing on through the start of a flare makes it longer and more severe, costing far more academic time than it saved. If in doubt, stop. Communicate within the first day. Using your templates, send brief messages to the professors whose classes or deadlines are affected in the next few days. You do not need to know how long the flare will last; say that you are unwell, that you are using your accommodation, and that you will update them. If an examination is imminent, follow the procedure for missing it precisely, since testing centres often have strict notification windows. If the flare seems likely to be serious or prolonged, copy your disability services adviser. One email from bed in the first day prevents a cascade of missed deadlines and anxious messages later. Triage the academic work. Once the messages are sent, and ideally with a friend's help if concentrating is hard, sort everything due in the coming two weeks into three groups. Some things are time-critical and must be dealt with even during the flare, usually by arranging an extension or rescheduling rather than doing the work. Some can wait until you recover, covered by your accommodations. And some can be dropped entirely, such as optional readings, non-assessed exercises, or participation in activities that do not count towards a grade. Students often find that the third group is larger than they expected. Letting it go without guilt is part of recovering. Then rest properly. The temptation during a flare, especially once the messages are sent and the pressure feels lighter, is to try to do a little work each day to limit the backlog. For some conditions and some flares, gentle, limited work may be fine. For others, especially where exertion prolongs symptoms, it is counterproductive. Follow your clinicians' guidance and your own experience. A flare that is allowed to resolve fully is usually shorter than one repeatedly interrupted by attempts at productivity. Flares during examination periods Flares that arrive during examination periods deserve separate thought, because they combine the highest stakes with the least flexibility. Final examinations are usually scheduled centrally, grade deadlines follow soon after, and the ordinary extension arrangements rarely apply. Examination periods are also, for many students, a common time for flares, since they bring stress, disrupted sleep, long hours of study, and irregular meals all at once. Three things help. The first is to know, before the examination period begins, exactly what your institution's procedure is for a missed final: whom to notify, by when, what evidence is required, and when the deferred or make-up examination would take place. Many institutions have a formal deferred examination process separate from disability accommodations, often administered by the registrar or examinations office, and some require notification before or on the day of the missed examination. Put these details in your flare plan at the start of the period. The second is to resist the pressure to sit an examination you are too unwell to sit. Students often reason that since they have studied, they should attempt it anyway. But sitting an examination while acutely unwell usually produces a result that reflects the illness rather than the learning, and at many institutions, once you have sat an examination, you cannot later claim that illness affected it. If you are clearly too unwell, follow the missed-examination procedure instead. If you are uncertain, contact disability services or the examinations office before the examination, not after. The third is to spread the load of the examination period as much as your institution allows. Some disability offices can help arrange for examinations not to fall on consecutive days, or for a student to sit no more than one examination in a day, where the student's condition makes back-to-back examinations unmanageable. These arrangements usually need to be requested well before the examination timetable is finalized, which is one more reason to discuss the examination period with your adviser early in the semester rather than at its end. When a flare outlasts the accommodations Most flares are covered by flexible attendance and deadline accommodations: the student misses a few days, takes a few extensions, and recovers. Some are not. A flare may last several weeks, require hospital treatment, or arrive so late in the semester that there is no time to complete the remaining work before grades are due. In these situations, the question changes from how to manage the backlog to which formal option best protects the student's academic record and health. Institutions typically offer several options, whose names and rules vary. Table 3 compares the most common ones. It is essential to check your own institution's policies, which are usually found in the academic regulations or the registrar's website, and to discuss options with both your disability adviser and your academic adviser before deciding. Deadlines matter a great deal: many of these options have cut-off dates in the semester, after which they are harder to obtain. Table 3. Formal options when illness disrupts a semester. Option What happens Usually suits Things to check Extended extension Specific deadlines moved within the term Short disruption, time left in term Final grade submission dates Incomplete grade Course finished after the term ends Most work done; a defined portion remains Completion deadline; default grade if unfinished Course withdrawal Leave one or more courses One course unsustainable, others fine Withdrawal deadline; refund; load status Medical withdrawal or leave Leave all courses for the term on medical grounds Serious or prolonged illness Refunds; aid; housing; insurance; return process Retroactive withdrawal Withdrawal granted after the term Illness prevented timely action Evidence required; time limits for applying An incomplete is often the best option when a student has done most of the work in a course and a flare has prevented them from finishing the last part. The student and professor typically agree what remains to be done and by when, and the course is graded once it is complete. The risks are that incomplete work carries over into the next semester and adds to its load, and that if it is not finished by the deadline, it may convert to a failing grade automatically at some institutions. A student taking several incompletes can find the next semester overwhelmed before it begins. Incompletes work best when there are few of them, the remaining work is clearly defined, and the completion deadline allows realistic time, ideally during a break. Withdrawing from one course can rescue the rest of a semester. If one demanding course is consuming disproportionate energy, dropping it may allow the student to complete the others well. The cost is the need to retake the course later, possible financial consequences, and a mark on the transcript at some institutions, although a withdrawal is generally viewed far more kindly than a failing grade. A medical withdrawal or medical leave of absence takes the student out of all courses for the term. It is the right choice when illness is serious enough that continuing any academic work is unrealistic or harmful. Many institutions have specific procedures for medical withdrawal that may allow partial refunds of tuition beyond the normal refund schedule, protect the transcript, and define a process for returning. Students considering this should ask carefully about its effects on financial aid, housing, health insurance obtained through the institution, and, for international students, visa status. American students should also know that withdrawing can affect federal aid calculations for that term and that aid programmes generally have appeal processes for students whose academic progress has been affected by illness, which the financial aid office can explain. Retroactive withdrawal addresses a common and painful situation: a student became so unwell during a semester that they could not act in time, and the semester ended with failing grades that reflect illness rather than ability. Many institutions allow students to petition, after the fact and with supporting evidence, to have those grades replaced with withdrawals. The process and time limits vary. If this has happened to you, ask your academic adviser or dean of students' office whether it is available. The decision among these options is not purely academic. It is also a decision about health. A student who clings to a semester that has clearly been lost, taking incompletes in every course and planning to catch up over the break, may be setting up a second failed semester. A student who takes a medical leave, recovers properly, and returns with a well-designed plan often does far better. Your clinicians can help you judge whether your condition is likely to improve in time for you to finish; your advisers can help you understand the academic and financial trade-offs. The decision should draw on both. Consider an illustrative final-year student with lupus whose kidney involvement worsened in week nine of her last semester, requiring a hospital admission and a significant change in treatment. Her first instinct was to request incompletes in all five courses so that she could still graduate on time. Her disability adviser, her academic adviser, and her rheumatologist separately advised against it: her recovery would take weeks at least, and five sets of incomplete work would land on top of the treatment adjustment. She took a medical withdrawal for the semester, with a partial tuition refund under her institution's medical policy, and returned the following term with a reduced load. She graduated a semester later than planned, with her health stabilized and her grades intact. The decision felt at the time like defeat. In retrospect she regarded it as the most sensible thing she did in her degree. Return and review The end of a flare is not the end of the protocol. How a student returns shapes both the rest of the semester and the next flare. Return gradually. The instinct after a flare is to rush back to full activity to clear the backlog. That is the boom half of boom and bust, and it often triggers a relapse. Instead, return at a level below your usual baseline and build up over several days, prioritizing the most time-critical work. Your pacing diary will help you judge how quickly you can increase. Make a catch-up plan with help. List the work that has accumulated and, with your disability adviser or academic adviser if the backlog is large, decide on realistic new dates. Communicate those dates to professors in one clear message per course. It is better to propose realistic dates and meet them than to promise optimistic ones and miss them. Close the loops. Thank the people who helped, confirm new arrangements in writing, and make sure every professor knows where things stand. Then review. Once recovered, spend half an hour writing down what happened. What preceded the flare? Were there warning signs you noticed or missed? Did the flare plan work? Which parts of your accommodations were helpful and which were not? What would you do differently? This review is the most neglected step and one of the most valuable. Over several semesters, it turns individual crises into a growing understanding of your condition in the particular environment of your university, and it often reveals patterns, such as flares that consistently follow examination periods or late-night social events, that can be designed out of future semesters. If the review suggests your accommodations need to change, book an appointment with disability services while the details are fresh. Handled this way, each flare becomes part of the system's learning rather than a random disaster. The next chapter turns to the rest of university life, where many of the forces that shape flares and recovery actually operate. Chapter 8: Life Beyond the Classroom Most of what determines whether a student with a chronic illness has a good semester happens outside lectures and examinations. It happens in the residence hall at two in the morning, in the dining queue, in the pharmacy, at the party that ran later than planned, and in the private hours when the student wonders whether they belong at university at all. Accommodations and course design shape the academic side of the system. This chapter addresses the rest: where you live and eat, how you maintain healthcare away from home, how you handle friendships and social life, and how you look after your mental health and sense of self while managing a body that does not always cooperate. Housing, food, and daily logistics Where you live affects nearly every other part of managing a chronic condition. Distance from classes determines physical demand. Noise and shared facilities affect sleep. Access to a kitchen or a private bathroom may be a practical necessity. The temperature of a room matters for some conditions, and the presence of stairs for others. Most universities that provide student housing have a process for housing accommodations, usually run jointly by disability services and the housing office. Requests can include a single room, which helps with sleep, rest, and managing symptoms privately; a room near an accessible entrance, a lift, or the buildings where most classes happen; a private or nearby bathroom; air conditioning for conditions affected by heat; a kitchen or permission for a small refrigerator to store medication or special foods; and ground-floor placement. The most important practical fact about housing accommodations is that their deadlines are early, often months before the academic year begins, because room allocation happens early. A student who waits until arriving on campus to request a single room will often find none available. If housing matters for your condition, contact disability services about it as soon as you accept your place, and each year before the housing cycle begins. Food is the second daily logistic. Students on meal plans depend on dining hall hours and menus, which may not suit conditions that require particular diets, regular timing, or avoidance of specific foods. Many institutions can accommodate dietary needs, sometimes through arrangements with dining staff, sometimes by releasing a student from a compulsory meal plan so that they can cook for themselves. It is worth asking, and asking early. Whatever the arrangement, a supply of safe, easy food in your room is an important part of your flare plan, because the days when you most need to eat are often the days when getting to a dining hall is hardest. A few other logistics repay attention. If walking distances are a problem, find out whether your campus offers accessible transport or disability parking, and how to apply. If you need to carry medication or equipment, work out where you can store it safely on longer days. If your condition requires rest during the day, find a place near your classes where you can lie down, or ask disability services whether one can be arranged. None of these is glamorous, and each individually seems minor. Together they determine how much of your daily energy is available for learning rather than spent on getting through the day. Healthcare away from home Continuity of healthcare is one of the least discussed and most important parts of managing chronic illness at college. Students who move away for university often find themselves hours from the specialist who has managed their condition for years, unsure where to go when symptoms change, and dealing with prescriptions and insurance on their own for the first time. The first decision is whether to keep your existing specialist, establish care near your university, or do both. Many students keep their home specialist for long-term management, seeing them during breaks and by telemedicine where that is available and permitted, while establishing a relationship with a local clinician or the campus health service for day-to-day problems and urgent situations. What matters most is that someone near campus knows you and your condition before you need them urgently. A first appointment with a local clinician, arranged in the first weeks of term while you are well, means that when a flare comes you will call a practice that has your records rather than presenting yourself to a stranger in an emergency department. Ask your home specialist to send a summary of your condition and treatment to the new clinician, and keep a copy yourself. For students moving from paediatric to adult services, the transition may coincide with starting university, and it can be difficult. Paediatric services often involve parents closely and provide considerable coordination; adult services generally expect the patient to manage their own appointments, communicate directly, and take responsibility for treatment. Many specialist teams now run formal transition programmes, and if yours does, use it. If not, ask your paediatric team what the transition will involve and when it will happen, and try not to let it fall in the same months as your move to university. Prescriptions need planning. Find out how to transfer your prescriptions to a pharmacy near campus, how long that takes, and whether any of your medications require special handling, prior authorization, or specialist pharmacies. Build a buffer: arrive at university with enough medication to cover the gap while arrangements are made, and set reminders to reorder well before supplies run out. If you travel home during breaks, plan for medication there too. Insurance and costs vary enormously by country. In the United States, the arrangements are notably complex. Students may be covered by a parent's plan, since the Affordable Care Act generally allows young adults to remain on a parent's health insurance until the age of twenty-six; by a student health plan offered by their institution; or by other coverage. The practical question is whether your insurance works where you study, since plans often have networks of providers in particular regions and may cover out-of-area care only in emergencies. Before the semester starts, check which clinicians and pharmacies near your university are covered, and what your costs would be. Some institutions require students to have insurance and automatically enrol them in a student plan unless they can show comparable coverage elsewhere. In countries with national health systems, the equivalent question is usually how to register with a local primary care service near your university, and how that interacts with registration at home; students in the United Kingdom, for example, can generally register with a general practice near their term-time address. Keep a portable health summary. A single page listing your conditions, current medications and doses, allergies, key recent test results, your specialists' names and contact details, and any specific instructions for emergencies is invaluable. Keep it on your phone and a paper copy in your wallet or bag. If you are ever taken to an emergency department while unable to explain your history, it can speak for you. Many phones also allow medical information to be displayed on the lock screen for emergency responders. Friends, disclosure, and social life University social life is built around activities that can be hard for people with chronic illnesses: late nights, alcohol, spontaneous plans, crowded venues, and a culture in which being always available is quietly expected. Students with chronic conditions often find themselves choosing between participating and paying for it later, or protecting their health and feeling left out. There is no formula for this, but a few principles help. The first is that the pacing budget applies to social life as much as to academic work, and social life is not less important. Friendships are a central part of what university offers and a major source of the support that makes chronic illness bearable. Treating social activity as a frivolous extra that must be sacrificed to study is a mistake. The better approach is to budget for it deliberately: choosing which events matter most, going to them with a plan, and saying no to others without guilt. Going with a plan might mean deciding in advance when to leave, and leaving then even if the evening is going well. It might mean eating before going out, bringing what you need, or knowing where to sit. It might mean suggesting alternatives that suit you better, such as a meal in the afternoon instead of a night out, a film in someone's room, or a coffee rather than a party. Many friendships at university form in low-key settings, and friends who value you will adapt to plans that work for you. Alcohol deserves specific mention, since it is woven into much student social life. For many chronic conditions, alcohol interacts with medications, worsens symptoms, disrupts sleep, or is medically inadvisable. Your clinicians can tell you what applies to you. Whatever the advice, it is entirely normal to drink little or nothing at university, and far more students do so than the culture sometimes suggests. A simple, unapologetic "I'm not drinking tonight" is usually all the explanation anyone needs. Disclosure to friends raises similar questions to disclosure to professors, discussed in the fourth chapter, but with higher emotional stakes. Friends who know about your condition can support you, check on you during flares, and understand when you cancel plans. Friends who do not know may interpret cancellations as disinterest, and the friendship may drift. Most students find that telling at least a few close friends, at a level of detail that feels comfortable, makes university life considerably easier. Some prefer to tell people early and plainly; others wait until a friendship is established. Either is reasonable. What tends not to work is keeping the condition entirely secret while it repeatedly shapes your behaviour, since friends are then left to fill the gap with their own interpretations. Expect a range of responses. Some friends will be excellent, some awkward, and a few may be dismissive. It can help to be clear about what you want: "I don't need you to do anything, I just want you to know why I sometimes disappear" is a different request from "Could you check on me if I haven't replied in a couple of days?" People generally respond better to specific requests than to vague disclosures. Finally, look for others who understand. Many universities have student groups for students with disabilities or chronic illnesses, and disability services offices sometimes know of peer mentoring schemes or informal networks. Meeting other students navigating similar problems can be a significant relief, as well as a practical source of advice about which professors are helpful, which buildings have reliable lifts, and how others have handled particular offices. Family at a distance For many students, the relationship that changes most at university is the one with the parents or carers who managed their condition during childhood. Those parents may have tracked symptoms, spoken to doctors, handled school, and noticed a flare before the student did. At university, that role has to be renegotiated, and the negotiation is often awkward on both sides. Parents may worry intensely and want frequent updates, detailed symptom reports, or direct contact with clinicians and university staff. Students may want independence and resent what feels like surveillance, while also, at times, wanting exactly the help they have been refusing. Both reactions are understandable. The goal is a new arrangement that respects the student's adulthood while keeping the benefit of a family that knows the condition well. It helps to discuss the arrangement explicitly, ideally before term starts. What will you tell your family, and how often? What would you want them to do if you became seriously unwell? Would you like them to be able to speak with your clinicians or your university? In many countries, including the United States and the United Kingdom, adult students' medical and educational information is confidential, and clinicians and university staff will usually not share it with parents without the student's consent. If you want a parent to be able to speak to your doctor or your disability adviser, you will generally need to give permission in writing, and the relevant offices can tell you how. Many students choose to give limited consent, for example allowing contact in an emergency but not routine updates. Make sure that someone at the university has an emergency contact on file for you, and that the emergency contact knows the basics of your condition and has a copy of your health summary. If family live far away, consider whether a relative or family friend nearer to your university could serve as a local back-up. Finally, it is worth recognizing that parents going through this transition are also learning to let go of something they did for years, often with great care. A student who can explain calmly what they are handling themselves, and what help they would still welcome, usually finds that worried parents relax over the first year. The relationship that emerges can be one of the more valuable supports a student has, precisely because it is chosen rather than imposed. Mental health and identity Living with a chronic illness carries emotional weight, and university can add to it. Depression and anxiety are more common among people with chronic physical conditions than in the general population, and the relationship runs in both directions: illness is stressful and isolating, and low mood and anxiety can in turn make physical symptoms harder to manage and self-care harder to sustain. Students with chronic illnesses may also experience grief for the life they expected, frustration at being unable to do what peers do easily, and exhaustion from constantly explaining, planning, and adapting. None of this is a sign of weakness, and much of it is treatable. Most universities offer counselling services, and some have staff with experience of supporting students with chronic illness. Your own clinicians can also help, and many specialist teams can refer patients to psychological support specific to their condition. If you notice persistent low mood, loss of interest in things that matter to you, sleep problems beyond what your condition explains, or anxiety that is interfering with daily life, speak to someone early rather than waiting for it to become severe. Mental health conditions can also qualify for accommodations in their own right, and your disability services adviser can discuss this with you. If you are ever in crisis, having thoughts of harming yourself or feeling unable to keep yourself safe, seek help immediately. In the United States, you can call or text 988 to reach the 988 Suicide and Crisis Lifeline, or call 911 in an emergency. Other countries have their own crisis lines and emergency numbers, and it is worth finding them now and saving them in your phone. Campus counselling services often provide urgent support as well, and residence hall staff are trained to help students in distress. Beyond clinical support, many students with chronic illnesses wrestle with a quieter question of identity. Should they think of themselves as disabled? As ill? As fundamentally healthy people with a medical problem? The introduction noted that this book does not ask you to adopt any particular identity, and that remains true. But it is worth noticing that students who come to accept their condition as part of their lives, rather than as an enemy to be hidden and defeated, often find the practical work of managing it easier. Acceptance, in this sense, does not mean resignation. It means ceasing to spend energy pretending, and redirecting that energy towards building a life that fits the body you actually have. For some students that shift comes through connection with the wider disability community, whose writers and activists have long argued that the problem lies less in bodies than in environments built for only some of them. For others it comes through faith, therapy, friendship, or simply time. The practical steps in this book, including registering with disability services, pacing, designing semesters, and planning for flares, all become easier once the condition is treated as a fact to be designed around rather than a secret to be managed alone. That shift is itself one of the most important things a student with a chronic illness can accomplish at university, and it is not taught in any course. Conclusion: The Semester That Bends The argument of this book has been that managing chronic illness at college is a design problem. Having followed that argument through rights, conversations, pacing, semester planning, flares, and daily life, it is possible to say something more precise about what the design consists of, where its limits lie, and what it asks of students and of the institutions they attend. Two halves of one system The chapters on communication and the chapters on pacing may have seemed to address separate subjects. One concerned what the university owes you, the other what you owe your own body. The most important conclusion of the book is that they are not separate at all. Each half fails without the other, and the way it fails is instructive. Accommodations without pacing produce a particular kind of semester. The student has flexible deadlines and uses them, but uses them to recover from flares brought on by surges of overwork, which were themselves driven by the backlog from the last flare. Extensions accumulate. Each one is legitimate, but together they push work later into the semester until the final weeks become impossible, and the student ends with incompletes or a withdrawal despite having an excellent accommodation plan. The accommodations have been used as a way of surviving boom and bust rather than escaping it. Pacing without accommodations produces a different failure. The student is disciplined, knows their baseline, and spreads work evenly. But the semester's structure does not bend, and when a flare arrives, as it will, there is no mechanism for absorbing it. The student either pushes through, breaking the pacing that was protecting them, or falls behind with no formal protection. Careful self-management is overwhelmed by an inflexible calendar. Together, the two halves reinforce each other. Pacing reduces the frequency and severity of flares, so the accommodations are used less often and each use is more manageable. Accommodations absorb the flares that pacing cannot prevent, so the student is not forced to abandon pacing in order to meet deadlines. This is why the book has insisted that the work of arranging accommodations and the work of pacing be done together, before the semester, when there is time to think about both. Where the burden falls It would be dishonest to finish without acknowledging something uncomfortable. The system described in this book asks a great deal of the student. It asks them to register early, gather documentation, prepare functional descriptions of their own illness, negotiate with a disability office, introduce themselves to every professor, write flare plans, keep diaries, design timetables, weigh financial and visa consequences, and review each semester's performance. Much of this work falls on people who are, by definition, dealing with limited energy. Some of it would be unnecessary if institutions were designed differently from the start. There are signs that some institutions are moving in that direction. Recorded lectures, flexible deadline policies available to all students, examination processes that make deferral straightforward, and course designs that do not depend on unbroken attendance all reduce the need for individual accommodations, because they build flexibility into the default. Educators often describe this approach as universal design for learning. Where it has been adopted, students with chronic illnesses benefit, and so, frequently, do many others: students who work, students with caring responsibilities, and students who simply fall ill for a week. The case for this kind of design is strong, and students with chronic illnesses who have the energy to argue for it, through student government, disability groups, or feedback to their departments, are contributing to something that will make the work easier for those who follow them. But the pace of institutional change is slow, and students cannot wait for it. For now, the practical reality is that the student must do much of the design work themselves, using the processes that exist. That is not a statement about what is fair. It is a statement about what works in the institutions as they are. What remains uncertain Several questions remain genuinely unsettled, and a student should hold them in mind rather than assume the answers are known. The evidence on pacing varies considerably between conditions. For some, the case for energy management and the risks of overexertion are well recognized in clinical guidance; for others, gradual increases in activity are part of good treatment; for many, the evidence is limited, and patients and clinicians work partly from experience. That is why this book has repeatedly returned you to your own diary and your own clinicians. Generic pacing advice is a starting point, not a prescription. Institutional practice also varies more than the legal frameworks might suggest. Two universities operating under the same laws can differ greatly in how quickly they process registrations, how precisely they define flexible accommodations, how readily they grant medical withdrawals, and how well their faculty understand fluctuating conditions. When choosing where to study, and certainly after arriving, it is worth finding out how your institution actually behaves, not just what its policies say. And the relationship between chronic illness and academic ambition has no settled answer. Some students will find that with good design they can do everything they hoped, on the same timeline as their peers. Others will need to accept a slower pace, a different programme, or a changed sense of what success at university means. Neither outcome says anything about the student's ability or worth. Both are consequences of the particular interaction between a particular body and a particular environment. What to do differently If the book's argument is right, several things follow for how a student with a chronic illness should approach university. They are worth stating plainly, because they cut against the instincts many students bring with them. Register with disability services before you need to, and treat it as insurance rather than admission of weakness. Plan every semester around your worst realistic week, not your average one. Choose your course load by measuring your sustainable capacity, not by matching your peers. Pace deliberately, especially on good days, and trust your own records over the feeling that you could do more. Write your flare plan while you are well, and follow it when you are not. When a semester is clearly lost, choose the formal option that protects your health and your record, rather than clinging to an impossible recovery. And after every flare, every semester, and every difficult conversation, review what happened and adjust the design. These practices will outlast university. The workplace brings its own versions of the same problems: inflexible schedules, fixed expectations of availability, deadlines that cluster, colleagues who do not see what you manage. In the United States, the Americans with Disabilities Act applies to employment as well as education, requiring many employers to provide reasonable accommodations, and similar protections exist in many other countries. The skills built at university, including describing your condition functionally, negotiating arrangements in advance, pacing, and planning for disruption, transfer almost directly. Students who learn them at college enter working life with an advantage that their healthy peers do not need and do not have: a practiced ability to build a life that bends without breaking. That is the real aim of all the work this book describes. Not a semester without bad weeks, which no student with a chronic illness can guarantee, but a semester designed so that bad weeks, when they come, are absorbed rather than catastrophic. A semester that bends. Built with care on good days, it will hold on the bad ones. Notes Introduction The figure of 19 percent of undergraduates reporting a disability in 2015-16 is from the US National Center for Education Statistics, Fast Facts: Students with Disabilities, available at nces.ed.gov/fastfacts/display.asp?id=60. The broadened definition of disability, including episodic conditions and conditions in remission, comes from the ADA Amendments Act of 2008, Public Law 110-325. Chapter 2 Section 504 of the Rehabilitation Act of 1973 is codified at 29 U.S.C. 794. The Americans with Disabilities Act of 1990 is codified at 42 U.S.C. 12101 and following sections; Title II covers state and local government entities, including public colleges, and Title III covers places of public accommodation, including most private colleges. The contrast between school and college obligations draws on the US Department of Education, Office for Civil Rights, Students with Disabilities Preparing for Postsecondary Education: Know Your Rights and Responsibilities, published on the US Department of Education website. Rights over education records in the United States are set out in the Family Educational Rights and Privacy Act, 20 U.S.C. 1232g. On documentation practices, see Association on Higher Education and Disability, Supporting Accommodation Requests: Guidance on Documentation Practices (2012), available at www.ahead.org. The duty to make reasonable adjustments in Great Britain is set out in the Equality Act 2010. Australian obligations are set out in the Disability Discrimination Act 1992 and the Disability Standards for Education 2005. The provision allowing a reduced course load for F-1 students with a documented medical condition is at 8 CFR 214.2(f)(6)(iii). Guidance for students and schools is published by the US Department of Homeland Security at studyinthestates.dhs.gov. Chapter 5 National Institute for Health and Care Excellence, Myalgic encephalomyelitis (or encephalopathy)/chronic fatigue syndrome: diagnosis and management, NICE guideline NG206 (October 2021), available at www.nice.org.uk/guidance/ng206. Christine Miserandino, "The Spoon Theory" (2003), published at butyoudontlooksick.com. Chapter 8 The provision allowing young adults to remain on a parent's health plan until age twenty-six was introduced by the Patient Protection and Affordable Care Act of 2010. Details are available at www.healthcare.gov. The 988 Suicide and Crisis Lifeline launched its three-digit number in the United States in July 2022; see 988lifeline.org. Further Reading US Department of Education, Office for Civil Rights. Students with Disabilities Preparing for Postsecondary Education: Know Your Rights and Responsibilities. A short, plain-language guide to how rights change between high school and college in the United States. Essential reading in the year before starting college. Association on Higher Education and Disability (AHEAD), www.ahead.org. The professional association for disability services staff in US higher education. Its published guidance on documentation and accommodation practice explains how many offices approach requests. National Institute for Health and Care Excellence. Myalgic encephalomyelitis (or encephalopathy)/chronic fatigue syndrome: diagnosis and management (NG206, 2021).** The UK clinical guideline, with a detailed section on energy management that is useful to anyone thinking seriously about pacing, whatever their condition, though its specific recommendations apply to ME/CFS. Kate Lorig and colleagues. Living a Healthy Life with Chronic Conditions. Bull Publishing, various editions. The companion book to the Chronic Disease Self-Management Program developed at Stanford. Practical, well organized, and wide-ranging on the self-management skills that apply across conditions. Toni Bernhard. How to Be Sick: A Buddhist-Inspired Guide for the Chronically Ill and Their Caregivers. Wisdom Publications, 2010. Written by a former law professor who became chronically ill, this book addresses the emotional side of living with illness with unusual honesty and practical sense. Meghan O'Rourke. The Invisible Kingdom: Reimagining Chronic Illness. Riverhead Books, 2022. A literary and investigative account of living with poorly understood chronic illness, valuable for understanding why fluctuating and invisible conditions are so often doubted. Susan Wendell. The Rejected Body: Feminist Philosophical Reflections on Disability. Routledge, 1996. A philosophical work, written by an academic with ME/CFS, that examines how societies treat chronic illness and disability. Demanding but rewarding for students interested in the identity questions raised in the final chapter. Christine Miserandino. "The Spoon Theory." butyoudontlooksick.com, 2003. The short essay that gave many people with chronic illness a shared vocabulary for limited energy. Worth reading, and worth sharing with friends. Job Accommodation Network, askjan.org. A US service offering free guidance on workplace accommodations for specific conditions. Useful after graduation, and for students with part-time jobs or employment roles at their university. CAST. Universal Design for Learning Guidelines, udlguidelines.cast.org. The framework behind many efforts to build flexibility into courses by default. Useful for students who want to advocate for institutional change. Hashtags: #ChronicIllnessAtCollege #ChronicIllnessManagement #CollegeWithChronicIllness #StudentHealth #StudentWellbeing #ChronicIllnessAwareness #InvisibleDisabilities #DisabilityAwareness #DisabilityServices #CampusAccessibility #AcademicAccommodations #ReasonableAccommodations #DisabilityRights #AccessibleEducation #InclusiveHigherEducation #WorkloadManagement #AcademicPacing #EnergyManagement #EnergyBudgeting #SpoonTheory #BoomAndBustCycle #ChronicFatigueManagement #FlareUpManagement #AcademicFlexibility #FlexibleDeadlines #StudentSelfAdvocacy #CommunicatingWithProfessors #CollegeHealthSupport #SemesterPlanning #SustainableStudying

Download the Book (PDF): Introduction On the night of 18 March 2018, a modified Volvo XC90 operated by Uber's Advanced Technologies Group struck and killed Elaine Herzberg as she walked a bicycle across a four-lane road in Tempe, Arizona. When the National Transportation Safety Board published its findings the following year, the most unsettling detail was not that the car had failed to see her. It had seen her. The vehicle's radar and LiDAR registered an object in the roadway about 5.6 seconds before impact, at a distance where a competent human driver, or a competent machine, would have had ample time to slow. What failed was everything that happened after detection. The perception system repeatedly changed its mind about what the object was, cycling between vehicle, bicycle and "other". Each time the classification changed, the software discarded the tracking history it had built up, so it never accumulated a stable estimate of where the object was heading. Without a predicted path crossing the car's lane, the planner saw no conflict. By the time the system concluded that a collision was imminent, about 1.2 seconds remained, and a design choice intended to suppress false alarms delayed any braking response further. The vehicle's own factory emergency braking had been disabled while the development software was in control. The safety driver was looking down. That crash is worth beginning with because it overturns the most common intuition people bring to self-driving cars: that the hard part is seeing. Sensors have become remarkable. A modern automotive LiDAR returns well over a million range measurements per second; imaging radar resolves objects in elevation as well as azimuth; cameras with high dynamic range can read a road sign while looking into a low sun. Neural networks trained on millions of labelled frames detect pedestrians with a reliability that would have seemed implausible fifteen years ago. And yet Tempe happened with a sensor suite that detected the hazard in time. The failure was one of belief management: how evidence was accumulated over time, how ambiguity was represented, how uncertainty in one module propagated into decisions in another, and what the system was built to do when it was unsure. This booklet argues that autonomous driving software is best understood as a single machine for managing uncertainty, and that its safety is a property of the whole chain rather than of any one component. Perception turns photons and radio echoes into probabilistic hypotheses about the world. Sensor fusion reconciles hypotheses that disagree. Localization places the vehicle within a map that may be out of date. Prediction assigns likelihoods to what other road users will do. Planning chooses an action that remains acceptable across the plausible futures, not merely the most likely one. And the fail-safe architecture guarantees that when any of these links breaks, as some eventually will, the vehicle can still reach a condition of minimal risk. Each stage can be engineered to impressive accuracy in isolation and the system can still be unsafe, because the errors that kill are the ones that slip between stages: a confident classification that was wrong, a covariance that was too tight, a prediction that ignored the unlikely-but-possible, a fallback that assumed the primary system would always be there to hand over gracefully. Why the highway The description of this booklet points at highway autonomy, and that focus is deliberate. Highways are, in one sense, the easy case. Traffic flows in one direction per carriageway. There are no pedestrians in principle, no cyclists, no traffic lights, no unprotected left turns. Lane markings are standardized and usually well maintained. This is why the first legally approved systems that allow a driver to take their eyes off the road, beginning with Mercedes-Benz's Drive Pilot under the United Nations regulation for Automated Lane Keeping Systems, have been confined to divided motorways, and why commercial driverless trucking has started on interstate corridors such as the Dallas to Houston route in Texas. In another sense the highway is the hardest case, and it is the sense that matters most for engineering. Speed changes everything. At 120 kilometres per hour a vehicle covers a third of a kilometre every ten seconds. A heavy truck at highway speed needs well over a hundred metres to stop on dry pavement. That means perception must reach farther, with the sensors that degrade most at range; prediction must look further ahead, where uncertainty compounds; and when something fails there is no option of simply stopping where you are. A car that halts in the fast lane of a motorway has not reached safety. It has created a new hazard. The highway is where the question "what does the system do when it is no longer sure it can drive?" stops being academic, and where redundancy stops being an engineering nicety and becomes the core of the design. The highway also concentrates the long tail. Most highway driving is monotonous; the rare events, a mattress falling from a pickup, a stalled car beyond a crest, a wrong-way driver at night, a tyre carcass lying in the lane, a crash ahead seen only as a sudden wave of brake lights, are rare enough that a development fleet might drive millions of kilometres without meeting a specific variant, and consequential enough that any one of them can be fatal. Debris is a canonical highway problem precisely because it is small, low to the ground, poorly represented in training data, and has to be detected at a range where LiDAR returns are sparse and camera pixels are few. What this booklet covers, and what it does not The chapters follow the flow of information through a vehicle's software, from photon to steering command, and then step back to ask how the whole is kept safe and how anyone can know that it is. Chapter 1 frames driving as a software problem: what the stack consists of, how responsibility is divided among its modules, what the levels of automation actually mean, and why the architecture debate between modular pipelines and end-to-end learned systems matters less than the question of where uncertainty is represented. Chapter 2 examines the sensors themselves through their physics, because almost every downstream design decision traces back to what cameras, LiDAR and radar can and cannot measure. Chapter 3 follows the raw data into perception: the neural networks that detect objects, the move to bird's-eye-view representations and occupancy, and the problem of things the network has never been trained to name. Chapter 4 is the centre of the booklet: sensor fusion, the mathematics and engineering of reconciling streams that disagree into a single coherent model of the world, with tracking and data association as its heart. Chapter 5 turns to localization and maps, the problem of knowing precisely where the vehicle is and what the road ahead looks like, and the trade-off between relying on detailed prior maps and perceiving the road afresh. Chapter 6 treats prediction, the discipline of forecasting what other road users will do, which is where uncertainty grows fastest. Chapter 7 covers path planning and trajectory optimization: how a planner turns a world model and a set of forecasts into a smooth, safe, legal trajectory, and what it means for that trajectory to be optimal. Chapter 8 addresses fail-safe and redundant architecture, the design of systems that keep functioning, or degrade gracefully, when sensors, computers, power supplies or software fail. Chapter 9 asks how any of this is validated: how a developer builds an argument that the system is safe enough, given that driving enough miles to prove it statistically is impractical. The booklet does not attempt to survey every company in the industry or to forecast when robotaxis will be everywhere. It touches on urban driving where the contrast is illuminating, but the reference case throughout is the controlled-access highway. It explains algorithms in words rather than equations, though it does not shy away from describing how a Kalman filter or a model predictive controller actually works, because a reader who understands the mechanism can reason about its failures. A handful of short code fragments appear where they make a mechanism concrete. The aim is that an engineer moving into the field, a product manager, a regulator, an investor or a curious driver should finish with a working mental model of how these systems are built and, more importantly, of where they are fragile. One word on terminology before beginning. "Autonomous vehicle", "self-driving car", "automated driving system" and "driverless" are used loosely in public conversation, and they mean different things in regulation. The engineering standard that defines the vocabulary, SAE J3016, prefers "automated driving system" for the hardware and software that perform the entire driving task within some defined domain. This booklet uses that term where precision matters and the more familiar phrases where it does not. What matters is not the label but the allocation of responsibility: at any moment, who or what is accountable for noticing that something has gone wrong, and who or what is responsible for doing something about it. Almost every hard problem in this field is, underneath, a question about that allocation. Chapter 1: Driving as a Software Problem A human driver does not experience driving as a sequence of separate tasks. You look, you understand, you anticipate, you steer, all at once and largely without conscious effort. When engineers first tried to automate the task seriously, at the DARPA Grand Challenges in 2004 and 2005 and the Urban Challenge in 2007, they had to break that seamless experience into pieces a computer could handle. The decomposition they arrived at, refined over two decades, still structures almost every system on the road, and understanding it is the first step to understanding where such systems fail. The canonical stack The standard architecture divides the work into a pipeline of modules, each consuming the outputs of the one before. Sensing is the hardware layer: cameras, LiDAR, radar, ultrasonic sensors, a satellite navigation receiver, an inertial measurement unit, wheel-speed encoders and steering-angle sensors. Its output is raw data, stamped with the time at which it was captured. Perception turns raw data into a description of the environment: the positions, sizes, headings and velocities of other road users; the geometry of lanes and road edges; the state of traffic signals; the presence of static obstacles, construction zones and debris; and, increasingly, a dense estimate of which parts of the surrounding space are occupied and which are free. Localization determines where the vehicle itself is, both globally, on the surface of the Earth, and locally, relative to the lane it is in. On a highway the second matters far more than the first: knowing your latitude to within a metre is useless if you do not know whether you are in lane two or lane three. Prediction forecasts what each perceived agent will do over the next several seconds. A car in the adjacent lane with its indicator on may merge; a truck slowing on an on-ramp may or may not yield. Planning chooses what the vehicle should do. It is usually split into layers: a route planner that picks the sequence of roads, a behaviour layer that makes tactical decisions such as whether to change lanes or follow, and a motion planner that computes a specific trajectory, a curve through space and time with a speed profile attached, a few seconds into the future. Control converts that trajectory into actuator commands, steering angle, throttle and brake pressure, and corrects continuously for the gap between what was planned and what the vehicle actually does. Wrapped around all of this is a layer that rarely appears in popular explanations but is central to real systems: monitoring and safety supervision. Something has to watch the health of every sensor, computer and process, check that outputs are plausible and timely, and decide when the system as a whole is no longer fit to continue driving. This pipeline is an engineering convenience, not a law of nature, and its seams are where trouble collects. Every interface between modules is a compression. Perception does not hand the planner a picture of the world; it hands it a list of objects with attributes, or a grid of occupancy probabilities. Whatever perception decided not to report, the planner cannot reason about. If perception reports a bounding box with a velocity but not the confidence in that velocity, the planner will treat a guess as a fact. Tempe was, in part, a failure at exactly such a seam: the tracker's decision to discard history on reclassification meant the predictor never received a usable motion estimate, and the planner never received a conflict. What the levels mean The vocabulary that structures public and regulatory discussion comes from SAE International's standard J3016, first published in 2014 and revised several times since. It defines six levels of driving automation, numbered zero to five. The levels are frequently misunderstood as a ladder of capability, as if a Level 4 car were simply a better Level 2 car. They are better understood as a description of who is responsible for what. At Levels 0 to 2, the human is driving. Level 1 systems handle either steering or speed, such as adaptive cruise control. Level 2 systems handle both at once, such as the combined lane-centring and adaptive cruise systems now widely sold, but the human must supervise continuously and is responsible for detecting anything the system misses. Crucially, a Level 2 system may be quite capable; what makes it Level 2 is that its designers have not taken responsibility for monitoring the environment. At Level 3, the automated driving system performs the entire dynamic driving task within its operational design domain, and the human may take their attention off the road. But the human must remain receptive to a request to intervene and take over within a reasonable time. Level 3 is the most intellectually awkward level, because it asks a disengaged human to become an engaged driver on demand, and it puts a hard requirement on the machine: it must be able to keep driving safely for the several seconds it takes a person to re-engage, even after it has detected that something is wrong. At Level 4, the system performs the whole task within its domain and also handles its own fallback. If something goes wrong, it does not rely on a human; it brings the vehicle to what J3016 calls a minimal risk condition itself. A Level 4 robotaxi or driverless truck may have no human driver at all. Level 5 is the same without any limit on domain, a theoretical endpoint that no one claims to have reached. The concept that makes the levels precise is the operational design domain, or ODD: the specific conditions under which a system is designed to function. An ODD is defined by road type, speed range, geography, weather, lighting, traffic conditions, and anything else the system's safe operation depends upon. Mercedes-Benz's Drive Pilot, when first approved in Germany in 2021, was limited to marked motorways with a physical barrier between carriageways, in daylight or adequate lighting, in dense traffic, at speeds up to 60 kilometres per hour, with a lead vehicle to follow. The German authorities later approved operation up to 95 kilometres per hour, at the end of 2024. Those are not marketing restrictions; they are the boundaries within which the developer was prepared to accept responsibility for the outcome. The ODD is the single most important lever a developer has for making a system safe enough to deploy, because every condition excluded from it is a class of long-tail events the system never has to handle. The fallback problem A theme that will return throughout this booklet is that the difficulty of automation grows sharply at exactly the point where the human leaves the loop. A Level 2 system can be engineered to fail "safe" in a simple sense: when it is unsure, it disengages and hands control back with an alert, and the human is supposed to be there. The system does not need to be able to drive after it has failed, because someone else will. A Level 3 or Level 4 system cannot do that. When its main perception stack crashes, or a camera is blinded by spray from a truck, or the computer overheats, it must still be capable of controlling the vehicle for long enough either to hand over to a human who may be reading a book (Level 3) or to bring the vehicle to a stop in a safe place (Level 4). The engineering term is fail-operational: the system must continue to provide at least a degraded version of its function after a fault. Fail-operational design is what drives the redundant computers, duplicated power supplies, dual-wound steering motors and separate brake circuits that distinguish a serious highway automation platform from a driver-assistance package. Chapter 8 treats this in detail. The point here is that it shapes the software architecture from the start. A stack built with a single path from sensor to actuator, however accurate, cannot be made fail-operational by adding a backup at the end. Modular pipelines and end-to-end learning The past few years have seen a vigorous debate about whether the modular pipeline should survive at all. The alternative is an end-to-end system: a single large neural network, or a small number of jointly trained networks, that takes in raw sensor data and outputs a trajectory or even direct control commands, learning the intermediate representations it needs rather than having them specified by engineers. The case for end-to-end learning is serious. Hand-designed interfaces lose information, as described above; a jointly trained system can pass rich, uncompressed representations between stages and optimize them all for the final objective of driving well. Hand-written rules for behaviour are brittle and accumulate into unmaintainable thickets; a learned policy can absorb the subtlety of human driving from data. A notable academic result in this direction was UniAD, presented at the CVPR conference in 2023 and given the best paper award, which integrated detection, tracking, mapping, motion forecasting, occupancy prediction and planning into one network trained toward the planning objective. Tesla has publicly described its recent Full Self-Driving software as moving to an end-to-end neural architecture, and several companies, including Wayve in the United Kingdom, have built their approach around learned driving from the beginning. The case for retaining modularity is also serious, and it is strongest on the highway at higher levels of automation. A modular system can be tested one piece at a time. Its intermediate outputs are interpretable: an engineer can look at the object list and ask why a truck was missed. Its components can carry explicit uncertainty estimates and be checked against physical constraints. Most importantly for safety assurance, a modular system makes it possible to write requirements for each part and verify them, which is the method on which the functional safety standards described in Chapter 9 depend. A monolithic network that maps pixels to steering is hard to argue about in those terms. In practice the industry has converged on hybrids. Many current systems use large learned models for perception and prediction, often jointly trained, with learned or partly learned planners, wrapped in a layer of explicit, rule-based safety checks that can veto or override the learned output. Waymo has described research on end-to-end multimodal models while continuing to deploy a system with structured intermediate representations. The architecture question is less binary than it is often made to sound. The more useful question, and the one this booklet keeps returning to, is where uncertainty is represented and whether it survives to the point where decisions are made. A modular system that throws away uncertainty at every interface is fragile. An end-to-end system that produces a single confident trajectory with no measure of its own reliability is fragile in a different way. A system of either kind that carries an honest estimate of what it does not know, and that has a planner and a supervisor built to act on that estimate, is the one that can be made safe. Time, latency and the real-time loop One more feature of the problem shapes everything downstream: it runs in real time, on a moving platform, under hard deadlines. A typical stack runs its main loop somewhere between ten and a hundred times per second, depending on the module. Cameras commonly run at around thirty frames per second; spinning LiDARs at ten or twenty revolutions per second; radars at similar rates. Each sensor captures its data at a slightly different instant, and the vehicle is moving while it does so. Latency is not just a performance number; it is a safety parameter. At 120 kilometres per hour, a hundred milliseconds of end-to-end delay between a photon striking a camera sensor and the brakes beginning to apply corresponds to more than three metres of travel. If the perception network takes sixty milliseconds, the fusion step another twenty, prediction and planning another fifty, and the control and actuation path another fifty, the vehicle is acting on a world that is nearly two hundred milliseconds old. Every module must therefore reason not about the present but about a slightly predicted present, extrapolating its inputs forward to a common time. Getting this wrong produces subtle errors. A fusion module that combines a camera detection timestamped 40 milliseconds ago with a radar return timestamped 5 milliseconds ago, without compensating for the motion in between, will place a fast-moving car in the wrong place, and the error grows with relative speed, which is exactly what is high on a highway. This is why time synchronization, a seemingly mundane topic, receives so much attention in real systems. Sensors are typically synchronized to a common clock using protocols such as the Precision Time Protocol over the vehicle's Ethernet network, and many LiDARs and cameras can be triggered so that a camera's exposure coincides with the moment the LiDAR beam sweeps through its field of view. A system that does not know precisely when each measurement was taken cannot know precisely where anything was. One second on the motorway It helps to follow a single event through the whole stack. A car is travelling at 110 kilometres per hour, about 30.5 metres per second, in the middle lane of a three-lane motorway at dusk. Some 250 metres ahead, beyond a gentle right-hand curve, a van has stopped in the same lane with its hazard lights on after a minor collision. Traffic in the lanes either side is moving normally. The long-range radar is the first to register something: a strong return, at roughly the right range, with a closing speed equal to the car's own speed, which means the object is stationary. Taken alone, that return is ambiguous. Stationary radar returns on a curving motorway come from barriers, signs, gantries and bridge supports as well as from stopped vehicles, and at 250 metres a conventional radar's angular resolution cannot place it confidently in one lane rather than the barrier. A fraction of a second later, the forward camera's detector picks up a small vehicle-shaped patch with flashing amber lights, at a range its depth network estimates with an uncertainty of perhaps tens of metres. The LiDAR, at that distance, sees a handful of points. The fusion module must decide whether these are the same thing and whether it is real. The camera's bearing and the radar's bearing agree within their uncertainties; the camera's coarse range is consistent with the radar's precise one; the LiDAR points, though few, sit at the same place and at a height consistent with a vehicle body rather than the road surface. Over the next few cycles all three persist and remain consistent. A tentative track is confirmed: a stationary vehicle, probably in the ego lane, about 245 metres ahead. Localization, matching the curve of the barriers against the map, confirms which lane the car itself is in, and the map's lane geometry through the curve allows the system to place the van in the middle lane with growing confidence. Prediction assigns the van a near-certain forecast of remaining stationary. For the vehicles in the adjacent lanes it forecasts mostly continued travel, but with a rising probability that the car behind the van in the left lane, or the truck in the right lane, will brake or swerve as their drivers notice it. The planner considers its options: brake in lane, change to the left lane, or change to the right lane. The left lane has a fast-approaching car in the mirror; the right lane has a truck slightly behind. The planner chooses to begin a smooth deceleration in lane immediately, buying time, while evaluating a lane change into a gap that is opening on the right. The controller converts the plan into a modest brake pressure. Total elapsed time from first radar return to the beginning of deceleration might be well under a second, and at that point the van is still more than 200 metres away. Every step of that sequence involved uncertainty, and every step could have gone wrong in a way that only showed up later. If the fusion module had suppressed the stationary radar return as roadside clutter, the system would have waited for the LiDAR and camera to become confident, perhaps at 120 metres, leaving far less margin. If localization had placed the car in the wrong lane, the van would have appeared to be beside it rather than ahead. If prediction had ignored the possibility of the truck moving out of its lane, the planner might have chosen a lane change into a gap that was about to close. None of those errors would have been visible as a malfunction of the module that made it. The rest of this booklet is, in effect, an account of how each of those links is built and how each is kept honest. With the architecture in place, the natural starting point is the hardware at the front of the pipeline. The sensors are where the system meets physics, and physics sets limits that no amount of software can wholly overcome. Chapter 2: What the Sensors Can Know Every automated driving system is built on a set of bets about which sensors to use and how much to trust each one. Those bets are often presented as matters of corporate philosophy, cameras-only against LiDAR-heavy, but underneath the rhetoric they are bets about physics. Each sensing modality measures a different physical quantity, with different strengths and different blind spots, and the blind spots are the reason fusion exists. A reader who understands what each sensor actually measures can predict most of the failure modes of a perception system before seeing a line of its code. Cameras: rich, cheap and blind to distance A camera measures the intensity and colour of light arriving from each direction in its field of view. That makes it the only common automotive sensor that can read: lane markings, which are defined by paint contrast; traffic signs; the colour of a traffic light; the brake lights and turn indicators of the car ahead; the posture of a pedestrian deciding whether to step off the kerb. Cameras are also cheap, physically small and mature, with high resolution. A modern automotive camera might have eight megapixels, and a vehicle may carry anywhere from six to a dozen of them for full surround coverage. What a single camera does not measure is distance. A photograph collapses three dimensions into two. A small car close by and a large truck far away can occupy the same pixels. Depth must be inferred, either from geometry, by comparing images from two cameras a known distance apart (stereo), or by tracking how the image changes as the vehicle moves (structure from motion), or by learning, training a neural network to estimate depth from cues such as object size, texture, perspective and the position of an object's contact point with the road. Learned monocular depth has improved dramatically, and camera-only systems now estimate the range of cars at moderate distances with useful accuracy. But the error grows with range, roughly in proportion to the square of the distance for geometric methods, and it is precisely at long range, where a highway system needs to decide whether the slowing traffic three hundred metres ahead requires action now, that the estimates are weakest. Cameras are also passive: they depend on ambient light. At night they rely on headlights and street lighting, and their performance degrades. Low sun directly ahead can saturate the sensor even with high-dynamic-range imaging. Rain drops on the lens, road spray from a truck, snow, fog and dirt degrade them, sometimes abruptly. A camera has no way to measure an object's velocity directly; speed must be estimated from change in position across frames, which compounds the depth uncertainty. The most infamous camera failure in automotive history illustrates the problem of appearance-based recognition. In May 2016, near Williston, Florida, a Tesla Model S operating on Autopilot drove under the trailer of a tractor-trailer crossing the highway, killing its driver. Tesla stated at the time that neither the driver nor the system noticed the white side of the trailer against a brightly lit sky. The NTSB's investigation also emphasized that the system was a Level 2 driver-assistance feature being used outside the conditions for which it was designed and that the driver was not attending. But the perception lesson stuck: a large, flat, uniformly coloured surface at an unusual height, seen broadside, did not look like any of the things the system had been built to detect. LiDAR: geometry by the nanosecond LiDAR, light detection and ranging, sends out pulses of laser light, typically in the near infrared at around 905 or 1550 nanometres, and measures how long each pulse takes to bounce back. Because light travels about 30 centimetres per nanosecond, timing the round trip to a fraction of a nanosecond yields range to within a few centimetres. Sweep the beam across the scene, mechanically by spinning the sensor or with mirrors, or electronically with solid-state designs, and the result is a point cloud: a set of millions of three-dimensional points per second, each recording where a laser pulse hit a surface. LiDAR's great virtue is that it measures geometry directly. It does not need to infer that a box-shaped cluster of points twenty-three metres away is a car to know there is something there, twenty-three metres away, roughly 1.8 metres wide and 1.5 metres tall. That makes it the natural sensor for detecting objects of unfamiliar appearance, the debris and the unusual vehicle, because it responds to shape rather than to learned visual patterns. It works equally well in darkness, since it supplies its own illumination. Most robotaxi developers, including Waymo, use LiDAR as a primary sensor, and the highway-focused systems approved for Level 3 operation in Europe, such as Mercedes-Benz's Drive Pilot, include a LiDAR unit in the front of the vehicle. LiDAR's limitations are equally physical. The point cloud is sparse at range: the angular gap between adjacent beams is fixed, so the spacing between points on a surface grows linearly with distance. A spinning sensor with, say, 0.1 degrees of vertical resolution places beams about 35 centimetres apart at 200 metres, which means a tyre fragment lying in the road at that distance may be struck by one beam or none. Range itself is limited by how much light returns: dark, low-reflectivity surfaces such as a black car or a tyre return far less than a white car or a road sign, and specifications for maximum range are usually quoted for a target of a stated reflectivity, often ten percent, for precisely this reason. Lasers at 905 nanometres are constrained by eye-safety limits on power; systems at 1550 nanometres can emit more power because that wavelength is absorbed before reaching the retina, which extends range, at the cost of more expensive detectors. Weather hurts LiDAR in a characteristic way. Rain, fog, snow and exhaust plumes scatter the beam, producing spurious near-range points and attenuating returns from distant objects. Spray kicked up by a truck on a wet motorway can appear to a LiDAR as a solid wall of points behind the truck. Filtering such clutter without also filtering real objects is a significant part of LiDAR processing. LiDARs traditionally did not measure velocity, though frequency-modulated continuous-wave LiDAR, which measures Doppler shift, can, and several manufacturers have developed such sensors. Radar: velocity through anything Radar works on the same time-of-flight principle as LiDAR but with radio waves, in the automotive band around 76 to 81 gigahertz. Its defining strengths follow from the wavelength, about four millimetres, which is long enough to pass through rain, fog, snow, dust and darkness with little attenuation. Radar is the all-weather sensor. It also measures relative radial velocity directly and precisely, through the Doppler shift of the returned signal, so it knows immediately whether the object ahead is closing or receding, and how fast. And it reaches far: long-range automotive radars detect vehicles at two hundred metres or more. That combination of range, weather robustness and direct velocity is why radar has been the backbone of adaptive cruise control since the late 1990s. Its weakness has been angular resolution. Resolution scales with the size of the antenna relative to the wavelength, and an automotive radar must fit behind a bumper. Conventional automotive radars could tell that something was at a given range and closing speed but could localize it only coarsely in angle, and many had little or no ability to measure elevation. This created a notorious problem: a stationary object detected by radar could be a stopped car in your lane, or a metal sign gantry above the road, or a manhole cover, or a bridge, and a radar with no elevation resolution could not tell them apart. Because stationary returns from roadside and overhead infrastructure are so common, many radar-based driver-assistance systems were designed to suppress or heavily de-weight stationary objects, a sensible trade-off against constant false braking that has been implicated in collisions with stopped emergency vehicles and stationary traffic. The modern response is imaging radar, sometimes called 4D radar because it measures range, azimuth, elevation and velocity. By using many more transmit and receive channels, often arranged as a virtual array several times larger than the physical one, imaging radars achieve angular resolution of around a degree or better and can distinguish an overhead gantry from a stopped car beneath it. Their point clouds are still far sparser and noisier than LiDAR's, and radar returns suffer from multipath reflections, ghost targets created when signals bounce off guardrails or the underside of trucks before returning. But imaging radar has narrowed the gap considerably and is now a central part of many highway systems. The supporting cast Three further sensor types matter. Ultrasonic sensors, the small discs in bumpers, measure range to a few metres and are useful for parking but irrelevant at highway speed. A global navigation satellite system receiver, increasingly with real-time kinematic corrections that allow centimetre-level positioning when signals are clear, provides absolute position, though it fails in tunnels and degrades under bridges and among tall buildings. An inertial measurement unit of accelerometers and gyroscopes measures the vehicle's own acceleration and rotation at high rates, hundreds of times per second, and allows the system to estimate its motion through short gaps in every other source. Wheel-speed sensors and steering-angle sensors complete the picture of ego-motion. Together these proprioceptive sensors give the vehicle a continuous sense of how it is moving, which is essential for compensating every other measurement for motion between the moment of capture and the moment of use. The trade-offs among the principal sensors, as described in this chapter, are summarized in Table 1. No single row dominates; that is the whole point. Table 1. Principal exteroceptive sensors for highway automation. Sensor Measures directly Typical strength Typical weakness Weather tolerance Camera Light intensity and colour Semantics: signs, lanes, lights, intent cues No direct range or velocity Poor in glare, spray, dark LiDAR Range and 3D geometry Precise shape of unfamiliar objects Sparse at range, low-reflectivity targets Degraded by rain, fog, spray Conventional radar Range and radial velocity Long range, direct closing speed Coarse angle, little elevation Excellent Imaging (4D) radar Range, azimuth, elevation, velocity Separates overhead from in-lane Sparse, multipath ghosts Excellent GNSS and IMU Absolute position, ego-motion Global position, smooth short-term motion GNSS lost in tunnels, IMU drifts Unaffected by weather Calibration: the hidden sensor There is one more ingredient without which none of these sensors is useful: knowing exactly where each one is. A camera's measurements are meaningless unless the system knows its focal length, lens distortion and optical centre, its intrinsic calibration, and its precise position and orientation relative to the vehicle, its extrinsic calibration. The same applies to every LiDAR and radar. An error of a fraction of a degree in the yaw of a forward camera shifts the apparent lateral position of a car 150 metres ahead by more than a metre, enough to place it in the wrong lane. Sensors are calibrated in the factory, often using targets and turntables, but calibration drifts. A bumper is replaced after a minor collision; a mounting bracket flexes with temperature; a windscreen is changed. Production systems therefore run continuous online calibration, comparing the observations of overlapping sensors, or tracking lane markings and the horizon, to detect and correct drift. They also monitor calibration as a health signal: a camera whose observations persistently disagree with the LiDAR's in a consistent direction is probably misaligned, and a system that cannot correct this should consider that sensor degraded. Keeping the sensors able to see A sensor's specifications describe it clean, dry and new. On a working motorway it is none of those things for long. Insects accumulate on forward-facing surfaces at speed; road salt, brake dust and diesel soot form a film; spray from the vehicle ahead deposits water and grime continuously in wet weather; ice forms on sensor windows in winter. A LiDAR whose window is half covered in dried mud does not stop working; it simply returns fewer and weaker points from part of its field of view, which is exactly the kind of degradation that is hard to detect from the data alone. For this reason, serious automated vehicles treat sensor cleaning as a safety function. Robotaxis and driverless trucks carry washer nozzles and air jets aimed at each camera and LiDAR window, heated covers to shed ice and fog, and in some designs rotating or wiping covers. Some automated trucks mount sensor pods on the cab in positions chosen partly to reduce contamination and partly to see over and around the trailer. Mounting position matters for coverage too. Sensors mounted high see over the car in front and further down the road, which is valuable on a highway, but they see the ground close to the vehicle poorly and are exposed to more wind and vibration. Sensors mounted low, in the bumper, see low obstacles near the vehicle but are blocked by the vehicle ahead and are most exposed to spray. A typical highway layout therefore combines both. The system must also know when a sensor is impaired. Cameras can detect blockage from the statistics of their own images, such as loss of contrast, unchanging regions or saturated patches. LiDARs can detect it from the pattern of returns from their own window, or from missing returns in a sector where the road should be visible. Radars can detect blockage by snow or slush on the bumper from changes in their noise floor and in returns from the road. Cross-checks between sensors provide a final test: if the radar and LiDAR both see a vehicle ahead and the camera persistently does not, the camera is probably impaired, and its contribution should be down-weighted and reported. Without this self-knowledge, a degraded sensor is worse than a missing one, because the system continues to believe what it says. Why the sensor debate is really about fusion The well-known industry argument over cameras versus LiDAR is usually framed as a question of cost and capability. Tesla has argued since 2019 that cameras alone, with sufficiently capable neural networks, can match human driving, since humans drive with vision. Most other developers of highly automated systems have argued that LiDAR and radar provide independent physical measurements that vision cannot, and that independence matters for safety. The physics in this chapter clarifies what is really at stake. The argument is not about whether a camera-only system can be accurate most of the time; it plainly can. It is about failure independence. If every perception input comes from the same physical phenomenon, reflected visible light, processed by networks trained on the same data, then the conditions that blind one camera, glare, spray, an unusual object that looks like nothing in the training set, tend to blind them all at once. A radar does not care about glare. A LiDAR does not need to have seen a tyre carcass before to know there is a solid object fifty centimetres high in the lane. Combining sensors with different physics gives the system multiple, partially independent opportunities to detect a hazard, and the probability that all of them miss it together can be far lower than the probability that any one does. That reasoning only holds if the system combines its sensors well, which turns out to be much harder than it sounds, because independent sensors routinely disagree. Before reaching fusion, though, it is worth seeing how each stream is turned into objects in the first place. Chapter 3: Perception, From Signals to Scenes Perception is the stage that converts measurements into meaning. Its output is the system's answer to the question "what is out there?", expressed in a form the rest of the stack can use. For most of the history of the field that form was an object list: a set of detected things, each with a class label, a three-dimensional bounding box, a heading, a velocity and some measure of confidence. More recently it has expanded to include dense representations of the whole scene, maps of occupied and free space and of lane geometry. The transformation in how perception is done over the past decade, from hand-engineered features to large neural networks operating in a shared spatial frame, is the single largest reason highly automated driving became practical. Detection in images Camera perception rests on deep convolutional networks, and, increasingly, transformer networks, trained to detect objects. The basic form of the problem is to find every instance of a category of interest in an image, cars, trucks, pedestrians, cyclists, motorcycles, traffic cones, and draw a box around it. Two-dimensional detection in the image plane was largely solved for common categories in clear conditions by the late 2010s, through architectures such as the Faster R-CNN and YOLO families. But a box in an image is not what a planner needs. It needs to know where the object is in the world, in metres, relative to the vehicle. The direct approach is monocular 3D detection: train a network to output, for each object, not just an image box but a 3D box with its range, dimensions and orientation, learned from training data in which the ground truth was measured with LiDAR. This works surprisingly well at moderate range because the network learns powerful cues: a car's apparent size relative to its known typical size, the image row at which its wheels meet the road, the convergence of lane lines. It fails gracefully at long range, with errors growing steadily, and less gracefully with unusual objects whose typical size the network cannot know. Camera networks also do a great deal beyond object detection. Semantic segmentation assigns a class to every pixel, separating road, lane paint, kerb, vegetation, sky and vehicles. Lane detection networks output the geometry of each lane boundary as a curve, together with its type, dashed or solid, white or yellow, because the type determines what manoeuvres are legal. Traffic-sign recognition reads speed limits. Brake-light and turn-signal classification provide early hints of other drivers' intentions. On the highway, lane perception is arguably more safety-critical than object detection, since the most common automated manoeuvre, staying in lane, depends on it entirely. The bird's-eye view A conceptual shift since around 2020 has reorganized camera perception. Rather than detecting objects separately in each camera's image and then trying to stitch the results together in 3D, modern systems transform the features from all cameras into a single shared bird's-eye-view representation: a top-down grid around the vehicle, perhaps a hundred metres or more in each direction, in which each cell holds a learned feature vector describing what is at that ground location. Detection, lane estimation and other tasks then run on this unified grid. The difficulty is the transformation itself. Lifting a two-dimensional image feature into three-dimensional space requires knowing its depth, which a camera does not measure. Different methods handle this differently. The approach introduced in the Lift-Splat-Shoot paper from NVIDIA researchers in 2020 predicts, for each image pixel, a probability distribution over possible depths, and spreads the feature along the camera ray accordingly, letting the network express uncertainty about depth rather than committing to a single guess. BEVFormer, published in 2022, takes the opposite direction: it defines queries at locations in the bird's-eye-view grid and uses attention to pull in features from whichever image locations those grid cells project to, also attending to the previous frame's grid to accumulate information over time. Tesla described a similar move to a unified vector space across its cameras in public presentations in 2021. The bird's-eye view matters for two reasons beyond tidiness. First, it is the frame in which driving actually happens; lanes, vehicles and free space all live on the ground plane, and a planner reasons in exactly this space. Second, and central to this booklet, it is a frame in which data from different sensors can be combined naturally. A LiDAR point cloud can be rasterized into the same grid. Radar detections can be placed in it. The bird's-eye view became the meeting place for fusion, as Chapter 4 explains. Learning from point clouds LiDAR perception poses a different problem. A point cloud is not an image; it is an unordered set of points scattered irregularly through space, dense near the sensor and sparse far away, with large empty regions. Early methods clustered points by proximity, fitted boxes to clusters and classified them with hand-crafted features. These clustering approaches are still valuable in one respect: they find objects without needing to know what they are, which makes them a useful safety net. Learned methods took over through a sequence of influential architectures. VoxelNet, from Apple researchers in 2018, divided space into a three-dimensional grid of voxels, learned a feature for the points within each voxel, and applied 3D convolutions, effective but computationally heavy. PointPillars, published in 2019 by researchers at nuTonomy, simplified this by collapsing the vertical dimension into tall columns, or pillars, producing a bird's-eye-view pseudo-image that could be processed with fast 2D convolutions. Its speed made real-time LiDAR detection practical on automotive hardware and it became a widely used baseline. Later methods such as CenterPoint, from 2021, represented each object by its centre point in the bird's-eye view rather than by anchor boxes of predefined sizes and orientations, and tracked objects by matching centres across frames, improving both accuracy and simplicity. LiDAR detection is generally more precise in 3D than camera detection, especially for position and size, because it measures geometry rather than inferring it. Its difficulty is classification at range, where a distant pedestrian may be a dozen points and a distant motorcyclist barely more, and with small objects close to the ground. Highway debris detection is a classic hard case: a piece of tread lying flat may present only a few points just above the ground plane, and separating those from the road surface itself, which is being hit by many beams at grazing angles, requires careful ground-plane estimation. Occupancy: perceiving what has no name All the detection methods above share a structural weakness. They find instances of categories they were trained on. A detector trained to find cars, trucks, pedestrians and cyclists will, at best, ignore a mattress in the road, and at worst try to force it into one of those categories, perhaps with low confidence that is then filtered away. The world's long tail is full of things that have no category: a ladder, a sofa, a fallen tree branch, an overturned trailer, a load of gravel spilled across two lanes, a deer. On a highway at speed, these are often the most dangerous objects precisely because the system was never explicitly taught about them. The response has been to augment object detection with occupancy prediction: estimating, for every cell of a three-dimensional voxel grid around the vehicle, the probability that it is occupied by something solid, regardless of what that something is. Occupancy grids are an old idea in robotics, dating to work by Hans Moravec and Alberto Elfes in the 1980s, but they were traditionally built from range sensors. The modern variant, sometimes called an occupancy network, predicts occupancy from cameras, LiDAR or both using learned models, and often predicts the motion of occupied cells as well, so that it captures moving objects without classifying them. Tesla publicly described an occupancy network in its 2022 presentations, and occupancy prediction has become a standard research benchmark. The planner can then reason in two complementary ways: about named agents, for which it has rich models of behaviour, and about generic occupied space, which it must simply avoid. A sound highway system treats occupancy as a floor beneath object detection: if something solid is in the lane, the car must respond to it whether or not any classifier can say what it is. The arithmetic of range How far does a highway system need to perceive? The answer comes from simple kinematics, and it is sobering. Suppose a passenger car is travelling at 120 kilometres per hour, a little over 33 metres per second, and must come to a stop for a stationary obstacle in its lane. If the system reacts with a total latency of half a second from first reliable detection to the onset of braking, the car covers about 17 metres before braking begins. Braking at a firm but not violent 5 metres per second squared, roughly half a g, which passengers will find uncomfortable but not alarming, takes about 110 metres to stop. The total is around 127 metres. Braking at the limit of a good tyre on dry asphalt, close to 9 metres per second squared, reduces the stopping distance to roughly 60 metres, but that is an emergency stop that risks a collision from behind and is unavailable on wet roads, where available deceleration may be halved. So the system needs reliable detection of stopped obstacles at well over 130 metres just to stop comfortably for them on a dry road, and considerably more to stop smoothly enough that following traffic is not endangered, or to change lanes instead. For a loaded heavy truck, whose braking is much weaker, the figures are far larger; reliable detection beyond 250 metres is a common design goal for automated trucking. Those are exactly the ranges at which, as Chapter 2 described, LiDAR becomes sparse, camera depth estimates become poor and conventional radar struggles to separate the lane from the roadside. This arithmetic shapes perception design in several ways. It is why highway systems often use a narrow-field, long-focal-length forward camera alongside wide-angle ones, trading field of view for resolution at range. It is why LiDAR designers concentrate beams around the horizon, where distant objects appear, rather than spacing them evenly. It is why imaging radar matters so much on highways. And it is why perception at long range must be treated explicitly as probabilistic: the system will have to begin responding to objects it is not yet sure about, because by the time it is sure, it may be too close to respond comfortably. Debris is the hardest case of all. A piece of truck tyre tread may be less than 20 centimetres high. Seen from a camera mounted at windscreen height 150 metres away, it subtends a tiny angle, a few pixels tall even with a telephoto lens, and it is dark against a dark road. A LiDAR with 0.1 degrees of vertical spacing places its beams more than 25 centimetres apart at that range, so the object may be struck by a single beam or not at all. Detecting such debris reliably at the range needed to stop or change lanes comfortably remains one of the open problems of highway perception, and it is one reason the systems that have been approved for eyes-off operation began with low speed limits. Uncertainty and the confident wrong answer Neural networks produce outputs with confidence scores, and it is tempting to read those scores as probabilities. They are not, or not reliably. A classifier's softmax output measures how strongly the input resembles the training examples of each class relative to the others; it says little about how far the input is from anything in the training data. A network can assign 0.98 confidence to "car" for an object that is in fact a painted advertising image of a car on the side of a truck, or to "plastic bag" for a rock. Researchers distinguish two kinds of uncertainty. Aleatoric uncertainty is inherent noise in the measurement: a distant object really is ambiguous in a low-resolution image, and more training will not fix that. Epistemic uncertainty is the model's ignorance: the input lies outside what it has learned. Aleatoric uncertainty can be learned directly, by training networks to output a variance alongside each estimate. Epistemic uncertainty is harder, and practical methods include ensembles of independently trained networks, whose disagreement signals unfamiliar inputs, and dedicated out-of-distribution detectors. None is perfect, and the honest state of the art is that a perception network's confidence in an unfamiliar situation should be treated with suspicion. This is the deep reason that the Tempe failure was not simply a perception bug. The NTSB found that the Uber system's classification of Elaine Herzberg changed multiple times, and that each reclassification discarded the object's tracking history, so the system could not build a usable estimate of where she was going. A classifier that genuinely does not know what something is should say so, and the downstream stack should treat "I see something solid, I do not know what it is, and its motion is uncertain" as a reason for caution, not as a reason to wait for a confident label. Where perception's outputs feed a tracker that discards history on every relabelling, the uncertainty is not merely unrepresented; it is actively destroyed. Training data and the data engine Perception quality depends overwhelmingly on training data, and much of the engineering effort in the field goes into building what Tesla's former AI director Andrej Karpathy called a data engine: a loop in which the deployed fleet identifies situations where the network performs poorly, those situations are collected and labelled, the network is retrained, and the improved model is shipped and the loop repeats. The trigger for collection may be a disagreement between the network and a human driver's actions, a disagreement between sensors, a low-confidence output, or an engineer's query for a specific scenario such as "motorcycles lane-splitting at night". Public datasets have shaped the research field. KITTI, recorded in Karlsruhe and released in 2012, was the first widely used benchmark for driving perception. nuScenes, released by nuTonomy in 2019, added full 360-degree camera, LiDAR and radar coverage with annotations. The Waymo Open Dataset, released in 2019 and described in a CVPR paper in 2020, provided high-quality LiDAR and camera data from a production-grade sensor suite at much larger scale. These datasets made it possible to compare methods directly, and much of the progress described in this chapter was measured on them. They also illustrate the limits of benchmark progress: a network that improves average precision on a public dataset by a few points may be no better at the rare failure that matters on the road, because the rare failure is barely represented in the benchmark. Labelling has itself been transformed. Manual annotation of 3D boxes in point clouds is slow and expensive. Developers increasingly use auto-labelling: large offline models, running with no real-time constraint, with access to the full recording including the future, produce high-quality labels that are then used to train the smaller real-time networks that run in the car. An offline model can look at an object across its entire trajectory and see it clearly when it passes close by, then propagate that knowledge back to the frames where it was a distant smudge. This is a powerful technique, and it underlines a theme: the system in the car is always working with less information than the system that taught it, and part of good engineering is making sure the car knows what it does not know. Perception, done well, delivers an honest but imperfect set of hypotheses from each sensor stream. Those hypotheses will often disagree. The camera sees a car at 80 metres; the radar sees a return at 74 metres closing at 12 metres per second; the LiDAR sees a sparse cluster at 76 metres; the occupancy network sees something solid but unclassified nearby. Turning that cacophony into one coherent belief about the world is the work of sensor fusion. Chapter 4: Sensor Fusion, Building One World from Many Sensor fusion is the discipline of combining measurements from multiple sources into a single estimate that is better than any of them alone. Put that way it sounds like averaging, and in the simplest cases it nearly is. In a moving vehicle surrounded by moving objects, observed by sensors that sample at different times, with different geometry, different noise, different failure modes and different notions of what an object is, it becomes one of the hardest problems in the stack. It is also the place where the controlling idea of this booklet is most visible. Fusion is not about collecting more data. It is about maintaining a calibrated belief: knowing not only where things probably are but how sure the system should be, and updating that confidence honestly as evidence arrives. The core idea: beliefs, not measurements The conceptual foundation of fusion is probabilistic estimation, and its classic text is Probabilistic Robotics by Sebastian Thrun, Wolfram Burgard and Dieter Fox, published in 2005. Thrun led the Stanford team whose car, Stanley, won the 2005 DARPA Grand Challenge, and went on to found Google's self-driving project, which became Waymo. The book's central premise is that a robot should never treat its estimate of the world as a fact. It should maintain a belief, a probability distribution over possible states of the world, and update that belief in two alternating steps. The prediction step moves the belief forward in time using a model of how things change. If a car was at a certain position travelling at 30 metres per second, then a tenth of a second later it is probably three metres further on, though slightly less certainly so, because it may have accelerated or braked. Prediction always increases uncertainty: the future is less known than the present. The update step incorporates a new measurement. A radar return arrives saying the car is at a particular range. The belief is revised to combine what was expected with what was observed, weighting each by its reliability. Update always decreases uncertainty, by an amount that depends on how informative the measurement is. The engine that implements this loop for most tracking problems is the Kalman filter, published by Rudolf Kálmán in 1960 and used in the Apollo navigation computers within the decade. The Kalman filter represents a belief as a Gaussian distribution, a mean estimate and a covariance matrix describing the uncertainty in each dimension and the correlations between them. Its update has an intuitive heart. The filter computes a gain that decides how far to move the estimate toward a new measurement, and that gain is large when the current estimate is uncertain and the measurement is precise, and small when the estimate is confident and the measurement is noisy. Fusion falls out naturally: a radar measurement with precise range but poor lateral accuracy will correct the range of a tracked car strongly and its lateral position barely at all; a camera measurement with good lateral angle but poor range will do the reverse. Over successive updates the filter builds an estimate that is precise in every dimension, even though no single sensor was. A minimal illustration of the update for a single quantity, say the range to the car ahead, conveys the mechanism. # belief: mean x, variance P; measurement z with variance R K = P / (P + R) # gain: trust measurement more when P >> R x = x + K * (z - x) # move estimate toward measurement P = (1 - K) * P # uncertainty shrinks after update Real systems track many quantities at once: position, velocity, acceleration, heading, turn rate, and object dimensions. The relationships between them are non-linear; a car turning in an arc does not move in a straight line, and a radar measures range and angle rather than Cartesian coordinates. The extended Kalman filter handles non-linearity by linearizing the models around the current estimate; the unscented Kalman filter propagates a small, carefully chosen set of sample points through the non-linear functions, which is often more accurate. Where the belief is not well described by a single Gaussian, for instance when an object may be in one of two quite different places, particle filters represent the belief with thousands of weighted samples. On the highway, the Kalman family dominates for object tracking because highway motion is relatively smooth and well modelled. The single most important property of a filter in a safety-critical system is that its covariance be consistent: that the uncertainty it reports matches the errors it actually makes. A filter that is overconfident, reporting a tight covariance when its errors are larger, will reject valid measurements as outliers and fail to react when an object does something unexpected. A filter that is underconfident will be jittery and slow to converge. Tuning filters for consistency, using real data and statistical tests on the residuals between predicted and observed measurements, is unglamorous work that matters enormously. Data association: which measurement belongs to which object The Kalman filter assumes it knows which measurement belongs to which object. In reality, that is the hardest part of the problem. Every sensor cycle produces a fresh batch of detections, from multiple sensors, with no labels saying "this is the blue car you were tracking". The tracker must decide which detection corresponds to which existing track, which detections are new objects that need new tracks, which are false alarms to be ignored, and which existing tracks received no detection this cycle because the object was occluded or missed. This is data association, and it has a long literature from air and missile defence, where tracking many targets with noisy radar was a central problem decades before cars. The textbook treatment by Yaakov Bar-Shalom and colleagues remains standard. The simplest approach, global nearest neighbour, assigns detections to tracks to minimize total distance, usually measured by the statistical Mahalanobis distance that accounts for each track's covariance, and solves the assignment with the Hungarian algorithm. It works well when objects are well separated. In dense traffic, with cars close together in adjacent lanes and frequent occlusions, a single wrong assignment can swap two tracks' identities or merge two vehicles into one. More robust methods hedge. Joint probabilistic data association does not commit to a single assignment; it updates each track with a weighted combination of all plausible detections, weighted by the probability that each is the right one. Multiple hypothesis tracking maintains several alternative assignment histories and defers the decision until later evidence resolves the ambiguity, pruning hypotheses that become improbable. A more recent family, based on random finite sets and developed by Ronald Mahler and others, treats the entire set of objects as a single random variable, which elegantly handles uncertainty about how many objects exist at all. Learned approaches now also play a role, with neural networks predicting association scores or performing tracking directly in the bird's-eye-view feature space across frames. Track management is the policy layer around association. A new detection that matches no existing track becomes a tentative track, which must be confirmed by further detections before the planner treats it as real. An existing track that receives no detections is coasted forward on its prediction for some time, its uncertainty growing, before being deleted. These thresholds encode a fundamental trade-off. Confirm tracks too quickly and false alarms from spray, ghosts and shadows cause phantom braking. Confirm them too slowly and a real hazard is ignored for precious fractions of a second. Delete tracks too quickly and an occluded car that reappears is treated as new, losing its velocity history; delete them too slowly and ghosts linger. The Tempe crash belongs here. A tracker that resets an object's history whenever its class label changes has coupled classification and tracking in exactly the wrong way. The geometry of the object, its position and motion across successive LiDAR and radar frames, did not change when the classifier vacillated. A sound design tracks the geometry persistently and treats the class as one attribute of the track, itself estimated probabilistically, rather than as the identity of the track. Where to fuse: early, intermediate and late Fusion can happen at different points in the pipeline, and the choice is one of the main architectural decisions in a perception system. The options differ in what information survives, how independent the sensors remain, and how hard the system is to build and verify, as Table 2 summarizes. Table 2. Levels at which sensor data can be fused. Fusion level What is combined Main advantage Main drawback Early (raw data) Points, pixels, radar returns Nothing discarded before combining Requires precise calibration and sync; sensors lose independence Intermediate (feature) Learned features, often in bird's-eye view Rich information, jointly learned Hard to interpret; one network can fail for all sensors Late (object) Each sensor's own detections and tracks Sensors stay independent; easy to verify Information lost before fusion; disagreements hard to resolve Late fusion was the dominant production approach for many years. Each sensor runs its own detection and often its own tracking, producing an object list, and a central fusion module associates and combines the object lists, typically with the Kalman-family machinery described above. Its great virtue is modularity and independence: if the camera pipeline fails, the radar pipeline still produces objects, and the fused output degrades rather than vanishing. Each sensor's pipeline can be developed and validated separately. Its weakness is that each pipeline must decide on its own whether something is an object, before seeing what the other sensors saw. A faint LiDAR cluster and a faint radar return at the same location may each fall below their own detection thresholds and both be discarded, even though together they would be convincing evidence. Early fusion combines raw data before detection. The simplest form projects LiDAR points into camera images, so that each pixel near a LiDAR point acquires a depth, and runs detection on the enriched image; or, conversely, paints LiDAR points with the semantic class predicted by a camera network, as the PointPainting method of 2020 did. Early fusion preserves the most information but demands extremely precise calibration and time synchronization, since a few centimetres or milliseconds of error misaligns the data. It also makes the sensors interdependent: a fused detector that has learned to rely on both inputs may perform poorly when one is degraded. Intermediate or feature-level fusion has become the leading research approach and is increasingly used in production. Each sensor's data passes through its own neural encoder into a shared representation, most often the bird's-eye-view grid described in Chapter 3, where features are combined and passed to shared detection and prediction heads. BEVFusion, a 2022 method from researchers at MIT, showed that fusing camera and LiDAR features in a unified bird's-eye view improved both 3D detection and map segmentation, and similar architectures followed. Transformer-based methods let the network learn which sensor to attend to at each location, so that it can, in principle, lean on radar in fog and on cameras for lane semantics. The engineering trade-off is between information and independence. The more deeply the sensors are fused, the more the network can exploit their complementarity, and the less the system retains independent paths that fail separately. Most serious highway systems resolve this by doing both: a deeply fused primary perception system optimized for accuracy, and one or more independent, simpler paths, perhaps a radar-and-camera late-fusion tracker or a LiDAR-only obstacle detector, that run in parallel to provide a check. This pattern recurs in Chapter 8, where it becomes the basis of redundancy. When sensors disagree The real test of a fusion system is not combining sensors that agree but deciding what to do when they do not. Disagreements arise constantly, and their causes vary. Sometimes one sensor is simply wrong: a radar ghost from a guardrail reflection, a LiDAR "object" made of spray, a camera false positive on a picture of a pedestrian painted on the back of a van. Sometimes one sensor sees what the others cannot: the radar detects a stopped car beyond a crest that the LiDAR and cameras cannot yet see, because radar waves can bounce beneath the vehicle ahead. Sometimes a sensor has degraded: a camera lens is smeared with road grime, or a LiDAR's window is iced. A principled fusion system distinguishes these cases using several kinds of evidence. Physical plausibility is one: an object that appears in LiDAR with no thickness and moves exactly with a truck's spray plume is probably spray. Temporal consistency is another: a real object persists and moves smoothly across frames, while many ghosts flicker. Cross-sensor geometry is a third: a radar return at a location where LiDAR shows clear road all the way to the ground beyond it is suspect, unless the radar return is at a height the LiDAR could not see. Sensor health monitoring provides the fourth: if a camera's images have lost contrast and its detections have stopped matching the LiDAR's for several seconds, the system should reduce that camera's weight and report reduced capability, rather than averaging its bad data into the fused result. The deepest design question is what a disagreement should mean to the rest of the system. Consider a highway at night: the radar reports a strong stationary return in the ego lane 150 metres ahead; the camera sees nothing unusual; the LiDAR, at that range, has too few points to say. Is it a stopped car with its lights off, or a metal expansion joint in the road? Suppressing the radar return, as many early driver-assistance systems did with stationary objects, avoids false braking at the cost of occasionally ignoring a real hazard. Braking hard on it creates the opposite risk, a sudden stop that a following driver may not anticipate. A well-designed system does neither reflexively. It carries the ambiguity forward as an uncertain hypothesis, perhaps easing off the accelerator to buy time and distance while the LiDAR and camera close in and resolve it. The ability to express "something may be there" and to plan for it proportionately is precisely what fusion should give the planner, and it is lost if fusion forces every disagreement into a binary yes or no. The world model as the product The output of fusion, together with localization and maps, is often called the world model: the vehicle's current best belief about its surroundings, including tracked objects with states and covariances, occupancy, lane geometry, drivable space, traffic controls and the vehicle's own state. The world model is the product of everything before it and the input to everything after it. Its quality is not measured by how often it is right in the easy cases but by how honest it is in the hard ones: whether it knows when it is unsure, and says so in a form the planner can use. With that world model in hand, the next questions are where, precisely, the vehicle is within it, and what the road ahead looks like beyond the reach of the sensors. Chapter 5: Knowing Where You Are A human driver on a motorway rarely thinks about localization. You know which lane you are in because you can see it, and you know roughly where you are on the route because of signs and familiarity. An automated system needs both kinds of knowledge with a precision and reliability that people never consciously achieve. It needs to know its position within its lane to within a few centimetres, its heading to a fraction of a degree, and which lane it occupies with near certainty, at every instant. And it often wants to know what lies beyond the reach of its sensors: that the road curves sharply in 400 metres, that the left lane ends in a kilometre, that there is a toll plaza after the next junction. Localization and mapping are how a vehicle answers those questions, and the trade-off between relying on stored maps and perceiving the road afresh is one of the defining design choices in the field. Why satellite positioning is not enough The obvious answer to "where am I?" is satellite navigation, and it is part of the answer. A standard GNSS receiver using signals from the GPS, Galileo, GLONASS and BeiDou constellations typically achieves a few metres of accuracy in open sky. That is plenty for turn-by-turn navigation and nowhere near enough for driving: a lane on a motorway is about 3.5 to 3.75 metres wide, so a two-metre error is the difference between being centred and being on the line. Precision can be improved greatly. Real-time kinematic positioning uses corrections from nearby reference stations and measurements of the carrier phase of the satellite signal itself, not just the timing codes it carries, to achieve centimetre-level accuracy. Precise point positioning uses corrections for satellite orbits and clocks broadcast globally, reaching decimetre or better accuracy after a convergence period. Several production driver-assistance and automated systems use corrected GNSS as one input to localization. But GNSS has a property that makes it unsuitable as a primary localization source for automated driving: it fails in ways the receiver cannot always detect. In tunnels the signal is lost entirely, which at least is obvious. Under bridges, beside tall trucks and in cuttings, signals are partially blocked. Near buildings and large vehicles, signals reflect, creating multipath errors in which the receiver measures a path longer than the true one and computes a position that may be metres off while reporting high confidence. Signals can also be jammed, by cheap illegal devices sometimes used by drivers to defeat fleet tracking, or spoofed. An automated system that trusts GNSS without cross-checking it against other evidence is building on sand. Dead reckoning and the inertial backbone The complement to GNSS is dead reckoning: estimating motion from the vehicle's own sensors. An inertial measurement unit measures linear acceleration along three axes and rotation rate about three axes, hundreds of times per second. Integrate rotation rate once and you get heading; integrate acceleration twice and you get position. Wheel-speed sensors give distance travelled, and steering angle or a yaw-rate gyroscope gives turning. Combined in a Kalman filter, these sources give a smooth, high-rate estimate of how the vehicle is moving that is unaffected by weather, tunnels or anything outside the car. The weakness of dead reckoning is drift. Small biases in the accelerometers and gyroscopes integrate into growing errors: a tiny constant error in measured acceleration becomes a position error that grows with the square of time. Automotive-grade inertial units drift too quickly to be trusted alone for more than a few seconds to tens of seconds at the precision needed for lane keeping, though higher-grade units extend that. Wheel odometry suffers from tyre wear, pressure changes and wheel slip. Dead reckoning is therefore the glue that holds localization together between absolute fixes, rather than a solution in itself. Its importance in fail-safe design will reappear in Chapter 8: when every exteroceptive sensor fails, the inertial estimate is what allows the vehicle to keep its lane for the few seconds needed to stop. Localizing against a map The approach that most robotaxi developers adopted, and that many highway systems use in some form, is to localize the vehicle by matching what its sensors see against a previously recorded map. If a survey vehicle has driven the road before and recorded a detailed three-dimensional model of its surroundings, the kerbs, barriers, poles, signs, bridge piers and the texture of the road surface, then a vehicle driving the same road can compare its current LiDAR point cloud with the stored map and find the position and orientation at which they best align. This is map-relative localization, and done well it achieves accuracy of a few centimetres, independently of GNSS. The classic algorithms for this alignment include the iterative closest point method, which repeatedly pairs each observed point with the nearest map point and solves for the rigid transformation that minimizes the distances, and the normal distributions transform, introduced by Peter Biber and Wolfgang Strasser in 2003, which represents the map as a grid of local Gaussian distributions and finds the pose that makes the observed points most probable under them. The normal distributions transform is smoother and more robust to the sparse, uneven sampling of LiDAR and is widely used, including in the open-source Autoware stack. Camera-based map localization matches observed visual features or lane markings against the stored map. On a highway, map-relative localization faces a particular difficulty: the world is self-similar. A long stretch of motorway with concrete barriers on both sides, uniform lane markings and no distinctive structures looks almost identical from one hundred-metre segment to the next. Lateral position and heading can be pinned down well, because the barriers and markings constrain them, but longitudinal position, how far along the road the vehicle is, may be poorly constrained for long stretches. Systems handle this by combining map matching with GNSS and odometry, each constraining the dimensions where the others are weak, which is fusion again, applied to the vehicle's own state. High-definition maps and their discontents The maps used for automated driving are far richer than navigation maps. A high-definition map typically contains, for every road segment, the precise geometry of each lane centre-line and boundary to within roughly ten centimetres; the type of each lane boundary; the connectivity of lanes through junctions and interchanges, including which lanes may be entered from which; the positions of signs, signals and stop lines; speed limits; and often a dense three-dimensional layer used for localization. In effect, an HD map is a prior: knowledge about the road, gathered in advance, that lets the vehicle know things its sensors cannot yet see and resolve ambiguities its sensors cannot resolve. The benefits are substantial. A planner that knows the lane geometry 500 metres ahead can plan lane changes for an upcoming exit long before the exit is visible. Perception can use the map to know where to look and what to expect: a traffic signal should be at this position; this lane ends here. Prediction can use lane connectivity to forecast that a car in an exit lane will probably take the exit. Mercedes-Benz's Drive Pilot, for example, uses a high-precision map as part of its localization and operates only on mapped motorway sections. The costs are equally substantial. HD maps are expensive to build, requiring survey vehicles to drive every road and significant processing and quality assurance. They are expensive to maintain, because roads change. Construction zones shift lanes overnight; lines are repainted; barriers are moved; new junctions are opened. And the moment a map is out of date, it becomes not a help but a hazard, because the vehicle may trust its prior over its own eyes. A system that believes, because the map says so, that the lane continues straight, when temporary markings divert traffic to the left, can steer confidently into a barrier. The response has developed in two directions. The first is to keep maps fresh by using the production fleet itself as a mapping network: vehicles detect discrepancies between what they see and what the map says, report them, and the map is updated, sometimes within hours. The second is to rely on maps less. Several developers have moved toward map-light or mapless approaches in which the vehicle perceives lane geometry and topology online, using learned networks that predict lane graphs in the bird's-eye view, with only a lightweight navigation map for routing. Research methods such as HDMapNet in 2021 and MapTR in 2022 learned to produce vectorized map elements directly from sensor data in real time. Tesla has long argued for a mapless approach for scalability. The question is not maps versus no maps. It is how much weight the system should give to its prior relative to its live perception, and how it detects when the two conflict. That is, once again, a question of uncertainty management. A well-designed system treats the map as one more sensor, a sensor whose readings were taken some time ago, whose reliability declines with age and in areas flagged as likely to change, and whose disagreement with live perception is itself an important signal. When the map says three lanes and the cameras see two plus a row of traffic cones, the correct inference is not that one of them is broken but that the world has changed, that the system is probably in a construction zone, and that it may be approaching the edge of its operational design domain. Through a tunnel: localization as fusion A motorway tunnel shows how the pieces of localization work together, and what happens when several of them falter at once. As the vehicle approaches the portal, it has a good estimate: corrected GNSS agrees with LiDAR map matching to within ten centimetres, and the cameras see lane markings exactly where the map predicts. Inside, the GNSS signal disappears within a second or two. That is not a problem in itself; the localization filter simply stops receiving GNSS updates and relies on the others. Inside the tunnel, though, LiDAR map matching faces the self-similarity problem in its most extreme form. The walls are smooth and parallel, the lighting fixtures repeat at regular intervals, and nothing distinguishes one ten-metre section from the next. Lateral position and heading remain well constrained, because the walls and lane markings fix them. Longitudinal position, how far into the tunnel the vehicle is, is now constrained mainly by wheel odometry and the inertial unit, and its uncertainty grows steadily with distance. A well-designed filter reports this honestly: its lateral uncertainty stays at a few centimetres while its longitudinal uncertainty grows to a metre, then several metres. For the immediate task of keeping lane and following traffic, that longitudinal uncertainty barely matters. It matters a great deal for anything that depends on knowing where along the road the vehicle is: the location of a lane drop shortly after the tunnel exit, or of a junction whose lane assignments the planner needs to prepare for. A system that knows its longitudinal uncertainty has grown can respond sensibly, perhaps by waiting until it can see the lane drop rather than relying on the map's position for it. On exit, the vehicle experiences a second hazard: the cameras, adjusted to the tunnel's dim lighting, are briefly dazzled by daylight, while GNSS takes a moment to reacquire. For a few seconds the system may have both its best absolute reference and its best semantic sensor impaired at once. The inertial estimate and LiDAR map matching carry it through, and the filter's uncertainty bounds tell the planner how much to trust the result. The scenario is mundane, repeated thousands of times a day on any motorway network, and it is a good test of whether a localization system's uncertainty estimates are trustworthy. Localization integrity Aviation offers a concept that automated driving has adopted: integrity. It is not enough for a positioning system to be accurate on average; it must also provide a trustworthy bound on its own error, and warn promptly when that bound can no longer be guaranteed. In aviation, the protection level is a computed radius within which the true position is guaranteed to lie with a specified, extremely high probability, and an alert must be raised within a specified time if the error might exceed an alert limit. Applied to driving, integrity means the localization system must produce not just a position but a reliable estimate of how wrong that position could be, laterally and longitudinally, and must notice when its sources disagree. If map matching says the vehicle is centred in lane two, GNSS says it is half a metre to the left, and the cameras see the lane markings exactly where the map-matched position predicts, the system can be confident. If map matching and GNSS diverge by two metres and the cameras see no markings because of snow, the system's honest answer is that it does not know which lane it is in. That answer must propagate. A planner that does not know which lane it is in should not change lanes, and a supervisor that sees localization integrity degrade toward the limit of safe operation should begin a controlled handover or a stop. Localization thus fits the pattern that runs through the whole stack. The goal is not an estimate that is always right; no such estimate exists. It is an estimate that is honest about its errors, built from sources whose failures are as independent as possible, and connected to a system that knows what to do when the honest answer becomes "not sure enough". With perception, fusion and localization giving the vehicle a calibrated picture of the present, the next step is to anticipate the future, and the future is where uncertainty grows fastest. Chapter 6: Predicting Other Road Users Driving is a social activity carried out at speed. Every decision a driver makes depends on guesses about what others will do: that the car ahead will keep its lane, that the truck on the slip road will slow to merge behind you, that the car approaching fast in the outside lane will not cut across into your gap. People make these guesses fluently and mostly unconsciously, drawing on years of experience of how other drivers behave, and they make them in both directions, anticipating others while signalling their own intentions. An automated system has to do the same thing explicitly. Prediction, also called motion forecasting, is the module that turns the world model's estimate of the present into a set of possible futures, and it is where uncertainty compounds most quickly. Why prediction is hard The difficulty of prediction is easy to underestimate. Tracking, discussed in Chapter 4, already contains a prediction step: the Kalman filter extrapolates each object's state forward to the next sensor cycle, a tenth of a second or so ahead, using a simple motion model such as constant velocity or constant turn rate. Over a tenth of a second that is entirely adequate. The planner, however, needs to know what other vehicles will do over the next several seconds, typically five to eight on a highway, because that is the horizon over which its own trajectory must be safe. Over that horizon, simple extrapolation fails badly. A car travelling straight at constant speed might continue doing so; it might also brake, change lanes, or take an exit. The future is not a slightly blurred version of the present. It branches. Three properties make the branching difficult. First, the future is multimodal: there are several distinct likely outcomes, not one outcome with noise around it. A car approaching an exit will either take it or not, and the average of those two futures, a car driving along the gore area between them, is the one thing that will certainly not happen. A predictor that outputs a single mean trajectory will systematically produce physically absurd forecasts at exactly the decision points that matter. Second, behaviour is interactive. What the car in the next lane does depends on what the automated vehicle does, and vice versa. A merging truck will slow if the automated car holds its speed, and accelerate into the gap if the car backs off. Predicting each agent independently ignores the most important source of structure in the scene. Third, intention is latent. A driver's goal, to take the next exit, to overtake, to pull onto the hard shoulder, is not directly observable. It can only be inferred from cues: position within the lane, slight drifts toward a boundary, indicator lights, deceleration, the map's lane topology, the driver's earlier behaviour. These cues are weak, often ambiguous and sometimes misleading; plenty of drivers change lanes without signalling and plenty leave indicators on for kilometres. From physics to learned forecasts Prediction methods developed in roughly three stages, all of which remain in use in some form. The first stage was physics-based: extrapolate each agent's state using kinematic models. Constant velocity, constant acceleration and constant turn rate models are simple, interpretable and accurate for the first second or so, and they remain the right baseline for agents that are poorly understood, such as debris rolling across the road or an animal. The second stage was manoeuvre-based: recognize the set of discrete manoeuvres an agent might perform, keep lane, change left, change right, take exit, and for each one generate a plausible trajectory, using map geometry for the path and kinematic models for the speed profile. Classifiers estimate the probability of each manoeuvre from observed cues. This approach produces multimodal forecasts naturally and has the advantage that its outputs can be inspected: one can see that the system assigns 70 percent to "keep lane" and 30 percent to "change right" for a particular car. The third and now dominant stage is learned prediction, in which neural networks trained on large datasets of recorded driving map a representation of the scene directly to forecast trajectories. Early learned methods rendered the scene as a top-down raster image, with lanes, agents and their histories drawn in different colour channels, and applied convolutional networks. Subsequent methods represented the scene more efficiently as a set of vectors: polylines for lane boundaries and centrelines, and sequences of past positions for each agent. VectorNet, published by Waymo and Google researchers in 2020, used graph neural networks over such vectorized elements, and many later methods built on this idea with transformer architectures that let each agent attend to the lanes and other agents relevant to it. Waymo's MultiPath and MultiPath++ models, and its later Wayformer architecture, are examples of this lineage. Learned predictors produce multimodal outputs explicitly. A typical model outputs a small number of candidate trajectories, perhaps six, for each agent, each with a probability and often with a spatial uncertainty around each point. Training such models is subtle, because the recorded data shows only one future for each situation, the one that actually happened. The usual remedy is a "winner-takes-all" loss that rewards the model if any one of its predicted modes is close to the observed future, which encourages the modes to spread out to cover different possibilities. Public datasets again drove progress. The Waymo Open Motion Dataset, released in 2021 and described in a paper presented at ICCV that year, contains more than 100,000 scenes, each twenty seconds long, of real interactive driving with high-quality tracked agents and map data, and it was designed specifically to emphasize interesting interactions. The Argoverse datasets from Argo AI and nuScenes' prediction challenge served similar roles. The standard metrics on these benchmarks, such as the average and final displacement error of the best of several predicted trajectories, and the miss rate, measure how often the true future was covered by at least one prediction. These metrics are useful but have a well-known blind spot: they reward coverage of what actually happened, not the identification of what might have happened and would have been dangerous. Prediction on the highway On a controlled-access highway, prediction is in one sense simpler than in a city, because the set of plausible manoeuvres is small and constrained by lane geometry. Vehicles follow lanes, change lanes, merge from on-ramps, diverge onto exits, and occasionally stop. Pedestrians and cyclists should not be present at all, though they occasionally are, and a robust system cannot simply assume them away. The highway's difficulties come from speed and range. At 120 kilometres per hour, a five-second horizon covers 170 metres, and a forecast that is two seconds late in recognizing a lane change places the other car more than sixty metres from where it should be. Lane-change prediction is therefore the central highway problem. The cues for a lane change, lateral drift toward a boundary, an indicator, a gap opening in the target lane, a slower vehicle ahead, are subtle and must be read at a distance, often from behind the vehicle being predicted. Research on lane-change intention prediction has shown that combining lateral kinematics with contextual features about the surrounding gaps substantially improves early detection compared with kinematics alone, though the precise lead time achievable depends heavily on the dataset and the definition of when a manoeuvre begins. Cut-ins, when another vehicle moves into the automated vehicle's lane close in front, are a particular concern. They are among the most common triggers for hard braking by driver-assistance systems and among the scenarios that regulators have specified explicitly. The UN regulation on Automated Lane Keeping Systems, UN Regulation No. 157, requires the system to handle cut-ins and describes reference models of a competent and careful human driver against which the system's response to such scenarios is compared. A predictor that recognizes a cut-in half a second earlier gives the planner half a second more in which to respond smoothly rather than abruptly. Merges are another. A vehicle on an on-ramp must enter the main carriageway, and drivers already in the inside lane commonly adjust speed or move over to make room. Here interaction dominates. Whether the merging truck will go ahead of or behind the automated vehicle depends largely on what the automated vehicle does. A prediction module that forecasts the truck independently, without considering the ego vehicle's plan, gives the planner an inconsistent picture. This is why recent research has moved toward joint or conditional prediction: forecasting the scene's agents together, so that their predicted futures are mutually consistent, and conditioning predictions on candidate plans of the ego vehicle, so that the planner can ask "if I hold my speed, what does the truck probably do, and if I slow, what then?". The tail of the distribution The most important property of a prediction module, from a safety perspective, is not how well it forecasts the most likely future but how it handles the unlikely ones. A car in the adjacent lane is overwhelmingly likely to stay in its lane on any given second. But drivers do occasionally swerve without warning: to avoid debris the automated vehicle cannot yet see, because they are distracted, because they have suddenly realized they need the exit. A predictor trained to maximize accuracy on recorded data will assign such events low probability, correctly, and a planner that ignores low-probability futures will be caught out by them. There are two broad ways to deal with this. The first is to make the planner respond to the full distribution of predictions, weighting not just by probability but by consequence, so that a 2 percent chance of a cut-in that would leave no room to stop counts for more than a 30 percent chance of a mild slowdown. Chapter 7 discusses how planners do this. The second is to combine learned prediction with reachability reasoning: rather than asking what another vehicle will probably do, ask what it physically could do within its limits of acceleration, braking and steering, and ensure that the ego vehicle's plan remains safe, or at least that a safe escape remains available, against those possibilities. The responsibility-sensitive safety model proposed by Shai Shalev-Shwartz, Shaked Shammah and Amnon Shashua of Mobileye in 2017, discussed further in Chapter 7, takes this approach for longitudinal and lateral following distances, assuming bounded but adversarial behaviour by other vehicles within defined limits of reasonableness. The tension between these approaches is real. Pure worst-case reasoning makes driving impossible: in dense highway traffic, almost any neighbouring car could in principle swerve into the automated vehicle's lane, and a system that guarded against every such possibility would never move. Pure probabilistic reasoning makes driving unsafe in the tail. Practical systems combine them, using learned prediction for the likely futures and bounded physical reachability to define the envelope of possibilities against which safety margins are checked. Measuring what matters How should a prediction module be judged? The standard benchmark metrics reward geometric accuracy: how close the best predicted trajectory came to what actually happened, averaged over all agents and all scenes. On those metrics, a predictor can improve substantially while becoming no better, or even worse, at the situations that determine safety. Most agents in most scenes do something unremarkable, and accuracy on them dominates the average. A predictor that fails to anticipate one cut-in in a thousand will barely register on average displacement error, while that single miss might be the one that matters. Engineers therefore increasingly evaluate predictors in terms of their effect on planning. One approach is to weight prediction errors by how relevant the agent is to the ego vehicle's plan: an error on a car three lanes away on the other side of the barrier matters far less than one on the car about to merge in front. Another is to evaluate predictors on curated sets of difficult interactions, cut-ins, abrupt braking, merges in dense traffic, rather than on the natural distribution of driving. A third is to run the full planner with each candidate predictor in closed-loop simulation and measure the result in terms of safety and comfort: how often the planner had to brake hard, how often it came within an unsafe distance of another vehicle, how often it hesitated unnecessarily. Closed-loop evaluation is expensive and depends on the realism of simulated agents, a limitation discussed in Chapter 9, but it measures the thing that actually matters. Calibration deserves its own scrutiny. A predictor that says a lane change has a 20 percent probability should be right about one time in five across all the cases where it says so. If its 20 percent forecasts in fact come true 40 percent of the time, the planner, which weighs risks using those probabilities, is systematically underestimating danger. Calibration can be measured directly on held-out data, by grouping predictions by stated probability and comparing them with observed frequencies, and it can be corrected after training. It is a less glamorous property than accuracy, and for safety purposes it is often more important. Prediction and uncertainty Prediction is the point in the stack where the booklet's central concern, the propagation of uncertainty, becomes most acute. Every input to the predictor is itself uncertain: the agent's position and velocity come with covariances from tracking; its class may be ambiguous; the lane it is in depends on localization and map accuracy; its intention is latent. A prediction module that takes the mean of each input as truth, and outputs a confident forecast, has thrown away all of that uncertainty, and the planner downstream will make decisions on a false premise. The Tempe crash again sharpens the point. A pedestrian walking a bicycle across a four-lane road at night, away from a crossing, is an unusual scenario. The predictor had a mixture of classifications and, because of the tracking design, no motion history. A system built around honest uncertainty would have recognized that it had an object of unknown type, moving with poorly known velocity, near the vehicle's path, and treated that combination as a reason to slow down. The general rule that emerges is that a prediction module should become less confident, not more, as its inputs become less certain, and that "I do not know what this agent will do" is a legitimate and important output. A well-designed planner can do something useful with it. What it cannot do anything useful with is a confident forecast that happens to be wrong. Chapter 7: Path Planning and Trajectory Optimization Everything upstream in the stack exists to serve one decision, made many times per second: what should the vehicle do next? The planner receives the world model, the vehicle's localized position in the map, the route to the destination, and a set of probabilistic forecasts for every relevant agent. It must produce a trajectory, a precise path through space with a speed at every point, that is safe, legal, comfortable, achievable by the vehicle's physics, and that makes reasonable progress toward the goal. It must do so within tens of milliseconds, and then do it again, with fresh information, before the previous trajectory has been half executed. Planning is where uncertainty finally has to be turned into action, and the quality of a planner is judged by how well its actions respect what the rest of the stack does not know. The layers of decision Planning is usually divided into layers that operate at different timescales and levels of abstraction. The route planner chooses the sequence of roads from origin to destination. On a highway this is largely settled in advance, but it determines tactical needs, such as being in the right lane for an exit two kilometres away. Route planning uses graph search over the road network, the same family of algorithms as consumer navigation, and is not where the difficulty lies. The behaviour layer, sometimes called the decision or tactical layer, chooses among discrete manoeuvres: keep lane and follow the car ahead, change lane left, change lane right, prepare to exit, yield to a merging vehicle, slow down for a hazard, or begin a minimal risk manoeuvre. Early systems implemented this layer as a hand-written finite-state machine, with rules for transitioning between states. The vehicles of the 2007 DARPA Urban Challenge, including Carnegie Mellon's winning entry Boss and Stanford's Junior, used such structures. They remain common, because they are interpretable and testable, but they scale badly: every new situation adds rules and every rule interacts with others. The motion planner turns the chosen behaviour into an exact trajectory. This is where the mathematics of optimization comes in, and where most of the subtlety lives. The boundaries between these layers have blurred. Many modern planners do not commit to a manoeuvre first and then plan a trajectory; they generate candidate trajectories for several manoeuvres, evaluate them all with the same cost function, and choose the best, so that the decision to change lanes emerges from comparing a well-planned lane change with a well-planned lane keep. Learned planners blur the layers further. What makes a trajectory good A trajectory is a function of time: for each moment over the next several seconds, where the vehicle will be, which way it will point, how fast it will be going and how hard it will be accelerating. To choose among trajectories, the planner must judge them, and that judgement is usually expressed as a cost function: a weighted sum of terms, each penalizing something undesirable, together with constraints, conditions that any acceptable trajectory must satisfy absolutely. Typical constraints include staying on the drivable surface, not colliding with any obstacle, respecting the vehicle's physical limits on acceleration, braking, steering angle and steering rate, and keeping lateral acceleration below the friction the road can provide. Typical cost terms include deviation from the lane centre, deviation from the desired speed, magnitude of acceleration and of jerk, the rate of change of acceleration, which is what passengers feel as abruptness, proximity to other vehicles, and time spent in another vehicle's blind spot. The weights on these terms encode the system's driving character, and choosing them is part engineering, part judgement. Heavy weight on comfort produces smooth, gentle driving that may be too hesitant to merge into dense traffic. Heavy weight on progress produces assertive driving that may make other road users uncomfortable. The weights also embody safety choices implicitly: how much the planner is willing to trade a small reduction in clearance for a large gain in comfort is, ultimately, a decision about acceptable risk. Families of planners Motion planners fall into several broad families, which differ in how they search the enormous space of possible trajectories. A survey by Brian Paden and colleagues in 2016 remains a good overview. Table 3 compares the principal families as they apply to highway driving. Table 3. Principal motion-planning families for highway driving. Family How it searches Strengths Weaknesses Graph search on a lattice Connects pre-computed motion primitives on a grid Complete within resolution; handles complex manoeuvres Resolution limits; grows costly in many dimensions Sampling (Frenet-frame) Generates many candidate end states, connects with smooth curves, scores them Simple, fast, naturally explores manoeuvres Quality depends on sampling; may miss narrow solutions Optimization (MPC) Solves a constrained optimization over a receding horizon Smooth, dynamically feasible, explicit constraints Non-convex in general; can stall in local minima Learned Neural network outputs or scores trajectories Captures nuanced human-like behaviour Hard to verify; needs safety checks around it Graph search methods discretize the space of states and connect them with short, dynamically feasible motion primitives, forming a lattice, then search it for the lowest-cost path using algorithms such as A-star. They are well suited to structured environments and were used extensively in the DARPA challenges. Sampling-based methods generate a large number of candidate trajectories and evaluate them. A particularly influential version for road driving is the approach published by Moritz Werling, Julius Ziegler, Sören Kammel and Sebastian Thrun in 2010, which plans in the Frenet frame. Instead of using ordinary Cartesian coordinates, the planner describes the vehicle's position by its distance along the lane's centreline and its lateral offset from it. In this frame, keeping a lane means holding lateral offset near zero, and changing lanes means moving lateral offset smoothly from zero to one lane width. The planner samples a range of end states, different lateral offsets, speeds and manoeuvre durations, connects the current state to each with polynomials chosen to minimize jerk, converts them back to Cartesian space, discards those that violate constraints or collide with predicted obstacles, and selects the lowest-cost survivor. This approach is simple, fast, and naturally suited to highway geometry, and variants of it appear in many research and production stacks, including Baidu's open-source Apollo platform. Optimization-based methods, chiefly model predictive control, formulate planning as a mathematical optimization: find the sequence of control inputs over the next few seconds that minimizes the cost function subject to the constraints, using a model of the vehicle's dynamics to predict the result of each input. Only the first portion of the optimized plan is executed; then the problem is re-solved from the new state with fresh information, a strategy called a receding horizon. Model predictive control produces smooth, dynamically consistent trajectories and handles constraints explicitly, which is why it is popular for highway lane keeping and lane changes. Its difficulty is that collision avoidance makes the problem non-convex, with many local optima, one for passing an obstacle on the left and another for passing on the right. Practical systems often combine methods: a sampling or search stage selects the general manoeuvre and an optimization stage refines it. Learned planners train neural networks, by imitation of human driving or by reinforcement learning in simulation, to output or rank trajectories. They capture subtle aspects of human driving that are hard to write as cost terms, such as how assertively to merge in particular kinds of traffic. Their difficulty is assurance: it is hard to prove that a learned planner will never output a dangerous trajectory. The common response is to wrap them in explicit safety checks, so that any learned trajectory that violates a hard constraint is rejected in favour of a verified fallback. Planning under uncertainty The planner's hardest job is not finding a good trajectory in a known world but choosing a good one when the world is uncertain. The world model gives it tracked objects with position and velocity covariances. Prediction gives it several possible futures for each agent, each with a probability. Localization gives it a position with an error bound. A naive planner takes the most likely value of everything and plans as if it were certain. That planner works well most of the time and fails exactly when it matters. Several techniques make planning robust to uncertainty. The simplest is to inflate obstacles by margins that grow with their positional uncertainty and with the prediction horizon, so that a car whose future position is poorly known occupies a larger region of space-time. A second is to plan against multiple predicted futures simultaneously and require the chosen trajectory to be acceptable under all plausible modes, or to minimize expected cost weighted by probability and severity. A third, which matters most on the highway, is to preserve options: to favour trajectories that keep a safe escape available. A trajectory that follows the car ahead at a distance from which a full stop is possible, and leaves the adjacent lane clear, is preferable to one that is marginally faster but would leave no safe response if the car ahead braked hard. The most formal version of these ideas is contingency planning. The planner computes a nominal trajectory for the most likely future and, simultaneously, a contingency trajectory that shares its first segment but diverges to handle a less likely but dangerous future. Because the two share their initial portion, the vehicle can commit to the nominal plan now while guaranteeing that, if the dangerous future materializes, a safe response remains feasible. On a highway this might mean holding a following distance and lateral position such that, if the car in the adjacent lane does cut in, a comfortable brake rather than an emergency stop will suffice. Formal safety envelopes Alongside optimization-based planning, an influential line of work has tried to define safety formally, in terms that can be checked independently of how a trajectory was produced. The best known is responsibility-sensitive safety (RSS), proposed by Shai Shalev-Shwartz, Shaked Shammah and Amnon Shashua of Mobileye in a 2017 paper. RSS defines a safe longitudinal distance as the gap needed so that, if the vehicle ahead brakes as hard as it reasonably can, and the following vehicle takes a specified response time before braking at a specified minimum rate, no collision occurs. It defines analogous lateral safe distances and rules for what constitutes a dangerous situation and the proper response to it. The underlying philosophy is that an automated vehicle should never be the cause of an accident, and that "cause" can be defined mathematically by reference to reasonable assumptions about other road users' behaviour. The value of such a model is that it can act as an independent check on the planner. Whatever a learned or optimization-based planner proposes, a simple monitor can verify that the trajectory maintains RSS-safe distances, or that it performs the proper response when they are violated, and override it otherwise. The limitations are that the parameters, the assumed maximum braking of the car ahead, the response time, the minimum braking of the ego vehicle, involve judgements that determine how conservative the result is, and that the model addresses some situations far more completely than others. The IEEE has since published a standard, IEEE 2846, that sets out a framework for the reasonably foreseeable assumptions about other road users that such safety models depend on. A related approach from NVIDIA, the safety force field, takes a similar view based on reachable sets. A lane change, worked through Consider a concrete case that combines most of the ideas in this chapter. An automated car is travelling at 100 kilometres per hour, about 28 metres per second, in the right-hand lane of a two-lane motorway. Ahead, a truck is travelling at 80 kilometres per hour. The route requires staying on the motorway for another twenty kilometres, so there is no reason to stay in the right lane except comfort and courtesy. The planner must decide whether and when to overtake. The sampling planner generates candidates. Some keep the lane, slowing smoothly to match the truck's speed and settling at a following gap. Others change lanes to the left at various times, starting now, in one second, in two, in three, and over various durations, from about four to seven seconds, each defined in the Frenet frame as a smooth lateral transition from the centre of the right lane to the centre of the left lane, combined with a speed profile. For each candidate, the planner checks the predicted occupancy of the left lane. A car is approaching from behind in the left lane at 130 kilometres per hour, about 36 metres per second, and is currently 80 metres back. Its closing speed relative to the automated car is about 8 metres per second, so it will arrive alongside in roughly ten seconds if both hold their speeds. A lane change that begins now and completes in five seconds would put the automated car in the left lane with the approaching car about 40 metres behind and closing at 8 metres per second. That gap is legal and probably acceptable to many human drivers, but it would force the approaching car to brake to avoid closing to an uncomfortable distance, which a cost term penalizing the induced deceleration of other road users would discourage. The prediction module also gives some probability to the approaching car accelerating, which makes the gap tighter still. The candidates that change lanes now are therefore costly, and some violate the planner's safety margins under the less favourable predicted modes. Meanwhile the truck ahead is 70 metres away and closing at about 5.5 metres per second, so at constant speed the car would reach the truck in a little over twelve seconds, and would need to begin slowing well before that. The planner's best choice is a contingency plan: begin to ease off slightly in lane, letting the approaching car pass, and plan a lane change into the gap behind it. The nominal plan and the fallback share their first two seconds, gentle deceleration in lane, so the vehicle loses nothing by committing to them. Once the faster car has passed, the left lane is clear for a long distance, and the lane change candidates beginning at that point have low cost and comfortable margins. The planner selects one, the controller executes it, and the car passes the truck with a smooth trajectory that no passenger would remark on. Nothing in that sequence is exotic. Its significance is that every input was uncertain, the approaching car's speed and intent, the truck's speed, the automated car's own position in its lane, and the planner chose a sequence of actions that remained safe across the plausible range of those uncertainties while preserving the option to change course. The same scenario handled by a planner that took each prediction at face value might well have produced the aggressive lane change, which would usually work and occasionally would not. Control: closing the loop The trajectory the planner produces is a plan, not a motion. The controller converts it into steering, throttle and brake commands, and corrects continuously for disturbances such as wind gusts, road camber, changes in tyre grip and the vehicle's own imperfect response. Lateral control on highways is often done with geometric methods such as pure pursuit or the Stanley controller, named after the Stanford DARPA vehicle, or with model predictive control that uses a dynamic model of the vehicle's tyres and yaw response. Longitudinal control typically uses feedback loops on speed and gap, with feedforward terms from the planned acceleration. Control matters to safety in a way that is easy to overlook. A planner that produces trajectories the controller cannot follow, because they demand more lateral acceleration than the tyres can provide on a wet road, or steering faster than the actuator can move, will produce errors that the planner does not anticipate. The planner must therefore use a vehicle model faithful enough to the real dynamics, and the controller must report its tracking error back so that the planner and supervisor know when the vehicle is not doing what was intended. On the highway, at speed, the margin between a comfortable lane change and loss of control on a wet road is smaller than most passengers would imagine, and the planner's model of available friction is one of its most important and least certain inputs. Planning is where the whole stack's uncertainty is finally spent. Every metre of margin the planner keeps is paid for by perception's range, fusion's honesty, localization's integrity and prediction's coverage of the tail. But a planner, however robust, depends on all of those inputs existing. The next question is what happens when they stop. Chapter 8: Fail-Safe and Fail-Operational Design Every component in an automated vehicle will eventually fail. Cameras are blinded by mud and sun. LiDARs are struck by stones. Connectors corrode. Processors overheat or suffer memory errors from cosmic radiation. Software crashes, deadlocks, or encounters an input that drives it into an untested state. Power supplies brown out. Steering motors burn out. The question is never whether failures occur but what the vehicle does when they do. For a Level 2 system, the answer can be: alert the driver and disengage. For a system that has taken responsibility for driving, at Level 3 or 4, on a highway at 120 kilometres per hour, the answer must be that the vehicle continues to drive safely, perhaps in a degraded mode, for long enough to reach a safe state. That requirement, more than any perception or planning algorithm, determines what the hardware and software of a highway-capable automated vehicle look like. From fail-safe to fail-operational In conventional automotive engineering, fail-safe means that a fault drives the system into a safe state. If an electronic throttle detects a disagreement between its two position sensors, it closes the throttle; the car loses power, but nothing dangerous happens because the driver is still steering and braking. The safe state is easy to reach because a human remains in control. For an automated driving system, the naive safe state, stop everything, is not safe. Cutting power to the steering in a lane change at speed, or applying full braking in the fast lane of a motorway, can be catastrophic. The vehicle must instead remain fail-operational: after a fault, it must continue to perceive, plan and actuate at least well enough to perform a controlled manoeuvre to a minimal risk condition. SAE J3016 defines a minimal risk condition as a stable, stopped condition to which a user or the system may bring the vehicle to reduce the risk of a crash when a trip cannot or should not be continued. On a highway, that typically means a controlled deceleration to a stop, ideally on the hard shoulder or at least in the slowest lane, with hazard lights on and, where available, an automatic call to emergency services. The difficulty is that the minimal risk manoeuvre must be executed by a system that has, by definition, just suffered a failure. If the main perception computer has crashed, something else must still see the road. If the primary steering motor has failed, something else must still steer. If the primary power supply has shorted, something else must still power the systems doing these things. Fail-operational design is the art of ensuring that no single fault, and ideally no plausible combination of faults, removes the vehicle's ability to stop safely. Redundancy, layer by layer Redundancy is the obvious tool, but redundancy is not duplication. Two identical computers running identical software on identical inputs will fail together when the fault is in the software or the inputs. Effective redundancy requires independence: components that fail for different reasons, under different conditions, at different times. Engineers pursue it through physical separation, different power supplies, different hardware, different algorithms and different sensors. Table 4 summarizes how this is typically applied to each layer of a highway automation system, as the rest of this section describes. Table 4. Typical redundancy measures in a fail-operational highway automation architecture. Layer Primary path Redundant or fallback path Independence achieved by Sensing Full camera, LiDAR and radar suite Overlapping sensors of different physics Different modalities, mounting, wiring Compute High-performance main computer Separate safety computer running a simpler stack Different hardware and software, separate power Power Main low-voltage network Second independent supply and distribution Separate battery or converter, isolated circuits Steering Electric power steering motor Dual-wound motor or second steering actuator Separate windings, controllers and supply Braking Electronic brake booster Independent stability-control braking unit Separate hydraulic and electronic paths Communication Primary vehicle network Redundant network links Separate channels and routing Sensing redundancy comes from overlapping coverage by sensors of different physics. The forward view of a highway system is typically covered by at least one camera, one long-range radar and one LiDAR, so that the loss or degradation of any one leaves at least two sources. The diversity is what matters: as Chapter 2 argued, a camera and a radar are unlikely to be blinded by the same thing. Compute redundancy usually takes the form of a second, independent computer. In many architectures the main computer runs the full, high-performance stack, with large neural networks for perception and prediction and a sophisticated planner, while a separate safety computer runs a much simpler stack. That simpler stack might do nothing more than track the lane, monitor the space immediately ahead with radar and a camera, and execute a minimal risk manoeuvre. Because it does less, it can be designed and verified to a higher standard of integrity, and because it runs on different hardware with a separate power supply, a fault that kills the main computer is unlikely to kill it too. This is sometimes described as a "primary and fallback" or "doer and checker" architecture. Power redundancy is essential and often underappreciated. A conventional car has one low-voltage electrical system; if it fails, everything electronic fails together. Fail-operational vehicles use two independent power networks, often with separate batteries or DC-DC converters, so that the safety-critical consumers, one computer, one steering path, one braking path, and enough sensors to stop, are always powered by at least one supply. Actuator redundancy covers steering and braking. Steering redundancy is commonly provided by electric power steering with a dual-wound motor, two independent sets of windings with their own controllers and supplies within a single housing, so that one half can continue to steer if the other fails. Braking redundancy is often provided by the combination of an electronic brake booster and the separate stability-control unit, each able to build hydraulic pressure independently. Mercedes-Benz has described redundant steering, braking, power supply and sensors as part of the architecture for Drive Pilot, which is consistent with the requirements of Level 3 operation under UN Regulation No. 157. Monitoring: knowing that something has gone wrong Redundancy is useless if the system does not notice that the primary path has failed. A large part of fail-operational design is monitoring: continuously checking that every component is alive, timely and plausible. Some failures announce themselves. A sensor stops sending data; a process stops sending its heartbeat; a voltage drops below threshold. These are relatively easy to catch with watchdogs and timeouts. Far more dangerous are failures in which a component keeps producing output that is wrong: a camera whose lens is covered in road salt continues to send images, but they show nothing useful; a perception network continues to output object lists, but has missed the truck; a localization filter continues to report a position, confidently, two metres to the left of the truth. These silent failures must be caught by plausibility checks: cross-comparison between sensors, consistency with physics and with previous outputs, and comparison of the primary system's decisions against an independent check. The independent safety checker is the key mechanism. It might verify that the trajectory produced by the main planner keeps a safe distance from obstacles detected by an independent sensor path, respects the vehicle's dynamic limits, stays within the lane boundaries detected independently, and maintains a safe envelope of the kind formalized by responsibility-sensitive safety, discussed in Chapter 7. If the check fails, the checker can reject the trajectory and command its own. Because the checker is simpler than the system it checks, it can be more thoroughly verified; it does not need to drive well, only to recognize unsafe driving and stop it. Monitoring also covers the operational design domain itself. A system designed for dry motorways must detect that it has entered heavy rain; a system designed for mapped roads must detect that it has encountered an unmapped construction zone; a system designed for daylight must detect falling light. ODD monitoring is a form of self-knowledge: the system must recognize not only when its components have failed but when the world has moved outside the conditions under which its components were shown to work. Degradation and the minimal risk manoeuvre When monitoring detects a problem, the system's response should be proportionate. Not every fault calls for an immediate stop. A well-designed system has a graded set of degradation modes. If a single camera is impaired but radar and LiDAR coverage remain good, the system might continue at reduced speed with increased following distance. If the main computer fails, the safety computer takes over and begins a minimal risk manoeuvre. If the system determines that it is approaching the edge of its ODD, a motorway exit, the end of a mapped section, heavy rain ahead, it issues a takeover request to the human, if there is one, well in advance. For a Level 3 system, the takeover request is the first line of response, and the time allowed for it is a central design parameter. UN Regulation No. 157 requires that the system remain in control while issuing a transition demand, escalating its warnings, and, if the driver does not respond within a specified period, begin a minimal risk manoeuvre itself, slowing to a stop with hazard lights on. The regulation requires the system to be able to detect whether the driver is available, through driver monitoring of eyes and posture, and to respond appropriately to an unresponsive driver. The subtlety of Level 3 is that the system must remain fully capable during the transition period: it cannot hand over because it is failing; it must hand over while still working, with enough margin that it can continue working until the human is ready. For a Level 4 system, there is no human to fall back on, and the minimal risk manoeuvre is the system's own responsibility. The design of that manoeuvre is a planning problem in its own right, with reduced inputs. On a highway, the best minimal risk condition is a stop on the hard shoulder; but reaching the hard shoulder may require one or more lane changes, which require a working view of the adjacent lanes. If that view is unavailable, the safer choice may be to stop in the current lane, with hazard lights, and accept the risk of being struck from behind, a risk mitigated by decelerating gently enough that following traffic can react. Choosing between these options under degraded perception is one of the hardest problems in highway automation, and the answer depends on exactly which capabilities remain. A fault at speed, worked through Imagine a driverless truck travelling at 85 kilometres per hour in the right-hand lane of a three-lane interstate at night. Its main computer suffers a hardware fault and stops responding. Within a few tens of milliseconds, the safety computer's watchdog notices that the expected stream of trajectories and heartbeats from the main computer has stopped. The last valid trajectory the main computer produced covers the next several seconds, and the actuator controllers continue to follow it for the moment, which buys the safety computer time without leaving the vehicle unguided. The safety computer now takes control. It runs its own lane-keeping, fed by a camera and the forward radar wired directly to it and powered from the second supply. It does not have the main computer's full surround model, its long-range LiDAR perception or its sophisticated predictor. It does know, from its own sensors, that the lane ahead is clear for at least the next hundred metres, and it knows the vehicle's current speed and lane position. It also has the last state of the world model that the main computer shared before the fault, which can be used, with caution and with rapidly growing uncertainty, to judge whether the hard shoulder to the right was clear. The design question is what the safety computer should do next. Moving onto the hard shoulder requires confidence that the shoulder is clear and that nothing is in the blind spot beside the trailer. If the safety computer has its own side-looking radar for that purpose, it may be able to make that manoeuvre with confidence. If it does not, the safer choice is to stay in lane, activate the hazard lights and decelerate at a gentle, constant rate to a stop, perhaps over thirty seconds or so, giving following traffic every opportunity to see and react. At the same time the vehicle notifies its remote operations centre, which can dispatch assistance and alert the highway authorities. What makes this sequence possible is that it was designed in advance. The time the actuators can safely follow the last trajectory, the sensors wired to the safety computer, the power supply that keeps them alive, the deceleration profile and the decision rule for shoulder versus lane were all decided long before the fault, analysed against the hazards they mitigate, and tested by deliberately injecting faults on test tracks and in simulation. A vehicle that improvises its fallback at the moment of failure is not fail-operational; it is lucky. When the fallback itself is the hazard The minimal risk manoeuvre is not automatically safe. Its quality depends on whether the system has correctly understood the situation in which it is executing it. An incident in San Francisco in October 2023 illustrates the danger. A pedestrian was struck by a human-driven car and thrown into the path of a Cruise robotaxi, which braked hard but could not avoid hitting her. The vehicle then executed a pullover manoeuvre, an attempt to move out of traffic to a minimal risk condition, and in doing so dragged the pedestrian, who was trapped beneath it, about twenty feet. The vehicle's systems had not correctly recognized that she was under the vehicle. California's Department of Motor Vehicles suspended Cruise's driverless permits shortly afterwards, and General Motors announced in December 2024 that it would stop funding Cruise's robotaxi development. The lesson is subtle and important. A pullover after a collision is a reasonable default in most cases: a stopped vehicle in a traffic lane may cause further collisions. But the default embodied an assumption, that the system's perception of the space around and beneath it was adequate to make the manoeuvre safe. In this case it was not. Fail-operational design must consider not only whether the fallback can be executed but whether the conditions that would make it safe can actually be verified. Sometimes the safest response to an unexpected event is to stop and do nothing more until the situation is understood. Software faults and the limits of redundancy Hardware redundancy addresses random hardware failures, which are well understood statistically. Software faults are different. They are systematic: a bug present in one copy of a program is present in every copy, and it will fail identically under identical inputs. Running the same software on two computers provides no protection against it. The classic response from aerospace is design diversity: two independently developed implementations of the same function, by different teams, in the hope that their bugs do not coincide. Experience in aviation has shown that independently developed software tends to fail more often on the same difficult inputs than a naive assumption of independence would predict, so diversity reduces but does not eliminate common-mode failure. In automated driving, the most significant systematic risk is not a coding error but a functional insufficiency: a perception network that fails to detect an unusual object, a predictor that does not foresee a rare manoeuvre, a planner that handles a novel situation badly. These are not bugs in the conventional sense; the software is doing exactly what it was trained or designed to do, and the specification itself is incomplete. The standard that addresses this class of risk, ISO 21448, known as SOTIF, the safety of the intended functionality, is discussed in Chapter 9. In architectural terms, the defence is again diversity: a primary perception path based on large learned models, and a fallback path based on different principles, perhaps a geometric obstacle detector working directly on LiDAR and radar data, that does not need to classify an object to know it is there. Diversity of principle is more valuable than diversity of implementation, because two different neural networks trained on the same data are likely to share the same blind spots. Fail-operational architecture is expensive. It roughly doubles the cost of the compute, power, and actuation systems and adds substantial complexity. It is the main reason why systems that take full responsibility for driving cost so much more than driver-assistance systems that look superficially similar, and why the first approved Level 3 systems operate in such narrow conditions. But it is not optional. A highway automation system without it is a Level 2 system with a misleading name, however good its perception and planning. The final question is how anyone, the developer, a regulator, or the public, can come to believe that the whole assembly is safe enough. Chapter 9: Proving It Is Safe Enough An automated driving system is not finished when it drives well. It is finished when its developer can make a convincing argument, to itself, to regulators, to insurers and to the public, that it drives safely enough to be deployed, and can keep making that argument as the software changes. That turns out to be one of the hardest problems in the field, harder in some respects than building the system, because the events that determine safety are rare, the space of situations is effectively infinite, and the most capable components, learned neural networks, are the least amenable to traditional verification. This chapter considers why the obvious approach does not work, what the standards require instead, and how simulation, scenario testing and the safety case fit together. Why you cannot simply drive it The intuitive way to show that an automated vehicle is safe is to drive it a long way and count the crashes. If it crashes less often than human drivers, it is safer. The difficulty is statistical. In the United States, roughly one person dies on the roads for every hundred million vehicle miles driven, a rate that has varied somewhat over the years but remained of that order. Demonstrating, with statistical confidence, that an automated system's fatality rate is lower than that requires driving many times that distance without a fatality. In 2016 the RAND Corporation published a report by Nidhi Kalra and Susan Paddock, Driving to Safety, that made this quantitative. They calculated that demonstrating with 95 percent confidence that an automated fleet's fatality rate was below the human rate would require hundreds of millions of miles of driving with no fatalities, and that demonstrating it was meaningfully better, say 20 percent better, could require billions of miles; their headline figure was about eleven billion miles for one such comparison. At the scale of a test fleet, that would take decades. And every software update, in principle, resets the count, because the system that accumulated the miles is no longer the system being deployed. The point is not that real-world miles are useless; they are essential, both as a source of training data and as a check on whether the system's behaviour in the world matches its behaviour in testing. The point is that mileage alone cannot carry the safety argument. Something else must, and the industry's answer has been a combination of systematic engineering process, scenario-based testing and simulation, held together by an explicit argument called a safety case. The standards Three families of standards define what that systematic engineering looks like. ISO 26262, Road vehicles — Functional safety, first published in 2011 and revised in 2018, is the foundation of automotive safety engineering. It addresses hazards caused by malfunctions of electrical and electronic systems: a sensor that fails, a computer that crashes, a bug that causes the wrong output. It requires a hazard analysis that assigns each safety goal an Automotive Safety Integrity Level, from ASIL A, the least demanding, to ASIL D, the most, based on the severity, exposure and controllability of the hazard. The ASIL then determines how rigorously each component must be designed, verified and validated, with requirements on everything from the architecture to coding practices, test coverage and the probability of random hardware failure. Steering and braking functions that could cause a fatal crash at speed are typically ASIL D. ISO 26262 is the reason redundant architectures look as they do: decomposing an ASIL D requirement across two independent channels allows each to be developed to a lower level, provided their independence is demonstrated. ISO 26262 has a significant blind spot for automated driving: it assumes that if every component works as specified, the system is safe. For a perception network, that assumption fails. A network that misses a pedestrian in an unusual pose has not malfunctioned; it is working exactly as trained. The hazard comes from the insufficiency of the specification and training, not from a fault. ISO 21448, Road vehicles — Safety of the intended functionality, known as SOTIF, published as a full standard in 2022, addresses that gap. SOTIF is concerned with hazards arising from functional insufficiencies and from reasonably foreseeable misuse, even when nothing has failed. It frames the problem as a space of scenarios divided into four regions: known safe, known unsafe, unknown unsafe, and unknown safe. The goal of SOTIF activities is to shrink the known unsafe region, by improving the system or restricting its ODD, and to shrink the unknown unsafe region, by discovering hazardous scenarios that had not been anticipated, through analysis, testing and field monitoring, until the residual risk is acceptable. SOTIF is the standard that most directly addresses the long tail. UL 4600, Standard for Safety for the Evaluation of Autonomous Products, first published by Underwriters Laboratories in 2020, takes a different approach. Rather than prescribing a process, it specifies what a safety case for a fully autonomous product must contain and how it should be assessed. Developed with significant input from Philip Koopman of Carnegie Mellon University, UL 4600 requires the developer to make explicit, structured claims about safety, support each claim with argument and evidence, and address a long list of topics that must be considered, from perception failures to cybersecurity to the safety of maintenance procedures. It is goal-based: it does not say how to make a system safe, only how to show convincingly that it is. These standards complement one another, along with regulatory frameworks such as UN Regulation No. 157 for automated lane keeping in the countries that apply UN vehicle regulations, and the reporting requirements of the US National Highway Traffic Safety Administration, whose Standing General Order, issued in 2021, requires manufacturers to report crashes involving vehicles with automated driving systems or Level 2 driver assistance engaged. Table 5 sets out what each of the three principal standards addresses. Table 5. Principal safety standards for automated driving. Standard First published Hazard focus Approach ISO 26262 2011 (revised 2018) Malfunctions of electrical and electronic systems Process-based, risk-graded by ASIL ISO 21448 (SOTIF) 2022 Functional insufficiencies with no fault Scenario-based, reduce unknown unsafe space UL 4600 2020 Whole autonomous product Goal-based safety case with evidence Scenarios and simulation If driving cannot cover the space of situations by brute force, the space must be explored more cleverly. Scenario-based testing organizes the problem around situations rather than miles. A scenario is a description of a traffic situation: the road layout, the agents present, their initial states and their behaviour. At an abstract level a scenario might be "a vehicle in the adjacent lane cuts in ahead of the ego vehicle". At a concrete level it specifies exact speeds, gaps and timings. Between them sit logical scenarios with parameter ranges: cut-in at relative speeds between zero and 30 kilometres per hour, with gaps between five and forty metres. Scenarios are drawn from several sources: analysis of crash databases, regulatory requirements such as those in UN Regulation No. 157, naturalistic driving data, the system's own field experience, and systematic hazard analysis. The German PEGASUS project, which ran from 2016 to 2019 with participation from automakers, suppliers and research institutes, developed much of the methodology for highway scenario-based assessment, and the ASAM OpenSCENARIO standard provides a common format for describing scenarios so that they can be exchanged between tools. Simulation makes scenario testing scalable. A simulator models the road, the other agents, the sensors and the vehicle's dynamics, and runs the automated driving software against it, either the whole stack or particular modules. Developers run millions of simulated scenarios per day, far more than any fleet could drive, and can concentrate on the dangerous ones. Simulation also allows counterfactual testing: taking a real recorded drive and asking what would have happened if another vehicle had braked a second earlier or cut in more sharply. Waymo has published analyses of this kind, reconstructing fatal human-driven crashes from police data and simulating how its system would have responded in the same circumstances. Simulation's fundamental limitation is fidelity. A simulator is a model of the world, and the automated system is only being tested against that model. Sensor simulation, in particular, is hard: generating camera images, LiDAR returns and radar reflections realistic enough that perception networks behave as they would in reality, including in rain, glare and spray, remains an active research area, with neural rendering techniques narrowing but not closing the gap. Behavioural models of other agents are another weakness: simulated drivers behave as their designers modelled them, and a system tested only against modelled behaviour may be surprised by real behaviour. A safety argument that rests on simulation must therefore include evidence that the simulation is valid for the purpose, typically by comparing simulated and real results on scenarios where both are available. A further technique is falsification: rather than testing scenarios chosen in advance, use optimization or search to find the parameter combinations within a logical scenario that cause the system to fail, or that bring it closest to failing. If the hardest cut-in a search can find within the ODD's parameter ranges is still handled safely, that is stronger evidence than a thousand randomly chosen cut-ins. The safety case All of this evidence, process compliance under ISO 26262, scenario coverage and residual risk under SOTIF, simulation results, track tests, field data, must be assembled into an argument. A safety case is a structured argument, supported by evidence, that a system is acceptably safe for a specific application in a specific environment. Safety cases originated in high-hazard industries such as nuclear power and offshore oil, where they became mandatory in the United Kingdom after disasters such as the Piper Alpha platform explosion of 1988, and they are standard in railway and aviation safety assurance. A safety case typically starts from a top-level claim, such as "the system is acceptably safe to operate within its ODD", and decomposes it into sub-claims, each supported by further sub-claims or by evidence. "The system detects and responds to vehicles in its lane" is supported by perception test results, fusion analysis, and scenario test coverage. "The system reaches a minimal risk condition after any single fault" is supported by architecture analysis, fault injection testing and redundancy verification. Notations such as Goal Structuring Notation help make the argument explicit and reviewable. The discipline of the safety case is that every claim must be supported, and every assumption, such as "the map is accurate" or "drivers respond to takeover requests within ten seconds", must be stated and either justified or monitored in operation. A safety case is not a document produced once. Because software, ODDs and the operating environment change, the safety case must be maintained, with field data flowing back to check whether assumptions still hold. UL 4600 places particular emphasis on this: the safety case should identify safety performance indicators, measurable signals from operation, such as the rate at which following distances fall below their safe value, or the rate of disagreements between primary and fallback perception, whose trends would reveal that an assumption is failing before it causes a crash. What the public data can and cannot tell you Several public data sources offer glimpses of how automated systems perform, and each comes with caveats that are worth understanding. California's Department of Motor Vehicles requires companies testing automated vehicles on public roads in the state to report collisions and, annually, the number of disengagements, occasions on which the automated system was deactivated, either by the system itself or by the safety driver, together with miles driven. For years, disengagement rates were widely treated as a league table of capability. They are a poor one. Companies define disengagements differently; some count every precautionary takeover by a cautious safety driver, others only those triggered by system failures. Fleets operating in dense urban traffic face harder conditions than those on quiet suburban roads. And a disengagement rate says nothing about what would have happened had the safety driver not intervened. NHTSA's Standing General Order crash data is broader, covering all manufacturers and operators in the United States, and it has made the occurrence of crashes involving automated systems much more visible. It too requires care: reports are submitted by manufacturers based on their own knowledge, the level of detail varies, and the data does not include the miles driven by each system, so it cannot by itself yield crash rates. Rates require exposure data that developers publish separately, if at all. The most informative analyses combine a developer's crash records with its mileage and compare the result with a carefully constructed human baseline: human-driven crashes on the same kinds of roads, in the same cities, at similar times of day, using comparable definitions of what counts as a crash. Building that baseline is difficult, because police-reported crash statistics are known to undercount minor collisions. These analyses can, given enough exposure, show a real difference between an automated fleet and comparable human drivers. What they cannot do is establish safety in conditions the fleet has not been exposed to. A fleet's urban record says little about its highway performance at higher speed, and its record in a dry climate says little about snow. Safety evidence is specific to its domain, which is another reason the operational design domain is the central concept in the whole enterprise. How safe is safe enough Underneath all this machinery lies a question that engineering alone cannot answer: what level of safety is acceptable? "Safer than the average human driver" is the most commonly cited benchmark, but it is less clear than it sounds. The average human driver includes drunk, distracted and exhausted drivers; a system that is safer than average might still be considerably less safe than an attentive, sober, experienced driver. The distribution of harm matters too: a system that reduces total crashes but causes a different kind of crash, one a human would almost never cause, may be judged more harshly than the statistics suggest. And there is a widespread view that society will hold machines to a higher standard than people, both because their failures are systematic, repeated across a fleet, and because they are designed rather than merely human. Philip Koopman's 2022 book How Safe Is Safe Enough? explores these questions in detail, arguing that a credible answer must go beyond a single statistical comparison to include absence of unreasonable risk to particular groups, consideration of whether failures were negligent by the standards of a competent human driver, and transparency about how the determination was made. Early operational data from the leading robotaxi fleets, published by Waymo and analysed by independent researchers including the reinsurer Swiss Re, has suggested substantially lower rates of injury crashes and of insurance claims than human benchmarks in the cities where those fleets operate, though such comparisons depend heavily on how human baselines are constructed and how crashes are counted. These are encouraging results, and they are also the beginning of the evidence rather than its end. Safety, in the end, is not a property that can be tested into a system after the fact, nor established by mileage alone. It is the product of the whole engineering chain this booklet has described: sensors chosen for independence, perception that knows what it does not know, fusion that keeps beliefs honest, localization with integrity, prediction that covers the tail, planning that preserves options, architecture that survives its own failures, and an argument that ties all of these together with evidence. Each piece can be checked. The safety case is where the pieces are shown to fit. Conclusion Go back to Tempe with the whole stack in view. The sensors did their job: radar and LiDAR registered an object in the road with more than five seconds to spare. Perception produced hypotheses, but it held them so loosely, and so carelessly, that each new label wiped out what had been learned about the object's motion. Fusion and tracking, which should have built up a steady belief about something solid crossing the road, never did. Prediction, given no usable history, forecast no conflict. The planner, seeing no conflict, kept going. When the danger finally became undeniable, a suppression rule written to prevent false alarms held back the response, and the vehicle's independent emergency braking, the one component designed to act regardless of what the main stack believed, had been switched off. The supervising human, the last layer of redundancy, was not watching. Each layer failed in a way that the layer after it was built to assume would not happen. That is the argument of this booklet at its starkest. The safety of an automated vehicle does not live in any one module. It lives in how uncertainty is carried from one module to the next, and in whether some independent part of the system is still able to act when the main chain of reasoning breaks down. Ask which component of the Tempe vehicle was faulty and you will not find a satisfying answer. Ask where the system lost track of what it did not know, and the answer is everywhere. What follows for engineers Several practical commitments follow, and none of them depends on choosing a particular architecture, sensor suite or algorithm. The first is to treat uncertainty as a first-class output of every module. A perception network that outputs a box without a calibrated measure of confidence, a tracker that outputs a state without a consistent covariance, a predictor that outputs one trajectory, a localizer that outputs a position without an integrity bound: each of these is incomplete. The planner downstream can only be as cautious as its inputs let it be. This is harder than it sounds, because neural networks are poorly calibrated by default, and because uncertainty adds work at every interface. But where uncertainty is dropped at an interface, the system will eventually make a decision that looks inexplicable afterwards. The second is to separate geometry from semantics. Something solid in the lane must be avoided whether or not a classifier can name it. The rise of occupancy representations, generic obstacle detection from LiDAR and imaging radar, and trackers that persist through reclassification all reflect this principle. It is the main defence against the long tail, because the long tail is mostly made of things that have no name in the training set. The third is to design for disagreement. Sensors of different physics will disagree, maps will disagree with perception, and the primary planner will sometimes disagree with the safety checker. These disagreements are not noise to be averaged away. They are among the most valuable signals the system has, often the first sign that a sensor has degraded, that the world has changed, or that the vehicle is approaching the edge of its operational design domain. A system that registers disagreement, reasons about its likely causes and responds in proportion, slowing a little, widening a gap, asking for a takeover in good time, is far more robust than one that resolves every conflict instantly to a confident answer. The fourth is to make the fallback independent in principle, not just in hardware. Two copies of the same software on two computers protect against a burnt-out chip, not against a blind spot. The fallback path that matters is one that would have caught what the primary path missed: a geometric obstacle detector that does not depend on classification, a safety envelope that does not depend on the planner's cost function, an emergency brake that does not depend on the main stack's belief about the world. The fifth is to treat the operational design domain as a safety mechanism, not a marketing restriction. The first systems approved for eyes-off highway driving were confined to dense traffic at modest speeds on mapped, physically divided motorways because those conditions removed a great deal of the long tail. Expanding the domain, to higher speeds, to night, to rain, to unmapped roads, should be done step by step, each step backed by its own evidence. The pressure to expand faster than the evidence supports is constant, commercially and culturally, and resisting it is part of the job. What remains unsettled Several questions are genuinely open, and a reader should be suspicious of anyone who claims they are closed. The first is whether end-to-end learned systems can be made to meet the kind of assurance that highway automation at Level 3 and above requires. They have clear advantages in capturing the subtlety of driving and in avoiding the information losses of hand-designed interfaces. But the standards and regulatory frameworks that govern safety-critical systems were built around decomposition, requirements and verifiable components. Either learned systems will need wrappers of independent, verifiable safety monitors, which is the direction most of the industry is taking, or the methods of assurance will need to evolve to handle learned components on their own terms. Both are happening, and neither is complete. The second is how much reliance on prior maps is wise. The robotaxi fleets with the most extensive driverless operations have relied heavily on detailed maps; scale-oriented developers have argued that maps are a crutch that limits expansion. Online map perception has improved markedly. The probable answer is that maps will become lighter, fresher and treated more explicitly as uncertain priors, but the trade-off between the safety benefit of prior knowledge and the risk of stale prior knowledge will persist. The third is how safe is safe enough, and who decides. Statistical comparisons with human drivers are necessary but not sufficient. Society will also want to know that automated vehicles do not create new classes of harm that human drivers would not, that their failures are investigated openly, and that the claims made for them are backed by evidence an independent party can scrutinize. The regulatory frameworks, UN Regulation No. 157 in Europe and elsewhere, state-level permitting and federal crash reporting in the United States, are still evolving, and the relationship between a developer's safety case and a regulator's approval is not settled in most jurisdictions. The fourth is the most technical and possibly the most important: how to measure, during operation, whether the assumptions of the safety case still hold. The safety performance indicators that UL 4600 calls for, rates of near-misses, of disagreements between primary and fallback paths, of excursions toward the edge of the domain, are the closest thing the field has to an early-warning system. Building them, publishing them and acting on them is likely to separate the developers who earn public trust from those who lose it. A closing thought It is easy, watching a driverless vehicle hold its lane on a motorway in the rain, to attribute its competence to its sensors or its neural networks. Those are the visible parts. The invisible parts are what make it safe: the covariance that grew when a camera lens was smeared, the tentative track that was held rather than discarded, the contingency trajectory that kept a gap open in case the truck ahead braked, the second computer quietly checking every decision the first one made, the second power supply, the second steering winding, the argument assembled over years explaining why each of these is enough. A highway-capable automated vehicle is, at bottom, a machine for knowing the limits of its own knowledge and behaving accordingly. The best human drivers do something similar, slowing when they cannot see, leaving room when they are unsure, but they do it intuitively and inconsistently. The promise of automated driving is to do it explicitly, measurably and every time. The engineering described in this booklet is how that promise is kept, or broken. Further Reading Sebastian Thrun, Wolfram Burgard and Dieter Fox, Probabilistic Robotics (MIT Press, 2005).** The foundational text on estimation under uncertainty for robots: Bayes filters, Kalman and particle filters, occupancy grids, localization and mapping. Mathematical, but the clearest statement of the idea that a robot should reason in beliefs rather than facts. Yaakov Bar-Shalom, X. Rong Li and Thiagalingam Kirubarajan, Estimation with Applications to Tracking and Navigation (Wiley, 2001).** The standard engineering reference on tracking filters and data association, written for practitioners who have to make them work on real sensor data. SAE International, J3016, Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles. The source of the levels of automation, operational design domain, dynamic driving task and minimal risk condition. Short, precise, and widely misquoted; worth reading in the original. Nidhi Kalra and Susan M. Paddock, Driving to Safety: How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability? (RAND Corporation, 2016).** The concise statistical argument that mileage alone cannot establish the safety of an automated fleet. Philip Koopman, How Safe Is Safe Enough? Measuring and Predicting Autonomous Vehicle Safety (2022).** A thorough, readable examination of what safety means for automated vehicles, why simple comparisons with human drivers are insufficient, and how safety cases and performance indicators should work, by one of the principal contributors to UL 4600. Philip Koopman and Michael Wagner, "Challenges in Autonomous Vehicle Testing and Validation," SAE International Journal of Transportation Safety (2016).** An early and influential paper setting out why conventional automotive validation methods break down for automated driving, especially where machine learning is involved. Shai Shalev-Shwartz, Shaked Shammah and Amnon Shashua, "On a Formal Model of Safe and Scalable Self-driving Cars" (arXiv, 2017). The paper introducing responsibility-sensitive safety, with formal definitions of safe distances and proper responses. Brian Paden, Michal Čáp, Sze Zheng Yong, Dmitry Yershov and Emilio Frazzoli, "A Survey of Motion Planning and Control Techniques for Self-Driving Urban Vehicles," IEEE Transactions on Intelligent Vehicles (2016).** A careful survey of the planning and control methods described in Chapter 7, from graph search to model predictive control. Moritz Werling, Julius Ziegler, Sören Kammel and Sebastian Thrun, "Optimal Trajectory Generation for Dynamic Street Scenarios in a Frenet Frame," IEEE International Conference on Robotics and Automation (2010).** The original description of the Frenet-frame sampling planner widely used for highway driving. National Transportation Safety Board, Collision Between Vehicle Controlled by Developmental Automated Driving System and Pedestrian, Tempe, Arizona, March 18, 2018, Highway Accident Report NTSB/HAR-19/03 (2019). The investigation that anchors this booklet. Its account of how detection, classification, tracking, prediction and safety culture combined to produce a fatal outcome is essential reading for anyone building these systems. Holger Caesar et al., "nuScenes: A Multimodal Dataset for Autonomous Driving," CVPR (2020), and Pei Sun et al., "Scalability in Perception for Autonomous Driving: Waymo Open Dataset," CVPR (2020).** Descriptions of two of the public datasets that shaped modern perception research, useful for understanding how the field measures progress and what its benchmarks leave out. Zhijian Liu et al., "BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation" (2022), and Yihan Hu et al., "Planning-Oriented Autonomous Driving," CVPR (2023).** Two representative papers on feature-level fusion in the bird's-eye view and on jointly trained end-to-end stacks, respectively. ISO 26262:2018, Road Vehicles — Functional Safety; ISO 21448:2022, Road Vehicles — Safety of the Intended Functionality; ANSI/UL 4600, Standard for Safety for the Evaluation of Autonomous Products; UN Regulation No. 157, Automated Lane Keeping Systems. The standards and regulation discussed in Chapters 8 and 9. They are long and technical, but their introductory sections set out clearly what each is trying to achieve. Hashtags: #AutonomousVehicleSoftware #AutonomousVehicles #SelfDrivingCars #AutomatedDrivingSystems #ArtificialIntelligence #AutonomousDriving #VehiclePerception #SensorFusion #PathPlanning #MotionPlanning #TrajectoryOptimization #ComputerVision #LiDARTechnology #AutomotiveRadar #ImagingRadar #CameraBasedPerception #ObjectDetection #ObjectTracking #KalmanFilter #BirdsEyeView #OccupancyNetworks #VehicleLocalization #HDMapping #MotionPrediction #BehaviorPrediction #ModelPredictiveControl #RealTimeSystems #FunctionalSafety #FailSafeSystems #AutonomousVehicleTesting