pace-cc

A compiler launcher in the style of ccache and sccache, backed by a pace-kv daemon over the compile-cache protocol.

It fronts every compile: on a hit it writes the object file and replays the compiler's captured output; on a miss it has a worker compile it when PACE_SCHEDULER names one, otherwise runs the real compiler, and stores the canonicalized result. Caching is an optimization — an unreachable or broken cache slows a build down, it never breaks one.

A cache failure does not switch distribution off. A cache and a compile fleet are two services, usually on two machines, so a daemon that refused this launcher or could not be reached says nothing about whether a worker can build the translation unit: the compile is still dispatched, and only the caching of it is lost. The reason is still reported and still counted under unavailable, and nothing further is offered to that daemon for the rest of the invocation — there is no point spending an object-sized transfer to be refused twice.

What a build against a broken cache does pay is the dispatch itself: a translation unit that would have been a hit is now preprocessed a second time -- dispatch sends #line markers, which the cache key deliberately suppresses -- and sent to a worker. That is the trade the fleet exists to make and it is still the right one, but it is not free, and it is the reason a wrong PACE_ADDR is worth fixing rather than living with.

This page is the reference: every flag, every environment variable, every fall-back reason by name. For the story of what happens between your build system and your compiler — direct mode, the key, the lookup, the dispatch — read How it works.

Why not just ccache or sccache?

Cache entries are portable across checkout paths. Before the key is computed, every path under the configured source root and build tree is rewritten to a token; on a hit, the stored output is rewritten back to the consuming machine's layout. Two machines with the same content at different checkout paths produce the same key and share entries.

That matters because a CI runner's checkout is rarely at the same path as a developer's. With a path-sensitive cache the two populations never share hits; here they do. It also keeps the replayed dependency information valid: header paths in /showIncludes output and in -MF depfiles are localized on the way out, so the build tool records dependencies that exist on this machine.

Requirements

• A running pace-kv daemon, reachable over TCP.
• A daemon value cap above your largest object file. The 256 MiB default covers the usual case — a large C++ codebase was measured at ~122 MB for its biggest object — so this normally needs no flag. Past that, --storage-max-value raises both the per-value cap and the wire frame-payload cap:

pace-kv --storage-max-value=512M

Supported compilers

Driver	Recognised as	Object flag	Dependencies
gcc, g++, cc, c++	GNU	-o	-MD -MF <path> depfile
clang, clang++	GNU	-o	-MD -MF <path> depfile
cl	MSVC	/Fo	/showIncludes on stderr — unverified, #825
clang-cl	MSVC	/Fo	/showIncludes on stdout

The stream column describes a compile run, which is what the replay path reproduces. It does not describe the key probe, and nothing keys off it: a preprocess-only run moves the notes (clang-cl puts them on stderr under /EP, so that they do not corrupt the preprocessed stdout), so the probe splits stdout unconditionally and reads notes from both streams rather than guessing.

Version-suffixed (g++-14, clang-18) and .exe forms are recognised. Any other argv[0] is treated as unknown and passed straight through uncached.

A command line is cacheable only when it compiles exactly one translation unit to an object file. Link steps, compile-and-link steps, preprocess-only runs (-E, /EP), and multi-source lines all fall back to the real tool.

Usage

pace-cc <compiler> <args...>                       Front a compile (as CMAKE_<LANG>_COMPILER_LAUNCHER)
pace-cc --show-stats | -s [--prefetch-group <id>]  Report cache statistics for this machine
pace-cc --html-stats [--prefetch-group <id>]
                          [--out <path>]                Render those statistics as a self-contained HTML dashboard
pace-cc --zero-stats | -z                          Discard the statistics log
pace-cc --print-toolchain-fingerprint <compiler>   Print the fingerprint a dispatched compile would use
pace-cc --help | -h | /?                           Flags and environment reference
pace-cc --version                                  Launcher version

--print-toolchain-fingerprint is the diagnostic for a fleet whose scheduler answers no-worker: run it on a client and compare with what the worker logs as serving. It recomputes rather than reading the cache, so it also repairs a stale entry on its way past.

--help is generated from the same table the launcher dispatches on, so it always lists exactly what the binary accepts.

Renamed flags. The statistics flags now use sccache's names: --stats is --show-stats and --clear-stats is --zero-stats. --reset is gone with no replacement — it read as if it reset the cache itself, when it only discarded this machine's statistics log. All three old spellings are now rejected with an "unknown option" diagnostic and exit 2, rather than being silently treated as a compiler to run.

Wire it into CMake:

cmake -S . -B build -G Ninja \
  -DCMAKE_C_COMPILER_LAUNCHER=pace-cc \
  -DCMAKE_CXX_COMPILER_LAUNCHER=pace-cc

The pace-kv build does this for itself: cmake/portable/CompileCache.cmake picks pace-cc up automatically whenever the binary is on PATH and a daemon answers at 127.0.0.1:6674 — at any other daemon, local or remote, when PACE_ADDR is exported when the build tree is created, at -DPACE_ADDR=host:port ahead of even that, nowhere if it is set empty — and injects PACE_SOURCE_DIR / PACE_BINARY_DIR from the source and binary directories, so those two need not be exported. It also injects PACE_DEPS_DIR from CPM's source cache (CPM_SOURCE_CACHE), and with pace-cc as the launcher it maps __FILE__ for the checkout and that cache (-fmacro-prefix-map=<checkout>/= and -fmacro-prefix-map=<cache>/=deps/). Together those are what let a second folder or a second machine reuse every object, test objects and objects built from dependency sources included: without the map a Catch2 test object names its checkout and is stored per checkout, and without the root an include directory into a cache under the user's home keeps that home in every key.

The configure environment seeds a build tree; it does not steer one. PACE_ADDR and PACE_SCHEDULER (below) are taken from the environment by the configure that creates the tree's cache entry, and from then on they are held in the build tree's cache. Only these move a held value:

• -DPACE_ADDR=<host:port> / -DPACE_SCHEDULER=<host:port>, and the empty -DPACE_ADDR= / -DPACE_SCHEDULER=, which turn the cache and dispatch off respectively. A -D always outranks the environment, including one that repeats the value the tree already holds;
• cmake -U PACE_ADDR / -U PACE_SCHEDULER, which removes the entry, so the configure it is part of seeds it again from its own environment;
• cmake --fresh, which seeds everything again.

A configure whose environment presents a different value does not apply it. It says so, naming the -D that would:

-- [cache] PACE_ADDR is '10.0.0.5:6674' in the environment and NOT applied: this build tree holds '127.0.0.1:6674', and the environment only seeds a tree with no PACE_ADDR entry -- -DPACE_ADDR=10.0.0.5:6674 applies it

This used to be the other way round: a change in the environment retargeted the entry unless it had been "changed by hand", and "by hand" was judged by comparing the entry with the value last applied. That comparison cannot see a -D that typed the value already held -- measured: a tree configured with dispatch off from one shell, reconfigured from a terminal holding PACE_SCHEDULER with -DPACE_SCHEDULER= to say off explicitly, came up on. A held value and a reported difference have no such blind spot.

Dispatch is decided at configure, and configure says which way. Beside the line naming the launcher, it prints one of:

-- [cache] dispatch: 10.0.0.9:6674 (configured, seeded from the configure environment and held in this build tree's cache; baked into the launcher, so the build's own environment does not change it; not probed -- each compile records DISPATCHED or DECLINED in invocations.log; -DPACE_SCHEDULER= turns it off)
-- [cache] dispatch: off (no PACE_SCHEDULER in the configure environment, and this build tree now holds that; every miss compiles on this machine -- -DPACE_SCHEDULER=<host:port> turns it on)
-- [cache] dispatch: off (PACE_SCHEDULER is empty in this build tree's cache, which outranks the 10.0.0.9:6674 in the environment; every miss compiles on this machine -- -DPACE_SCHEDULER=10.0.0.9:6674 turns it on)

Both open with [cache] dispatch: and then the endpoint or off, so one search finds the line whichever way configure decided. A value held from an earlier configure reads held in this build tree's cache, where a -DPACE_SCHEDULER or the configure that created the tree put it, and an environment presenting another scheduler adds the <host:port> in the environment is NOT applied -- -DPACE_SCHEDULER=<host:port> retargets it. The line is configuration, not reachability: nothing at configure asks the scheduler anything, and each compile records whether it was DISPATCHED or DECLINED in invocations.log.

The generated launcher carries exactly what the line says: PACE_SCHEDULER=<host:port> when dispatch is on, and PACE_SCHEDULER= (which the launcher reads as unset) when it is off. Set-but-empty and unset are the same thing in the environment here, unlike PACE_ADDR, because the launcher reads them the same way: neither dispatches.

So a PACE_SCHEDULER in the build's own environment no longer changes anything, in either direction. That is on purpose. The launcher used to read it only from the environment of whatever ran the build, so a value set in one shell and a build started from another compiled every miss locally -- measured on a workstation, 341 misses with NOT_CONFIGURED on each, noticed only because a fleet dashboard read 0 compiling -- while configure had said nothing about dispatch at all. And it is held rather than re-read because the first version of this re-read the environment at every configure, and ninja's own re-run from a build shell without the variable turned dispatch off in a log nobody reads (measured the same day). The configure probe never dispatches, whatever the setting: it asks whether the cache answers, and a slow or vanished scheduler must not make that question time out. Changing the value changes every compile command, so the next build recompiles everything -- served from the cache, since the key does not include the scheduler. ctest -R compile-cache-dispatch pins the printed line against the generated launcher, in both states, across reconfigures with and without a -D, and across ninja's own regeneration; it holds PACE_ADDR to the same rule. This applies only where the module chooses the launcher; a build that sets CMAKE_CXX_COMPILER_LAUNCHER=pace-cc itself still reads the environment at build time.

"Answers" is checked, not assumed: configure compiles one tiny translation unit through the launcher with PACE_VERBOSE=1 and accepts only a reported HIT/MISS, since a launcher whose daemon is down still compiles fine and would otherwise leave every TU paying a failed connect with nothing to show for it. That costs about 0.1 s against a daemon that answers, runs on every configure so that starting the daemon and reconfiguring is enough, and any other outcome — connect failed, a version mismatch, no daemon at all — falls through to ccache naming the address it tried; -DUSE_COMPILER_CACHE=OFF disables both.

A version mismatch is announced, not merely reported. It is the one rejection reason that says a daemon is answering and is one you installed — so it is a CMake Warning rather than a status line, and it says which end is behind: unsupported wire version 3; this server speaks 1..1 means this launcher speaks 3 and the daemon accepts 1 at most, so the daemon is the older of the two. Every other reason (not installed, no answer, an uncacheable probe) describes a cache nobody set up and stays at STATUS, because a warning that fires for all of them is one everybody learns to skip. Issue #815 is what that warning is for: a packaged daemon a release behind a hand-built launcher, refusing every exchange, with the build silently going through a different cache for weeks.

sccache is never selected automatically. It stays supported and keeps its place ahead of ccache, but only when a build asks for it with -DALLOW_SCCACHE_FALLBACK=ON; without that flag the module says Not using sccache and moves on. Being the unasked-for answer to pace-cc not working is exactly what made #815 invisible. (Pointing sccache at a pace-kv daemon as its storage backend is a different thing entirely and is unaffected — see below.) Where it is asked for, the fallback is still not equivalent, and the configure says so: selecting sccache under MSVC or clang-cl emits a CMake Warning naming the hazard below, because a build opted into it in silence is one whose symptom arrives hours later and somewhere else. -DUSE_COMPILER_CACHE=OFF opts out of caching entirely, and ctest -R compile-cache-caveat pins the warning, the silence for launchers that have no hazard, and both directions of the sccache gate.

On MSVC and clang-cl, one sccache cache must not be shared between checkouts

sccache preprocesses MSVC and clang-cl with /EP, which emits no line markers, so the text it hashes to find a cache hit carries no paths at all — while the /showIncludes stream it replays on that hit carries the absolute paths of the checkout that stored it. Two checkouts sharing one cache therefore record dependencies pointing into each other, after which editing a header in the checkout you are building rebuilds nothing: the build stays green and the objects are stale.

Measured on this project — a second worktree recorded 1097 dependency edges into the first and none into itself — and reproducible in two compiles: with sccache 0.14.0 and MSVC 14.51, a second directory took a cache hit from the first, and its /showIncludes named the first directory's header.

What is exposed is narrower than "sharing a cache". It is an incremental build across checkouts at different absolute paths. A clean build has no dependency graph to corrupt, and checkouts that all sit at the same path replay paths that are correct — a CI fleet is normally both, and this is not a reason to take the cache away from one. GCC and Clang are not exposed at all: their preprocessed output carries the paths, so two checkouts never share an entry to begin with — the same two compiles under g++ 14 were 0 hits and 2 misses.

A pace-kv-backed sccache is definitionally one cache shared by every checkout and every machine pointed at it, so the developer machines behind it are exposed even where CI is not. On MSVC and clang-cl, use pace-cc instead: it rewrites a hit's paths into the consuming checkout before replaying them, and refuses a hit whose replayed dependency is not there — which is why it does not have this failure mode, and why its entries are portable across checkout paths on purpose.

The probe carries its own ten-second cap, which is what bounds a remote address that drops packets rather than refusing them: PACE_TIMEOUT bounds the exchange, not the TCP connect(), so a firewalled or vanished host otherwise takes the kernel's connect timeout to fail (measured at 2m30s on macOS) — once per configure here, but once per translation unit in a build.

Installing it automatically

All of the above assumes pace-cc is already on PATH, which on a new repository or a fresh machine is a manual step someone has to remember. -DPACE_AUTO_INSTALL=ON removes it: when no launcher the build could actually use is installed — no pace-cc, no ccache, and no sccache that -DALLOW_SCCACHE_FALLBACK=ON has opted into — the module downloads a prebuilt pace-cc for the host's OS and architecture from the latest stable release, checks the SHA-256 the release publishes, confirms the binary runs here, and uses it.

It is off by default, because reaching out to the network during cmake is a different thing from using what is installed. It fires only when nothing else is available: a launcher you installed is a decision already made. And it cannot fail a configure — an unreachable network, an unpublished platform, a download that arrives corrupt each end in one status line and the same fall-through to ccache (or sccache, where it was asked for) or plain compilation that a missing launcher has always produced.

The binary is staged per user, under version and platform, so every repository and build tree on the machine shares one copy and later configures neither download nor ask. The release lookup is cached for a day, and honours GITHUB_TOKEN / GH_TOKEN where the unauthenticated limit of 60 requests an hour per address would otherwise be shared out among CI runners. Pinning PACE_AUTO_INSTALL_VERSION skips the lookup altogether, and pointing PACE_AUTO_INSTALL_DOWNLOAD_BASE at a mirror installs without reaching GitHub at all.

Auto-installing the launcher alone still leaves a genuinely clean machine uncached, since pace-cc has nothing to talk to: -DPACE_AUTO_START=ON additionally stages and starts a pace-kv daemon in the background — from the same release archive, persistently, off by default and independently of PACE_AUTO_INSTALL — when nothing answers at PACE_ADDR. cmake/portable/README.md covers it in full, including why it defaults off in CI as well as on a developer machine that has not opted in.

cmake/portable/README.md documents the full option set, and is written for projects vendoring the module rather than building this one.

Environment

Configuration is entirely environmental, so a launcher invocation stays a drop-in prefix. An empty value counts as unset.

pace-cc --help documents the same set, generated from the table in LauncherCli.cpp that is the launcher's single source of truth for these names. This page is the prose version; if the two ever disagree, --help is right.

Variable	Meaning	Default
PACE_ADDR	host:port of the cache — a pace-kv, or a pace-node's --listen-node. Hostnames, IPv4 literals, and bracketed IPv6 ([::1]:6674) all resolve. Set but empty is the opt-out and means no caching -- but see the PowerShell note below, where that spelling does not reach the launcher.	127.0.0.1:6674
PACE_SOURCE_DIR	Checkout source root, used for keying and path canonicalization.	unset — no caching
PACE_BINARY_DIR	Build output root.	unset — no caching
PACE_DEPS_DIR	Where this machine keeps third-party sources the build reads from outside the checkout — CPM's source cache. Keyed and replayed as a root of its own (<DEPROOT>), so two machines whose caches sit in different homes share objects; an object naming it is bound to the machine, as one naming the checkout is bound to the checkout.	unset — such paths are keyed as written, which shares only between builds keeping them in one place
PACE_PREFETCH_GROUP	Prefetch grouping id. Not part of the cache key, so it never partitions the cache.	default
PACE_VERBOSE	Print HIT/MISS and fall-back diagnostics to stderr.	unset (quiet)
PACE_NO_STATS	Do not record invocations to the statistics log.	unset (recording on)
PACE_NO_DIRECT	Disable direct mode, always preprocessing to derive the key.	unset (direct on)
PACE_CONNECT_TIMEOUT	Deadline for opening a connection — name resolution included. Like the three below, a duration: a whole number and one of ms, s, min, h, d; a bare number is not one and is ignored. 0s leaves the platform's own, which runs to minutes. Short on purpose: a cache that has not accepted within a second is one the build is better off without, and a wedged resolver would otherwise stall every translation unit. The mint's connect is capped below it: asking this machine's own node for a machine ticket, over loopback, gives up connecting after 50 ms whatever this says — a lower value lowers it, and 0s does not lift it — so a machine with no node pays about that once per compile rather than a second per exchange. pacectl and the node's operator verbs mint under the same bound.	1s
PACE_TIMEOUT	Deadline for one whole exchange with the daemon — or with a scheduler's LEASE/RELEASE — from the request to the last byte of the reply. 0s removes the bound. A daemon that accepts and then stalls, or dribbles one byte at a time, would otherwise block the compile forever. Bounds one exchange, not the whole invocation — see below — and not a remote compile.	10s
PACE_DISPATCH_TIMEOUT	Deadline for one whole COMPILE exchange with a worker. 0s removes the bound. Far larger than PACE_TIMEOUT because it bounds a different shape of conversation: a worker writes nothing until the compiler has finished, so the client waits out the entire remote compile in one read. Ten minutes because that is the scheduler's own lease timeout — waiting longer means waiting on a lease it has already reclaimed. See Distributed compilation.	10min
PACE_DISPATCH_IDLE	Deadline on silence during a COMPILE exchange. 0s removes the bound. A worker writes a five-byte progress frame every few seconds while it is compiling, so this bounds how long it may say nothing rather than how long the compile may take — which is what lets it be seconds while PACE_DISPATCH_TIMEOUT stays minutes. It is the only thing that sees a worker whose machine answers every keepalive probe while the process makes no progress. On expiry the launcher compiles locally and hands the lease back, and its fall-back line (with PACE_VERBOSE) reads stopped reporting progress rather than ran out of budget. See Distributed compilation.	30s
PACE_MAX_STORE_BYTES	Largest compiled result the launcher will offer to the daemon; 0 means no limit. A bigger result is simply left uncached. Matches the daemon's --storage-max-value default by construction rather than by negotiation — there is no handshake, so raise both or the other keeps refusing.	268435456 (256 MiB)
PACE_SCHEDULER	host:port of a fleet member's --listen-node port: a pace-node that runs consensus, whether it schedules (a voter) or not (a learner). On a miss the launcher asks it for a worker and sends that worker the preprocessed translation unit. Every refusal falls back to a local compile, with one exception: not-leader is an instruction rather than an answer about the fleet, so the launcher retries against the endpoint the refusal names (up to two hops, then it compiles locally). This value therefore only has to be a member of the cluster, not the current leader, and no launcher needs re-pointing after an election. A learner schedules nothing itself but answers a lease the same way, with not-leader naming the leader's scheduling endpoint, so this machine's own node (127.0.0.1:6674) is a value that works on every machine of a fleet whoever leads; the lease is released at the scheduler that granted it. A learner that has not heard from its leader for longer than an election timeout names nobody, and the miss compiles locally. A learner older than this release refuses the lease instead, and every miss compiles locally. The workers follow the leader too: a serving node registers where its fleet's formation record says the voters answer, follows not-leader when it registers and heartbeats, and remembers where the leader answered, so an election re-points the whole fleet rather than just the clients. Both halves are needed: a launcher that followed the redirect while the workers did not would reach a leader whose registry they had all expired out of, and every lease would answer no-worker. A cache that is unreachable or refuses counts as a miss for this purpose; it does not disable dispatch. See Distributed compilation. A build configured through cmake/portable/CompileCache.cmake has this decided at configure and baked into its launcher, so the build's own environment does not change it (see Usage).	unset — every miss compiles locally
PACE_TOKEN	The password for a cache at PACE_ADDR started with --requirepass, presented to that cache alone. Costs no round trip — it is pipelined ahead of the real command, not awaited. Safe against a cache that requires none: such a cache accepts it and ignores it. A compile node needs none: every exchange with another machine — the scheduler, a worker, a remote cache — presents a machine ticket this machine's node mints for that exchange alone, over loopback at PACE_ADDR's port, and a token sent there would admit nobody.	unset — no password sent
PACE_USER	Username to accompany PACE_TOKEN. Unset (the usual case) authenticates against the secret alone, which is what --requirepass configures. Ignored without a token — a username on its own is a misconfiguration, not a request to authenticate, and sending an empty secret would be refused by every server that wants one.	unset
PACE_VERIFY	Verify one hit in every N by compiling the translation unit again and comparing the objects — see Verifying that a hit is the right object. Costs a whole compile per verified hit, so it is for CI, a nightly, or reproducing a report. 1 checks every hit. Which hits are sampled is decided by hashing the key rather than by chance, so the rate holds over a build and a translation unit that verified verifies again. A value that is not a whole number reads as off rather than as an error: this is a diagnostic set by hand, and refusing to compile over a typo in it would break the build it was brought in to investigate.	unset (off)
PACE_MSVC_DEPS_PREFIX	The prefix a dispatched compile's synthesised /showIncludes notes carry — that is, this build's msvc_deps_prefix. Only dispatch needs it: a worker compiles preprocessed text and reports no dependencies, so the launcher writes the record itself, while a local compile emits the compiler's own notes and needs nothing here. Ninja matches the prefix literally and knows nothing about languages, so an English note against a localized msvc_deps_prefix records no dependencies at all for that translation unit and the next header edit does not rebuild it — see A localized MSVC toolchain.	unset — the English Note: including file:. Set it to auto to have the launcher ASK the compiler instead; see A localized MSVC toolchain for why that is opt-in

The statistics log is located from the usual per-user state variables rather than one of the launcher's own. These are read but never written:

Variable	Meaning	Default
LOCALAPPDATA	(Windows) Base for the log directory, %LOCALAPPDATA%\pace-cc.	unset — no statistics recorded
XDG_STATE_HOME	(POSIX) Base for the log directory, $XDG_STATE_HOME/pace-cc. Preferred over HOME.	unset — fall back to HOME
HOME	(POSIX) Base for $HOME/.local/state/pace-cc, used when XDG_STATE_HOME is unset.	unset — no statistics recorded

With no usable state directory there is nowhere to append to, so statistics are silently disabled. Caching itself is unaffected.

ADDR, SOURCE_DIR and BINARY_DIR must all be non-empty to cache. ADDR has a default and the other two do not, so in practice it is the roots that are missing when nothing caches — the build still succeeds, which is exactly why this is worth checking before concluding the cache does not help. With PACE_VERBOSE set, that case reports missing PACE_ADDR/SOURCE_DIR/BINARY_DIR.

The empty opt-out does not work from PowerShell

export PACE_ADDR= is a genuine set-but-empty entry on a POSIX shell, and the launcher reads it as the opt-out. PowerShell cannot express it. $env:PACE_ADDR = "" leaves the name present in PowerShell's own view -- Test-Path Env:PACE_ADDR answers True and $env:PACE_ADDR prints empty, exactly as an operator expects -- while the child process never receives the variable at all. pace-cc therefore sees it unset, falls back to the 127.0.0.1:6674 default, and keeps caching. Nothing reports this: the opt-out silently does not happen, and the one place anybody would check agrees that it did.

[Environment]::SetEnvironmentVariable('PACE_ADDR', '', 'Process') behaves the same way, so it is not a spelling problem with a workaround. From PowerShell, opt out at configure time instead -- -DPACE_ADDR= (the deliberate form, which wins over the environment) or -DUSE_COMPILER_CACHE=OFF -- both of which travel as command-line arguments rather than through an environment block.

The CMake integration honours the same opt-out, and until #372 it did not. cmake/portable/CompileCache.cmake decides whether the launcher is used at all, and it folded a set-but-empty PACE_ADDR into the default -- so on a POSIX shell, where the spelling above does travel, export PACE_ADDR= reached the launcher as an opt-out and reached the build integration as "say nothing", and the build went on being fronted. The two readings of one variable now agree: absent means the default, present-and-empty means no caching, in both -- on the configure that creates the build tree's PACE_ADDR entry. After that the tree holds its address and the environment only seeds a tree with no entry, so a later export PACE_ADDR= is reported as not applied and the launcher keeps the held address; opt a tree that already holds one out with -DPACE_ADDR=. The launcher itself reads the variable at every compile only where the module did not choose it. Note this is why the paragraph above still sends a PowerShell user to -D: that platform's problem is that the variable never reaches the child at all, which no predicate on the receiving side can repair.

PACE_ADDR defaults to 127.0.0.1:6674 rather than to nothing, so the launcher caches with no configuration at all against whichever of pace-kv or pace-node is running on this machine — the node's --listen-node defaults to the same address for exactly that reason. A remote default would be indefensible, since every translation unit on a machine with nothing listening would pay a connect timeout in silence; a closed loopback port refuses immediately, so a machine running neither pays microseconds per compile.

A localized MSVC toolchain

A Visual Studio carrying a language pack prints /showIncludes notes in that language, and CMake records whatever it printed as Ninja's msvc_deps_prefix. Ninja then matches that literal string, so the prefix a note carries has to equal it byte for byte.

That matters here on one path only: a dispatched compile. A worker compiles preprocessed text, which has no #include left in it, so the worker's compiler reports no dependencies and the launcher writes the record itself from what its own probe saw. Until #700 it wrote the English Note: including file: unconditionally. On a localized build Ninja matched none of those lines, recorded no dependencies for the translation unit, printed the notes as ordinary compiler output, and stopped rebuilding it when its headers changed — a wrong build under a zero exit code, which persists until someone cleans.

Set PACE_MSVC_DEPS_PREFIX to the value your build already holds:

Select-String msvc_deps_prefix build.ninja      # in your build directory
$env:PACE_MSVC_DEPS_PREFIX = 'Hinweis: Einlesen der Datei:'

The line that comes back is the whole assignment — msvc_deps_prefix = Hinweis: Einlesen der Datei:. What goes in the variable is the text after the =: a prefix that carries the assignment matches no line Ninja ever sees, and fails in exactly the same silence as writing English.

Ninja's behaviour here is measured rather than assumed, and the measurement is re-runnable: scripts/probes/ninja-msvc-deps-prefix.sh drives three cases (matching, mismatched, and localized-matching) and reports the dependency count and whether a header edit rebuilds. It needs no MSVC and no language pack, because the defect is a property of the string, not of any compiler's UI language.

Where the prefix comes from, in order. msvc_deps_prefix is a value the build holds and never exports, so there are three sources and the launcher takes the first that answers:

PACE_MSVC_DEPS_PREFIX	What the launcher does
A prefix, e.g. Hinweis: Einlesen der Datei:	Uses it. This is the only source that can be right about a build the launcher cannot see: you copied it out of build.ninja, so you are stating a fact.
auto	Asks the compiler. Writes a one-header translation unit somewhere temporary, preprocesses it with /EP /showIncludes, and reads the prefix back off the line that ends in the header it just wrote. Falls back to English if the compiler will not say.
unset	The English Note: including file:, on trust.

With PACE_VERBOSE the launcher names the prefix it used and where it came from, on every dispatched compile that synthesises notes — the one line that answers why did my build stop rebuilding this file.

Three settings, but four things that line can say, because auto has two outcomes and they need different remedies:

It says	What happened	What to do
named by PACE_MSVC_DEPS_PREFIX	You stated the prefix.	Nothing.
discovered from the compiler	auto, and the compiler answered.	Nothing.
the compiler was asked and did not say, so the default; name it with PACE_MSVC_DEPS_PREFIX	auto, and the probe learned nothing — so this build is on English and your notes may not match.	Copy msvc_deps_prefix out of build.ninja and set it directly.
the default; override with PACE_MSVC_DEPS_PREFIX	Nobody asked for anything; English on trust.	Nothing, unless your toolchain is localized.

The last two carry the same prefix and are told apart only by that line. They are separate because the remedy differs: override with PACE_MSVC_DEPS_PREFIX is advice already taken by whoever wrote auto, and telling them to do it again is a confident wrong answer to the one question they asked.

auto is opt-in because it costs a compiler spawn per file. One pace-cc process serves one translation unit, and under CMake + Ninja + MSVC /showIncludes is on every compile line — so a probe that ran whenever nobody named a prefix would start a second compiler for every file in your build, including on cache hits, where not running a compiler is the entire point. On an English toolchain it would pay that to rediscover the default. And the answer cannot be remembered to spread the cost: installing a language pack changes what the notes say without changing anything a cache stamp covers, so a remembered answer goes stale in the direction that produces a wrong result which looks right.

The chain, stated so nobody optimises it back into a cache later: the answer cannot be memoized, so the only correct automatic shape is per translation unit; per translation unit is too expensive to impose on every build; therefore opt-in. Each link depends on the one before it. If you find yourself about to add a cache here, the link that has to break first is the first one — and it does not break, because the thing that invalidates a discovered prefix is an installer run that touches no cache stamp.

So auto is for the build that needs it — and if you already know the string, setting it directly is both cheaper and more certain.

The probe runs in your build's own environment rather than the anglicized one the launcher uses for its other spawns, because the question is what your compiles print.

Two things it will not do, both deliberate. It refuses a line with blanks in front of the prefix, because a note begins at column zero — cl puts the inclusion depth after the prefix, not before it. And where two lines suggest two different prefixes it reports nothing rather than picking one: a #pragma message whose text happens to end in a path looks exactly like a note to this rule, and a wrong prefix is worse than none.

The localized case is untested against a real localized compiler, and that is recorded rather than implied: VSLANG selects among the language packs an installation has, so an English-only Visual Studio cannot be made to produce a German note. #878 stays open for that half. If you run a localized toolchain, set the variable.

Sharing a cache across UI languages is safe from generation 3 onward, and this paragraph records what it took, because the hazard was real and an operator running a mixed fleet on an older build still has it.

A note's prefix is now a canonical form, exactly as <SRCROOT> is a canonical form for a checkout root. The launcher rewrites its own prefix to the English Note: including file: before a value is stored, and rewrites it back to this build's prefix when a hit is replayed (NormalizeIncludeNoteMarker and RestoreIncludeNoteMarker — the direction is a name, not an argument position) — so what crosses the wire is locale-free, and neither machine ever sees the other's prefix. That is why nothing on either server changed: they only ever canonicalize the one spelling. It is the launcher that does both rewrites, and it has to be, since only the producing machine knows what language its own notes are in.

Before that, a German machine setting this variable fixed its own builds and began storing values whose notes an English peer replayed and could not match — new breakage on a machine that had none, since the cache key does not fold the prefix (a German and an English Visual Studio of the same toolset key identically, deliberately, because the identity probe is forced to English, #692). Closing it moved CompileValueVersion to 3 (#879), so a value written by a generation-2 build is refused rather than replayed.

The byte has moved three times since: to 4 for #202, to 5 for #1270, which put back the half of the note anchor #891 gave away, and to 6, where an MSVC-family compile's replayed warnings, errors and note: lines stopped naming the checkout that stored them and a stored path's .. segments are collapsed (#1593). Each of those is its own cold cache, and the paragraph below applies to every one of them unchanged. Expect one cold cache on the upgrade — one, because a refused generation now falls through to the miss path and the STORE that follows overwrites the key with a value of this generation. A bump that refused and then declined to re-store would leave the old value under a key that does not change with the generation, fetched and refused on every later build: not a cold cache but a dead one.

What this still does not cover. A local compile on a localized toolchain that has not set this variable stores a region the launcher cannot normalize, because nothing has told it what prefix that compiler uses — an unmatched marker rewrites nothing, so such a machine is exactly where it was rather than worse. Discovering the prefix from the compiler is #878, and it needs no further generation: the stored form is already locale-free, so #878 only supplies a better value to normalize with. Setting this variable fixes a localized machine's dispatched notes, and for an ASCII prefix it also gets that machine's stored regions canonicalized.

It does not do the second for a non-ASCII prefix, and that is a known gap rather than a claim being made carefully. The value arrives from the environment in this process's narrow text, which this project forces to UTF-8, while a local cl writes its notes in the console output code page — the same mismatch RootReconciler is handed a HostNarrowTextPolicy to deal with for paths. German happens to work because its prefix is ASCII; the Japanese and Chinese catalogues use a full-width colon (U+FF1A) and the Russian one is wholly non-ASCII, so the two byte strings differ, nothing matches, and the region is stored uncanonicalized with no diagnostic. A dispatched compile on the same host is unaffected, because the launcher synthesises those notes from the same UTF-8 value it compares against. Decoding the marker through the same policy as the paths is the repair, and it belongs with #878, which is already the ticket for learning the prefix properly.

How it works

1. Key. Direct mode first: re-hash the project headers a previous compile recorded and look up a manifest — far cheaper than preprocessing (~18 ms versus ~1.4 s on a large translation unit). If that misses, preprocess the TU, relativize checkout-rooted arguments against SOURCE_DIR/BINARY_DIR, and hash (compiler id + preprocessed text + relativized args + dependency paths) into a 128-bit key — MurmurHash3 x64_128, which is 128 bits of strength and not merely 128 bits wide; see the note in .agent/rules/compile-cache.md on why the four-CRC construction it replaced was not. Compiler id is the driver's banner and the target it generates for, where the driver can be asked: a banner alone identifies the driver, and a driver's code generation is not a function of the driver alone — clang-cl takes -fms-compatibility-version from the MSVC install beside it, and one stock g++ banner covers x86_64 and aarch64. A driver that states no target keeps its entries unchanged. Note this is the key, not the toolchain fingerprint a dispatched compile matches workers on, which deliberately omits the target — see Distributed compilation. The dependency set comes from that same preprocess run — -MD into a scratch depfile for GNU drivers, /showIncludes for MSVC ones — which costs about 1.5% of it, because the compiler has already opened every one of those files.
2. FETCH. On a hit, write the object to the requested output path and replay the cached stdout and stderr on their own channels, with header paths localized to this machine so the build tool's dependency records stay valid.
3. MISS. Run the real compiler capturing stdout and stderr separately, STORE the canonicalized result, and pass the output through on the true streams.
4. Any error. Fall back to a plain real compile.

A compile that fails is never cached. The object is stored once, under the preprocessed key; a manifest records that key rather than a second copy of the object, so a direct hit follows one extra fetch instead of doubling the cached volume (which, since the memory tier keeps values uncompressed, would land on RAM where compression cannot help).

An object that names its checkout is not shared with another one

The key is portable across checkouts by design, and a few things a compiler puts into an object are not: __FILE__ expanded under an absolute source path (every CMake compile), std::source_location, MSVC's assert message. Those name the checkout that compiled the object, and served into another checkout they point assertion messages and log lines at a tree nobody is building. No key can see them -- the direct-mode key never sees the __FILE__ expansion, and source_location is filled in after preprocessing -- so the launcher reads the object before storing it.

An object whose program data names neither PACE_SOURCE_DIR, PACE_BINARY_DIR, PACE_DEPS_DIR nor the compile's working directory is stored and shared exactly as before. One that names any of them is stored under a key that also folds in the directories it NAMES -- each absolute and resolved, so a relative export (.) identifies the checkout as well as an absolute one -- with a marker in its place under the ordinary key that lists them: the checkout that stored it hits it as usual, and any other checkout misses, compiles, and stores its own copy. Only what the object names is folded, so an object whose __FILE__ names the source tree is still shared by a second build directory of the same checkout, and one naming the build directory is not. The working directory is looked for because it is what cl /FC makes a relative path absolute against, whatever the roots say. Debug records (.debug$S, DWARF) are not counted -- see Debug paths in a replayed object -- so a debug build shares as much as it did. PACE_VERBOSE says which it was: root-bound object (it names … in .rdata (UTF-16LE)); storing it under key=… on the compile that stores, and MISS key=… (root-bound: the cached object names its build-tree; …) or HIT key=… (root-bound: served from key=…) on the ones after it.

The scan errs on the side of calling an object bound, because the opposite mistake serves another checkout's paths: a spelling it over-recognises costs that translation unit its cross-checkout sharing and nothing else. Whatever cannot be read as bytes is always treated as bound: an LTO object, clang's coverage map (-fcoverage-mapping, which keeps the source's absolute path in a compressed stream), and any compressed section. A ? where a root has a character the compiler's code page cannot hold (cl without /utf-8) is matched too.

Why the dependency paths are in the key

A hit reproduces two things: the object file, and the build system's dependency record — a GNU depfile, or the /showIncludes notes Ninja reads as deps = msvc. A cached value that cannot reproduce the depfile a compile names is not served: it is recompiled and stored again. Note that clang-cl under CMake's Ninja generator asks for a GNU depfile (-clang:-MD -clang:-MF<file>, deps = gcc), not for /showIncludes; the launcher reads that spelling, and a separated -clang:-MF -clang:<file> is compiled uncached, since its value cannot be read. Suppressing line markers keeps every path out of the hashed text, which is what makes a key portable across checkouts, and equally what once made it identical after a header moved: same bytes, new path, so the object was still correct and still served while the recorded paths were not. Replaying those makes the build system rebuild that translation unit on every build, forever, with a successful exit code each time.

Naming the dependencies in the key makes a move a different key, so the two layouts hold two entries and moving a header back finds the original one intact. Only machine-independent paths are hashed — those under the source root or build tree, plus relative ones. Toolchain and system paths are left out on purpose: they are the producing machine's spelling, the compiler identity in the key already covers them, and hashing them would stop two machines with the same compiler at different install prefixes from sharing anything at all.

That last exemption leaves one case open, so a hit is still checked before it is written: every dependency path it records that this machine is answerable for must exist. A hit that fails the check is discarded and the compile runs for real, which re-stores the entry with a correct record.

Reading the dependency set: line

Every preprocessed compile reports, under PACE_VERBOSE, what became of the paths the driver named:

pace-cc: dependency set: 61 of 635 reported path(s) keyed (476 toolchain; 60 filesystem call(s))

M is what the probe reported, N is the key's own dependency set, and the words before the semicolon say why the rest did not reach it.

They will usually not add up, and that is not a fault. M and the drop counts are per reported occurrence; N is the set after deduplication. /showIncludes names a header once per inclusion site, so the 159 occurrences left after the 476 drops above are 61 distinct files. A GNU depfile names each header once, so on those drivers the numbers do sum. What always holds is that the reasons account for every path that did not reach the key:

Reason	What it means
toolchain	Under neither root, or inside a vendored tree. The ordinary bulk of any compile: the compiler identity in the key already covers this content collectively. That identity is the compiler's own version banner — Microsoft (R) C/C++ Optimizing Compiler Version 19.51.36252 for x64, clang version 22.1.3 (…) — so two MSVC toolsets, or the x86 and x64 driver of one, key apart even though their headers are the same files.
drive-relative	A Windows C:foo path under neither root. It resolves against that drive's own current directory — per-process state on the producing machine that no cache entry can record. One under a drive-relative root is keyed like any other, or counted toolchain if it is vendored content.
unanchored	Relative, with no working directory to resolve it against.
no canonical form	Under no root, yet a character prefix of one: a root spelled almost right, such as /x/build-other against /x/build. Fix the root rather than looking for the file.
empty	The driver named an empty path.

0 of M with M non-zero is the line to act on, and the reason says which repair. Every path counted toolchain means the configured roots match nothing the compiler echoed back — on Windows usually an 8.3 short name in one of them, which is otherwise entirely silent: the key is empty, the replay guard has nothing to check, and the stored value keeps this machine's absolute paths. drive-relative or unanchored point at the build's own spelling of its include paths instead. 0 of 0 is a different fault: the driver reported nothing at all on the preprocess line.

Statistics

Each compile is its own short-lived process, so aggregation cannot live in memory. Every invocation appends one line to a per-user log and reporting folds it on demand:

• Windows: %LOCALAPPDATA%\pace-cc\invocations.log
• Elsewhere: $XDG_STATE_HOME/pace-cc/invocations.log, falling back to ~/.local/state/pace-cc/

Writes use an atomic append (FILE_APPEND_DATA / O_APPEND) so the hundreds of concurrent compilers in one build interleave whole lines instead of shredding each other's. Recording failures are swallowed: statistics never break a build.

A line is tab-separated and names its own format first: v2, then fifteen columns in this order — outcome, prefetch-group, value-bytes, elapsed-ms, source, detail, preprocess-ms, cache-ms, direct-ms, direct-hit, timestamp (seconds since the Unix epoch), dispatch, dispatch-detail, dispatch-specifics and exit-code, the code the compile exited with. A line with no version was written by an older launcher and is read by how many columns it has, as it always was; its exit code is absent, never a zero that would call every old compile a success. A line naming a version this launcher does not know — a newer launcher appending to the same log during an upgrade — is skipped, and --show-stats says how many it skipped, rather than being read by position and misread. The report counts failed compiles only among records that carry an exit code, which is what tells a compiler killed from outside apart from a cache that broke the build.

INPUTS-MOVED is a compile one of whose inputs changed while the key was taken or during the compile — an editor's save, a git checkout in another terminal — so the launcher did not store its object (a direct-mode manifest built from the content the key was taken over may still be recorded); the detail column says which of the two moments it moved in, and the next build caches as usual. An input that could not be observed at all — unreadable when the key was taken — is not a race and is recorded UNCACHEABLE instead. detail carries the fixed sentence only, so --show-stats ranks one row per cause rather than one per file; the file itself is named on the PACE_VERBOSE line.

The two dispatch columns are the distribution axis: the fixed reason tallied under why distribution did not help, and then what to act on about it when the reason alone does not say — the argument a worker would not take, in the worker's own words, the ceiling a job went over, what one worker could not do, or the flag this launcher would not send. A decline whose peer text would only name an endpoint, such as a redirect chain that found no leader, leaves it empty; PACE_VERBOSE carries that. The last column is recorded and never tallied, so the report stays one row per cause while the log still names the flag. It is cut at 320 bytes between characters, holds no control character (C0, DEL or C1), and spells a byte that begins no UTF-8 character as ?, since part of it is a peer's text.

$ pace-cc --show-stats

all prefetch groups
  compiles     : 4
  hits         : 2  (66.7% of 3 cacheable)
    via direct : 1
  misses       : 1
  unavailable  : 1  (25.0% of all compiles -- CACHE NOT REACHED)
  fall-back reasons
    1x  fetch exchange failed
  distribution
    not attempted   : 2
    unreachable     : 1
    dispatched      : 1  (50.0% of 2 asked of the fleet)
    why distribution did not help
      1x  the fleet could not be reached

  hit latency    2 samples  p50 12ms  p95 70ms  max 70ms
    preprocess   1 samples  all 65ms
    cache i/o    2 samples  p50 0ms   p95 1ms   max 1ms
  miss latency   1 samples  all 200ms

never cached (1 translation units)
  1x  volatile.cpp

The hit rate is computed over cacheable compiles — hits, misses, wrong objects (hits PACE_VERIFY rejected) and compiles an input moved during (INPUTS-MOVED), every compile the cache was asked for — not all invocations, so an unreachable daemon does not silently look like a low hit rate — it is reported separately as CACHE NOT REACHED.

Latency is shown as a distribution rather than an average because compile latency is routinely multi-modal: a mean of 300 ms tells you nothing about whether that is every TU or a fast majority plus a slow tail. Fall-back reasons are itemized so unavailable is actionable — a refused connection and a rejected STORE call for very different responses.

The distribution section

The cache and the fleet are two failure domains, and this report keeps them on two axes. A compile can be a cache miss and a dispatch failure at once: the daemon answered honestly, the object was compiled and stored, and the fleet was still no help. Recording the second as a cache outcome would inflate unavailable and send you to look at the daemon, so it has a section and a reason list of its own — why distribution did not help is never merged into fall-back reasons above it.

The section appears only when there is a fleet to report on. With no PACE_SCHEDULER set, this launcher does not distribute, and printing dispatched: 0 (0.0%) for it would render an absence as a total failure. So it prints nothing at all — as does a log written before this section existed, which is silent about dispatch rather than evidence of no fleet.

Line	Meaning
not attempted	A scheduler is configured and this compile never reached the dispatch decision — the cache served it, it was uncacheable, or the key was never derived. Nothing about the fleet follows from it.
refused here	This launcher would not send the compile. The fleet was never asked, so this is fixed on your machine, not in the fleet — see the reason table below for which of the three it was.
fleet declined	The scheduler or the worker said no: no worker on this toolchain, no capacity, the key already in flight, or the worker refusing the job. Ordinary; the question is the fleet's size or shape.
unreachable	The scheduler or the worker could not be reached, or an exchange broke. Also ordinary, and fixed somewhere else entirely — a machine that is down or a network that is not carrying, rather than a fleet that is merely busy. That difference is why these are two lines and not one.
crossed reply	A worker answered about a compile other than the one it was asked for, and the object was refused unread. A defect somewhere in the fleet, not a fleet declining to help — see the entry in the fall-back table above, which is where its reason is ranked.
result discarded	A worker ran the compiler and this client did not keep the object: the remote compile exited non-zero and was retried locally, or the object or depfile could not be written here. The exchange worked and the compile was still done twice.
dispatched	A worker ran the compiler and the object was used. Distribution worked.

The rate is taken over what was actually asked of the fleet — fleet declined, unreachable, crossed reply, result discarded and dispatched — and not over every compile. not attempted and refused here are excluded because the fleet never saw them, so a build that mostly hit the cache does not read as a fleet that mostly refused. It is the same reasoning as the hit rate being taken over cacheable compiles: an absence must not be counted as a failure. When nothing was asked at all the rate reads n/a rather than 0.0%.

result discarded is a line of its own for the same reason. Folded into dispatched with an explanatory reason underneath, a fleet whose every object was thrown away would headline 100% dispatched — a green number over work done twice. It is also the only place a node that fails compiles which are fine is visible at all, because the local retry succeeds and the build stays green; a rising count here is the signal, and the reason says which of the three causes it was.

The dispatched line prints even at zero. A fleet that dispatched nothing is exactly what this section exists to make visible, and omitting the line would read as "no data" rather than as "none". Every other line is dropped when it is zero.

--html-stats renders the same states as a panel, driven by the same table, so a state cannot appear in one report and be missing from the other.

Why distribution did not help

Every reason that appears under why distribution did not help. All of them still produce a correct object — the translation unit is compiled locally instead — and none of them is a caching failure.

Reason	Meaning
the command line is not dispatchable	RemoteCompileArgs found something on the line it cannot account for, so nothing was sent. Refusing costs one local compile, where stripping an unrecognised argument would change the generated code and hand back an object nobody asked for. PACE_VERBOSE names the offending flag, and so does the last column of the invocation log; it is deliberately not in this tally, or you would get one row per command line instead of one per cause.
the command line carries an argument no worker passes to a compiler	An argument on the line is one every worker refuses by a fixed rule -- a sub-tool pass-through such as -Xclang, a plugin loader, a linker switch, a path the worker would open. --allow-compile-arg cannot lift these, so the launcher reads the same table the workers read and compiles locally without asking for a lease. The last column of the invocation log names the argument.
the dispatch preprocess failed	A worker is fed a second preprocess, with #line markers that the cache key's copy suppresses, and that run failed. The key's own preprocess had already succeeded, so this is about the marker-emitting form of the command specifically.
this toolchain has no usable fingerprint	The toolchain digest does not identify this compiler, so no worker could match it. Deliberately refused here rather than sent: a scheduler asked for an unidentifiable fingerprint answers NoWorker, which reads as "the fleet has nobody on your toolchain" and sends you to look at the fleet for a problem on this machine. pace-cc --print-toolchain-fingerprint <compiler> says what this machine computes and why it is unusable.
no worker serves this toolchain	Nothing in the fleet carries your compiler's fingerprint. A machine is missing, or a fingerprint has drifted — upgrading the toolchain on the clients and not the workers looks exactly like this. Never read as a capacity problem: more machines of the wrong compiler change nothing.
the fleet was full of its own work	Every matching worker was busy with this fleet's own compiles. The fleet is too small, which is a different purchase from the row above and from the one below -- or, where a worker reads limited-by: compile-budget, its compiles hold its --compile-inflight-max instead: raise that on it.
matching workers had withdrawn their slots	Machines that serve your toolchain had slots free on paper and were not offering them — somebody else is using them, a scratch disk is full, or a survey is in flight. The fleet is big enough and unavailable.
a node had no room for another request (endpoint-busy)	A node answered endpoint-busy: its front door was full, for one of two reasons the launcher cannot tell apart, since both answer the same code. Its in-flight byte budget was full — on a worker, every compile running there holds its whole preprocessed translation unit until it answers, and together they filled the budget; raise the worker's --compile-inflight-max, or let it derive a larger one from more --slots (how it is sized). Or its listener was at its connection cap — the likelier cause when a scheduler refuses a lease, whose payload is kilobytes; that is a ceiling on concurrent conversations, which no byte budget changes. The node's own counters separate them: pace_worker_jobs_refused_endpoint_busy_total moves for the compile budget, pace_node_frame_connections_refused_at_capacity_total for the connection cap. A third case moves neither: a LEASE refused by the scheduler node's listener budget, which the scheduler surface deliberately leaves uncounted — a momentarily full surface is something the peer sees and retries, and summed into the scheduler's security series it would make that series unreadable. So this tally rising with neither counter beside it points at a scheduler node. Not the row above — nobody else is using the machines, and adding machines is not the remedy either.
another client was already building this key	Duplicate-work suppression, and not a failure: another client holds the lease for this exact object and this compile ran locally instead. Sixty clients missing one key after a header change is the ordinary shape of a shared cache. A high count here is the design working.
the fleet refused this client	A credential, membership or lease refusal — the fleet would not let this client ask. Fixed where the client is configured or on the scheduler's member list, never by adding machines.
the worker refused the job	A lease was granted and the worker then said no: a lease it would not honour, a scratch root it cannot write, a compiler it could not spawn. One machine to go and look at, and PACE_VERBOSE names it.
a worker would not take an argument of this compile	The worker's allowlist has no row for a flag on this command line, which every worker of the same build refuses alike — so it is a flag this fleet does not dispatch, not one machine. The last column of the invocation log carries the worker's sentence naming every argument it refused, so a line needing several entries shows all of them at once. If it runs no program and names no path, add it on the workers with --allow-compile-arg; otherwise those compiles stay local, correctly. Until the worker had a code of its own for this it answered malformed-frame, and this row read as the wire disagreement below.
the job is larger than a worker accepts	The translation unit's frame, or the size its compressed envelope declares, is over the ceiling a worker's compile surface takes. That ceiling is the worker's --compile-request-max (256 MiB unless raised). In a fleet that leaves it at the default, or sets it alike everywhere, every worker refuses the same unit: a property of the unit, not of one machine and not a wire disagreement. In a fleet whose workers set it differently it is one machine's setting instead, and the worker's sentence names the ceiling it applied. Either way the compile runs here, correctly, and the last column of the invocation log carries the worker's sentence naming the ceiling.
the fleet named no leader to ask	The scheduler chain answered NotLeader until the redirect ceiling. Transient during an election and permanent when a fleet is misconfigured; the rate is what separates those, which is why it is not folded in with the credential refusal above.
this launcher and the fleet disagree about the wire	A protocol, codec or framing mismatch, or a peer answering a compile with a refusal from another surface. Expected and bounded during a staggered upgrade; a count that keeps rising afterwards names a machine that never came back. Two machines of one build cannot produce it.
the fleet refused with a reason this launcher does not know	A refusal code newer than this launcher. Upgrade pace-cc; the verbose line carries the code the fleet actually sent.
the fleet could not be reached	Something past the LEASE did not answer, broke mid-reply, or ran out of budget — the scheduler's half is the four rows below, so this is usually the worker the lease named, and the verbose line names it.
the scheduler made no connection: its name did not resolve, it refused, or no connection completed within the connect bound	The LEASE never reached the scheduler. If every compile shows this, check the address in PACE_SCHEDULER before suspecting the fleet — a wrong one, or a name that does not resolve, looks exactly like a fleet that is entirely down. Rising in bursts on a busy build, it is the scheduler node's accept queue overflowing (#1668).
the scheduler's connection broke before it answered	The scheduler took the connection and it broke before the lease was answered — the scheduler node stopped or restarted mid-exchange.
the scheduler connected, and no answer arrived within the bound	The connection completed and no lease answer came back in time. A completed connect is the kernel's, not proof the scheduler's loop took it, so this is a scheduler node too busy to answer — or one whose accept loop has stalled.
the scheduler stopped reporting progress	The scheduler's progress pulses stopped mid-exchange: the node is alive at the TCP level and its answer stalled.
a worker compile failed and was retried locally	The remote compiler exited non-zero, so the result was discarded and the translation unit compiled here to confirm. Broken code produces this on every machine and is not a fleet problem; a rising count against a build that keeps succeeding is a worker producing failures that are not real, and the verbose line names the machine.
the dispatched object could not be written, the depfile for a dispatched compile could not be written	The object came back and this machine could not store it — a full disk or a read-only output path. Local, not a fleet problem, and the compile is redone here.

Exit codes

Situation	Code
Cache hit	0
Miss, fall-back, or non-cacheable line	the real compiler's exit code, verbatim
Compiler could not be spawned	1
--help, --version, --show-stats	0
--zero-stats failed	1
No arguments at all, or an unknown option	2 (diagnostic and usage printed to stderr)

Fall-back reasons

Every reason that appears under fall-back reasons, and what to do about it:

Every one of them still produces a correct object. Which of them leave distribution running is a narrower claim than that, and worth reading precisely: the two that describe a daemon failing to serve the lookup — an unreachable one (fetch exchange failed) and one that answered and refused (rejected (…)) — carry on and dispatch, which is the note at the top of this page. The rest end the invocation at a local compile, because each of them is a reason there is nothing to dispatch with: the key is not computed yet (the configuration and path reasons), the preprocess itself failed, the translation unit is one the launcher deliberately steps over, or the object could not be written on this machine. The ones marked uncacheable are the launcher's own refusals and are not about the daemon at all.

One reason in the table is not about the daemon or about a fleet declining to help, and it is the only one worth interrupting somebody for: a worker answered about a different compile. It is printed on stderr whether or not PACE_VERBOSE is set, because it means a machine in the fleet produced an object for work nobody asked it to do. Every other reason here is quiet unless you ask.

Reason	Meaning
missing PACE_ADDR/SOURCE_DIR/BINARY_DIR	Configuration incomplete — the cache was never contacted, and neither was a scheduler. Distribution is off here deliberately, and not merely for want of a key: PACE_ADDR= (set but empty) is how a build opts out of the launcher altogether, and without the two roots there is no portable key for a scheduler to suppress duplicates on. Set all three to use either.
preprocess failed	The compiler rejected the preprocess probe; the line may use an unsupported option form.
uses __TIME__/__DATE__/__TIMESTAMP__	Deliberate: the TU is non-deterministic and would never hit. Reported as uncacheable, not as an error.
a command-line path is drive-relative under no root, a reported dependency path is drive-relative under no root	Deliberate, and Windows-only. A path like C:foo\bar.hpp resolves against drive C:'s own current directory, which no cache entry records — so the launcher can neither key it (a header moved inside it would not re-key) nor check it on replay (there is no directory to stat it against). Caching such a compile could serve a stale dependency record under a zero exit code, so it is not cached at all. Reported as uncacheable, not as an error. Spell the path absolutely (C:\foo\bar.hpp), make it relative, or bring it under PACE_SOURCE_DIR/PACE_BINARY_DIR. The first is the rule applied to the command line, the second to what the compiler reported; PACE_VERBOSE names the offending path itself.
daemon does not support authentication; the configured credential was ignored	PACE_TOKEN is set but the daemon predates the AUTH verb. Caching works normally — the daemon steps over the verb it does not know and serves the command — but this traffic is not authenticated. Said once per invocation rather than per exchange. Upgrade the daemon, or unset the token if it was not meant to apply here.
no machine ticket: this machine's node did not answer MINT-TICKET, ... refused MINT-TICKET, ... answered MINT-TICKET with none, ... PACE_ADDR names no endpoint ...	This machine's node could not mint the ticket an exchange with another machine presents, so that exchange carried none. Whether it was then refused is its audience's decision: a cache or scheduler that requires a ticket refuses it, and the reason is recorded in place of that refusal because it names the thing to fix. The node is asked over loopback at PACE_ADDR's port, whatever PACE_SCHEDULER names: start it, or point PACE_ADDR at the port it listens on. did not answer with it could not be reached while a node runs on this machine is first a node that is not listening on loopback at that port — one on another port, or bound only to a non-loopback address, prints the same line; check that before anything else. A node that IS listening on loopback at that port and still prints it did not complete the loopback connect within the 50 ms cap — a node saturated by this machine's own build is the likely case, and it is not yet measured (#1668). No setting raises that cap (PACE_CONNECT_TIMEOUT only lowers it). That invocation presents no ticket for the rest of its life, so a cache or scheduler that requires one refuses it and the compile runs locally; lowering the build's parallelism is the only lever until #1668 is decided, and it is an inferred one, not a measured one. The build succeeds either way; only the remote work is lost. PACE_VERBOSE=1 says it with the endpoint and the node's own words, at most twice per invocation: once on the first failure, and once more if a later failure means the node is not asked again, since that changes what every later exchange carries. A node that did not answer is asked once per invocation and not again, under a connect bound sized for loopback rather than for the network, so a machine with no node pays about that bound, once per compile (on Linux, which refuses a closed loopback port at once, next to nothing); a cache with authentication off serves an exchange without one.
rejected (unauthenticated): ...	The daemon requires a credential. authentication required means none was sent — set PACE_TOKEN. authentication failed means one was sent and was wrong. The two are deliberately different messages because they are different mistakes. Either way the build succeeds and only the caching is lost — the compile is still dispatched if PACE_SCHEDULER names a scheduler, and runs locally otherwise.
fetch exchange failed, fetch decoded malformed	Transport or protocol trouble mid-request. Also how a PACE_TIMEOUT expiry surfaces: a daemon that accepted the connection and then went quiet. If these appear in bulk and each compile stalls for the full timeout first, suspect a wedged daemon rather than a flaky network. fetch exchange failed is also what a plainly wrong PACE_ADDR looks like — every compile, at once, with the fleet still doing the work. The two differ in what happens next: fetch exchange failed carries on and dispatches, while fetch decoded malformed — a daemon that answered with a value this launcher cannot read, which in practice means a mixed install — ends the invocation at a local compile.
fetch decoded another generation's value	The daemon served a value that IS a compile value, well formed, written under a canonicalization generation this build does not implement. Ordinary during a rolling upgrade, and not a damaged cache — which is the whole reason it is not fetch decoded malformed. Either direction: a producer behind this launcher reads exactly like one ahead of it. The compile happens locally, nothing is stored, and it clears once every server and launcher in the fleet are on one generation. The wire code's side of it is foreign-value-generation.
rejected (unsupported-version): …	The daemon answered and declined: it does not speak this launcher's wire version. The two binaries ship together, so this means a mixed install — an old daemon still running against a new pace-cc, or vice versa. The message names the range the daemon does support. Restart the daemon from the same package as the launcher.
rejected (payload-too-large): …	The object exceeded the daemon's --storage-max-value. Raise it, or accept that this TU will not cache.
rejected (…) (other codes)	The daemon refused the command and said why; see the error-code table.
could not write object on hit	The object output path was not writable.
the cached object names its <parts>; this compile's differs, or its copy was evicted; compiled this one's own	Not a fault, and still a miss. The object stored under this key names the checkout that compiled it -- a __FILE__ or a std::source_location -- so it is not shared, and this compile built and stored its own copy, which the next compile here hits. <parts> is what the object names, from source-root, build-tree and working-directory; an object naming only the build tree or the working directory also misses from a second build directory of the SAME checkout. The reason cannot say which part differed -- this end never sees the producer's -- and neither does the MISS trace line under PACE_VERBOSE, which carries the same text. A translation unit that shows up here from every checkout is one whose source keeps it from ever being shared across them; spelling its sources relatively AND compiling without /FC (which MSBuild turns on by default as "Use Full Paths", and which makes cl write the absolute path whatever the command line said), or not baking the file name into program data, is what changes that.
a worker answered about a different compile	A defect somewhere in the fleet, not a fleet declining to help. The worker's reply did not belong to the request that asked for it — see correlation. The object is refused unread and the translation unit is compiled locally, so the build is correct and the caching of it is unaffected (the outcome is still a miss). This is the one reason printed unconditionally rather than only under PACE_VERBOSE, and the line names the worker, the correlation this client expected and the one that arrived. Accepting such a reply would store a wrong object under a correct key and serve it to every other machine that fetches it, so there is no configuration that relaxes this. If it appears at all, find the machine the line names.
a reported dependency path is not text this host can read, a captured region names a path that is not text this host can read	Deliberate, and Windows-only. cl.exe writes the paths in /showIncludes in the console output code page, while this launcher's own roots arrive as UTF-8 -- so a header under a non-ASCII directory can reach it as bytes it cannot read as text. Such a path prefix-matches no root, which would key a project header as toolchain content and serve a stale object under a zero exit code, so the compile is not cached at all. Reported as uncacheable, not as an error. The fix is the console: chcp 65001 makes cl emit UTF-8 and this stops appearing.
an input changed during the compile	Not a fault. The file (named on the PACE_VERBOSE line) was saved — by an editor, a git checkout, a generator — while the compiler was running, so the object may describe either version of it and was not stored. In direct mode the manifest, built from the content the key was taken over, is still recorded unless recording it was declined; the PACE_VERBOSE line ends manifest recorded from the keyed content or no manifest recorded to say which. Recorded as INPUTS-MOVED. Ordinarily nothing to do: the next build compiles and caches as usual. If it recurs for files nobody edited — which here is a path named only by the build's own depfile, judged by the clock alone — see the row below for what to check.
an input changed while the key was taken	Not a fault. The file (named on the PACE_VERBOSE line) changed while the launcher was reading it for the key — between the start of the invocation and the snapshot taken right after the key's preprocess — so neither the object nor the manifest was stored. Recorded as INPUTS-MOVED. Ordinarily nothing to do: the next build caches as usual. If it recurs for files nobody edited, the clock is the suspect rather than an editor: a share whose clock runs AHEAD of this machine's (SMB, NFS, or /mnt/<drive> under a WSL clock that lags the Windows host) stamps a fresh file at or after the moment the launcher fenced, and so does a filesystem with whole-second (or FAT's 2-second) stamps for a file written in that same second. Then every build reports it again, generated headers on such a share included — check the share's clock against this machine's, or move the build tree to a filesystem with fine-grained stamps. The object is never wrong there; only its caching is lost.
an input could not be observed	The file (named on the PACE_VERBOSE line) could not be read or identified when the key was taken, so the launcher could not tell whether it changed during the compile and stored nothing. Recorded as uncacheable, because unlike a race it usually persists. Check the file's permissions, or for a tool that replaces it in place while builds run.

Verifying that a hit is the right object

A wrong object served under a correct key is this cache's worst failure: it links, it usually runs, and nothing says anything. PACE_VERIFY=<n> makes one hit in every n prove itself — the translation unit is compiled again and the two objects are compared. On a mismatch the freshly compiled object is the one left on disk, so a build that has just caught its cache is still a correct build, and the key is named on stderr whether or not PACE_VERBOSE is set.

It costs a whole compile per verified hit, so it is off by default.

What "the same object" means is not "the same bytes", and on Windows it cannot be. Every MSVC-family driver stamps the wall clock into the COFF header, and a cached object was compiled earlier than the fresh one it is checked against by construction. Measured on MSVC 14.51 and clang-cl, two compiles of one translation unit to one object path differ in that 4-byte TimeDateStamp and in nothing else, whether two seconds or five minutes apart, with /Z7 or without. That one field is therefore normalised, and nothing else is. On Linux nothing is normalised at all: clang 20.1 and GCC 14.2 objects are byte-identical between two compiles, -g included, so ELF keeps a strict byte comparison.

What that deliberately does not overlook is cl's record of where it compiled — the object's absolute path in .debug$S, and the source hash in .chks64. Those vary with the path rather than with the clock, and verification recompiles to the same path, so a hit produced on this machine differs in neither. A hit that does differ there was built somewhere else — another checkout, or another machine — and is reported, with the message saying so rather than leaving "wrong object" to mean either that or stale code.

Four outcomes reach you, and only one of them is a finding. The last two are different from each other in the way that decides what to do next — one is worth retrying, the other never will be:

Outcome	Meaning
(silence)	The cached object is the object this compiler produces.
WRONG OBJECT served for key …	It is not. The message names what differed — a section such as .text$mn is stale code; .debug$S or .chks64 is a foreign build path. The fresh object was used, so this build is unaffected. Find the machine that stored it. The compile is recorded as VERIFY-MISMATCH rather than HIT — in invocations.log, on the PACE_VERBOSE trace line, and as wrong object in --show-stats, which rates the hit rate against it — because the build did not use what the cache served.
could not verify the hit for key …	The check did not complete: the fresh compile failed, or a file could not be read. Nothing is known about the cached object either way, and the next hit may well answer.
cannot verify hits for this toolchain …	This build cannot lay out the object format its own compiler produced, so it can say nothing about any hit — not this one and not the next. A property of the toolchain, not a statement about your cache.

Debug paths in a replayed object

A cache hit replays an object that was built somewhere else, and a compiler with debug information on records where it was built — DWARF's DW_AT_comp_dir is the compile's working directory. That directory is on no command line, so nothing in the cache key can distinguish two checkouts by it, and a debugger opening the replayed object then looks for sources in a tree this machine may not have. Nothing fails; the paths are simply somebody else's.

On GCC and Clang the build does something about it for you. When cmake/portable/CompileCache.cmake enables a launcher it also appends

-fdebug-prefix-map=<source tree>=<relative path from the build tree>
-fdebug-prefix-map=<build tree>=.

which makes the objects byte-identical across checkouts and DW_AT_comp_dir exactly . — both measured, on both drivers. The build tree is mapped last because GCC and Clang honour the last matching rule, and the build tree usually lives inside the source tree; the other order leaves comp_dir naming the build directory, which is still checkout-independent and therefore invisible to an object comparison, while two build trees at one depth under different names then share a key. The source rule is emitted only when the relative path back is a pure ../ chain; for an out-of-tree build it would otherwise carry the checkout's own path, and the configure says so rather than mapping to something that only looks portable:

-- [cache] The source root is NOT mapped: the build tree lies outside it, so the
   relative path back would carry the checkout's own path

Passing your own -fdebug-prefix-map is fine and is keyed correctly: the launcher relativizes the root the flag names, so two checkouts still share, and leaves the replacement literal, so two machines mapping to different replacements miss rather than exchange objects that disagree. The mapping has to be the same everywhere a cache is shared, and that is enforced by the key rather than left to a convention.

-ffile-prefix-map and -fmacro-prefix-map are recognised too, and they are what a project passes so its objects stop naming its checkout (Catch2 passes -ffile-prefix-map=<checkout>/= itself). The key relativizes the root they name and leaves the replacement literal, exactly as above, with two differences that come from their rewriting __FILE__ in the preprocessed text:

• they stay on the preprocess line, so the key hashes the __FILE__ the real compile produces;
• on a dispatched compile, -fmacro-prefix-map travels to the worker verbatim: the worker applies the rule to this machine's paths in the text it is handed and opens nothing, so its object is the one a local compile would have written. A line carrying -ffile-prefix-map compiles locally, because that flag also rewrites debug records, which a worker maps by rules of its own.

The key also records where the build tree sits under the source root whenever one is present, since the mapping rewrites a build-tree path through the source root's rule; two checkouts share when their build trees sit in the same place.

A root containing a space is not mapped at all: these rules are spliced into CMAKE_<LANG>_FLAGS, which is space-separated, so one rule would arrive at the driver as two arguments and every compile would fail. The configure says so.

A dispatched compile records the same compilation directory a local one does. The flag itself is never forwarded — it is a path-valued argument, and RemoteCompileArgs drops every one of those, correctly: a worker needs a rule whose left-hand side is a path on the worker, which your machine has never seen. What travels instead is your own compile directory and what your mapping spells it as, and the worker builds the rules. So a fleet-built object and a locally built one both record ., and a debugger resolves sources the same way whichever produced the object in your cache.

The worker maps two directories, both to your replacement, because which one a dispatched object records is a fact about the driver rather than about the fleet. Measured, reading DW_AT_comp_dir:

the preprocess line	what the worker's object records
g++ -E	the worker's directory
g++ -E -g	your directory
clang++ -E, clang++ -E -g	the worker's directory

gcc's -fworking-directory is implicit under -g: it puts a line marker naming the preprocessing directory into the text, and the worker's compile adopts it. clang emits no such marker. Mapping both candidates gives one answer either way.

Consequences worth knowing:

• A build reached through a symlink is mapped correctly, and that takes care. Both drivers record — and match -fdebug-prefix-map against — $PWD when it is absolute and names the same directory as ., falling back to getcwd(3) otherwise. So if you cd to a build through a link, your rule spells the link and the driver agrees, while the resolved path matches nothing. The launcher predicts from the same value your compiler will use, so the dispatched object and the local one still agree. It is worth stating because getting it wrong is invisible: the mapping simply never travels, the fleet-built object keeps an absolute path, and nothing reports a problem. macOS is where this shows up first — /var is a symlink, so anything under $TMPDIR is.
• If your build passes no mapping, a worker adds none. An object built on the fleet then records the worker's directory, exactly as it did before — which is the honest answer, because there is no directory your build would rather see.
• A worker that cannot honour the mapping refuses the job rather than returning an object that disagrees, and your compile runs locally. That happens when either directory contains an = (gcc and clang split <from>=<to> at opposite ends, so no unambiguous rule exists) or when the worker's driver has no path-map switch at all. It shows up as a refusal on the worker's metrics and costs one local compile.
• A worker running in / maps only your directory, not its own. A prefix-map rule appends the unmatched tail, so a rule whose left-hand side is / would rewrite every absolute path in the object — /usr/include/... becomes .usr/include/.... The shipped pace-node.service sets no WorkingDirectory=, so that is the ordinary Linux deployment. Your own rule still lands, which covers gcc completely; under clang such a node's objects keep its working directory, exactly as they did before. Giving the unit a WorkingDirectory= of its own closes that too.

The #line markers a worker is sent still carry the dispatching machine's paths, so a dispatched object's line table names the producing checkout's headers. That half is #506's remainder rather than its subject: on gcc the file names come from those markers and already match a local compile's, and on clang the compilation unit's own name is the worker's scratch file (#660).

On Windows there is no equivalent and the paths stay. cl has no path-map switch, and -ffile-prefix-map does not reach the records that matter for clang-cl — CodeView's S_OBJNAME and the embedded -cc1 line are not remapped, measured. So a replayed object on Windows names the checkout that produced it in its debug records. That is an accepted cost of cross-checkout sharing, which on that platform is most of the value: closing it means keying on the compile's location, and then no two checkouts ever share an entry. It affects debug information only — .text, .data, .xdata, .pdata, the relocations and the symbol table are byte-identical.

Two things this is not:

• It is not __FILE__. The preprocessor expands __FILE__ and the key hashes preprocessed output raw, so a translation unit whose __FILE__ differs between checkouts also keys differently and the two never share an entry. There is no arrangement in which a program reports a source location from another checkout.
• It is not coverage. Coverage mapping has the same defect and is guarded outright: configuring a coverage build with a launcher imposed is a hard error, not a warning.

Known limitations

• __TIME__ / __DATE__ / __TIMESTAMP__ detection scans the source file text. Direct use is caught; use reached only through a header is not, so such a TU stays a permanent miss. Never incorrect — just never cached.
• A Windows drive-relative path (C:foo) reaching a compile takes that translation unit out of the cache entirely, rather than being ignored. Only clang-cl echoes such a path back unresolved (cl resolves it through the filesystem), and no common generator emits one, so in practice this costs nothing — but where it does fire it costs the whole TU rather than one path, deliberately: a partial answer here is the stale-dependency serve it exists to prevent. The refusal is silent unless PACE_VERBOSE is set, and shows in --show-stats as uncacheable.
• Diagnostics-stream paths outside the include grammar are not yet localized.
• Toolchain and system dependency paths are in neither the key nor the existence check, by design: they are the producing machine's spelling, the compiler identity in the key already covers them, and including them would make two machines with different system include prefixes share nothing. So a cache shared between machines whose compilers print the same version banner and target the same triple, from different prefixes, can still replay a dependency record naming a path the consumer lacks. Project headers — the ones that actually move — are covered by the key, so this is now confined to the toolchain.
• The cache key normalization is deliberately young (objkey-v7). Tune it against real developer↔CI hit rates before relying on it broadly. Bumping the schema re-keys the cache: existing entries miss once and are rewritten.
• Localized path separators may be normalized to / in some segments. Ninja matches dependencies separator-insensitively, so this is cosmetic.
• /showIncludes note paths are emitted with their .. segments collapsed, because Ninja refuses a note path longer than _MAX_PATH (260) by length, before it canonicalizes one — a fixed-buffer check no long-path policy reaches. A note path that is over the limit with nothing to collapse is still refused, and no lexical rule can change that; the remedy there is a shorter build root. The bytes stored in the cache keep the driver's own spelling.
• Non-C/C++ inputs (for example Windows .rc resource files) are correctly classified as non-cacheable and passed through.
• PACE_TIMEOUT bounds one exchange, not a whole invocation. Direct mode makes a separate manifest round-trip before the object fetch, so a compile against a daemon that accepts and then goes silent can wait up to twice the timeout before falling back.
• A dispatched compile exceeding PACE_DISPATCH_TIMEOUT is abandoned by the client, which hands the lease back and compiles locally — so the build still succeeds and the key is not pinned for the scheduler's lease timeout. The worker notices the client has gone and skips writing the object back, so the transfer is not paid for; the compile itself still runs to completion, so that CPU is spent twice.
• That deadline is no longer how long a dead worker goes unnoticed. The compile exchange dials with TCP keepalive armed, so a worker whose machine has vanished — powered off, unplugged, suspended, or cut off — is noticed in about 16 seconds on Linux and macOS and about 30 seconds on Windows, whatever the deadline is set to. Raising PACE_DISPATCH_TIMEOUT for a slow translation unit therefore does not make hard-failure detection slower.
• Nor is it how long a WEDGED worker goes unnoticed. Keepalive cannot see a machine whose kernel answers every probe while the process above it makes no progress, so the worker says so itself: a running compile writes a Status::Progress frame every five seconds, carrying nothing, and PACE_DISPATCH_IDLE bounds the silence between them at thirty seconds (#245). The launcher's fall-back line reports it as stopped reporting progress, which is a different remedy from ran out of budget.
• Those pulses are why the 0xFC protocol version is 3, and why version 3 is also the oldest a server accepts: a reply carries a status byte and no kind, so a client built before this cannot step over a frame it does not know the way it can step over an unknown request verb. An older launcher meeting a pulse would abandon the compile minutes in, as a transport failure naming nothing; refused at the request instead, it is told unsupported-version and which range would have worked.

Measured behaviour

Validated against a large real C++ codebase by replaying its compile_commands.json (4223 translation units) through the launcher twice — once to populate, once to measure. On a 400-TU sample the second pass reached a 99.7% hit rate (393 of 394 cacheable), with the classification of the non-cacheable remainder identical across both passes: stable and correct.