fastcache-cc¶

A compiler launcher in the style of ccache and sccache, backed by a fastcached daemon over the compile-cache protocol.

It fronts every compile: on a hit it writes the object file and replays the compiler's captured output; on a miss it has a worker compile it when FASTCACHE_SCHEDULER names one, otherwise runs the real compiler, and stores the canonicalized result. Caching is an optimization — an unreachable or broken cache slows a build down, it never breaks one.

A cache failure does not switch distribution off. A cache and a compile fleet are two services, usually on two machines, so a daemon that refused this launcher or could not be reached says nothing about whether a worker can build the translation unit: the compile is still dispatched, and only the caching of it is lost. The reason is still reported and still counted under unavailable, and nothing further is offered to that daemon for the rest of the invocation — there is no point spending an object-sized transfer to be refused twice.

What a build against a broken cache does pay is the dispatch itself: a translation unit that would have been a hit is now preprocessed a second time -- dispatch sends #line markers, which the cache key deliberately suppresses -- and sent to a worker. That is the trade the fleet exists to make and it is still the right one, but it is not free, and it is the reason a wrong FASTCACHE_ADDR is worth fixing rather than living with.

This page is the reference: every flag, every environment variable, every fall-back reason by name. For the story of what happens between your build system and your compiler — direct mode, the key, the lookup, the dispatch — read How it works.

Why not just ccache or sccache?¶

Cache entries are portable across checkout paths. Before the key is computed, every path under the configured source root and build tree is rewritten to a token; on a hit, the stored output is rewritten back to the consuming machine's layout. Two machines with the same content at different checkout paths produce the same key and share entries.

That matters because a CI runner's checkout is rarely at the same path as a developer's. With a path-sensitive cache the two populations never share hits; here they do. It also keeps the replayed dependency information valid: header paths in /showIncludes output and in -MF depfiles are localized on the way out, so the build tool records dependencies that exist on this machine.

Requirements¶

A running fastcached daemon, reachable over TCP.
A daemon value cap above your largest object file. The 256 MiB default covers the usual case — a large C++ codebase was measured at ~122 MB for its biggest object — so this normally needs no flag. Past that, --storage-max-value raises both the per-value cap and the wire frame-payload cap:

fastcached --storage-max-value=512M

Supported compilers¶

Driver	Recognised as	Object flag	Dependencies
`gcc`, `g++`, `cc`, `c++`	GNU	`-o`	`-MD -MF <path>` depfile
`clang`, `clang++`	GNU	`-o`	`-MD -MF <path>` depfile
`cl`	MSVC	`/Fo`	`/showIncludes` on stderr — unverified, #825
`clang-cl`	MSVC	`/Fo`	`/showIncludes` on stdout

The stream column describes a compile run, which is what the replay path reproduces. It does not describe the key probe, and nothing keys off it: a preprocess-only run moves the notes (clang-cl puts them on stderr under /EP, so that they do not corrupt the preprocessed stdout), so the probe splits stdout unconditionally and reads notes from both streams rather than guessing.

Version-suffixed (g++-14, clang-18) and .exe forms are recognised. Any other argv[0] is treated as unknown and passed straight through uncached.

A command line is cacheable only when it compiles exactly one translation unit to an object file. Link steps, compile-and-link steps, preprocess-only runs (-E, /EP), and multi-source lines all fall back to the real tool.

Usage¶

fastcache-cc <compiler> <args...>                       Front a compile (as CMAKE_<LANG>_COMPILER_LAUNCHER)
fastcache-cc --show-stats | -s [--prefetch-group <id>]  Report cache statistics for this machine
fastcache-cc --html-stats [--prefetch-group <id>]
                          [--out <path>]                Render those statistics as a self-contained HTML dashboard
fastcache-cc --zero-stats | -z                          Discard the statistics log
fastcache-cc --print-toolchain-fingerprint <compiler>   Print the fingerprint a dispatched compile would use
fastcache-cc --help | -h | /?                           Flags and environment reference
fastcache-cc --version                                  Launcher version

--print-toolchain-fingerprint is the diagnostic for a fleet whose scheduler answers no-worker: run it on a client and compare with what the worker logs as serving. It recomputes rather than reading the cache, so it also repairs a stale entry on its way past.

--help is generated from the same table the launcher dispatches on, so it always lists exactly what the binary accepts.

Renamed flags. The statistics flags now use sccache's names: --stats is --show-stats and --clear-stats is --zero-stats. --reset is gone with no replacement — it read as if it reset the cache itself, when it only discarded this machine's statistics log. All three old spellings are now rejected with an "unknown option" diagnostic and exit 2, rather than being silently treated as a compiler to run.

Wire it into CMake:

cmake -S . -B build -G Ninja \
  -DCMAKE_C_COMPILER_LAUNCHER=fastcache-cc \
  -DCMAKE_CXX_COMPILER_LAUNCHER=fastcache-cc

The fastcached build does this for itself: cmake/portable/CompileCache.cmake picks fastcache-cc up automatically whenever the binary is on PATH and a daemon answers at 127.0.0.1:6674 — at any other daemon, local or remote, when FASTCACHE_ADDR is exported, at -DFASTCACHE_ADDR=host:port ahead of even that, nowhere if it is set empty — and injects FASTCACHE_SOURCE_DIR / FASTCACHE_BINARY_DIR from the source and binary directories, so those two need not be exported.

Exporting FASTCACHE_ADDR retargets an existing build tree on its next configure, rather than being frozen at whatever the first configure saw, which is what ordinary cache semantics would do to it. A -DFASTCACHE_ADDR= passed on the current run still wins over the environment — including the empty value that opts out — since it is the more deliberate of the two.

"Answers" is checked, not assumed: configure compiles one tiny translation unit through the launcher with FASTCACHE_VERBOSE=1 and accepts only a reported HIT/MISS, since a launcher whose daemon is down still compiles fine and would otherwise leave every TU paying a failed connect with nothing to show for it. That costs about 0.1 s against a daemon that answers, runs on every configure so that starting the daemon and reconfiguring is enough, and any other outcome — connect failed, a version mismatch, no daemon at all — falls through to ccache naming the address it tried; -DUSE_COMPILER_CACHE=OFF disables both.

A version mismatch is announced, not merely reported. It is the one rejection reason that says a daemon is answering and is one you installed — so it is a CMake Warning rather than a status line, and it says which end is behind: unsupported wire version 3; this server speaks 1..1 means this launcher speaks 3 and the daemon accepts 1 at most, so the daemon is the older of the two. Every other reason (not installed, no answer, an uncacheable probe) describes a cache nobody set up and stays at STATUS, because a warning that fires for all of them is one everybody learns to skip. Issue #815 is what that warning is for: a packaged daemon a release behind a hand-built launcher, refusing every exchange, with the build silently going through a different cache for weeks.

sccache is never selected automatically. It stays supported and keeps its place ahead of ccache, but only when a build asks for it with -DALLOW_SCCACHE_FALLBACK=ON; without that flag the module says Not using sccache and moves on. Being the unasked-for answer to fastcache-cc not working is exactly what made #815 invisible. (Pointing sccache at a fastcached daemon as its storage backend is a different thing entirely and is unaffected — see below.) Where it is asked for, the fallback is still not equivalent, and the configure says so: selecting sccache under MSVC or clang-cl emits a CMake Warning naming the hazard below, because a build opted into it in silence is one whose symptom arrives hours later and somewhere else. -DUSE_COMPILER_CACHE=OFF opts out of caching entirely, and ctest -R compile-cache-caveat pins the warning, the silence for launchers that have no hazard, and both directions of the sccache gate.

On MSVC and clang-cl, one sccache cache must not be shared between checkouts

sccache preprocesses MSVC and clang-cl with /EP, which emits no line markers, so the text it hashes to find a cache hit carries no paths at all — while the /showIncludes stream it replays on that hit carries the absolute paths of the checkout that stored it. Two checkouts sharing one cache therefore record dependencies pointing into each other, after which editing a header in the checkout you are building rebuilds nothing: the build stays green and the objects are stale.

Measured on this project — a second worktree recorded 1097 dependency edges into the first and none into itself — and reproducible in two compiles: with sccache 0.14.0 and MSVC 14.51, a second directory took a cache hit from the first, and its /showIncludes named the first directory's header.

What is exposed is narrower than "sharing a cache". It is an incremental build across checkouts at different absolute paths. A clean build has no dependency graph to corrupt, and checkouts that all sit at the same path replay paths that are correct — a CI fleet is normally both, and this is not a reason to take the cache away from one. GCC and Clang are not exposed at all: their preprocessed output carries the paths, so two checkouts never share an entry to begin with — the same two compiles under g++ 14 were 0 hits and 2 misses.

A fastcached-backed sccache is definitionally one cache shared by every checkout and every machine pointed at it, so the developer machines behind it are exposed even where CI is not. On MSVC and clang-cl, use fastcache-cc instead: it rewrites a hit's paths into the consuming checkout before replaying them, and refuses a hit whose replayed dependency is not there — which is why it does not have this failure mode, and why its entries are portable across checkout paths on purpose.

The probe carries its own ten-second cap, which is what bounds a remote address that drops packets rather than refusing them: FASTCACHE_TIMEOUT bounds the exchange, not the TCP connect(), so a firewalled or vanished host otherwise takes the kernel's connect timeout to fail (measured at 2m30s on macOS) — once per configure here, but once per translation unit in a build.

Installing it automatically¶

All of the above assumes fastcache-cc is already on PATH, which on a new repository or a fresh machine is a manual step someone has to remember. -DFASTCACHE_AUTO_INSTALL=ON removes it: when no launcher the build could actually use is installed — no fastcache-cc, no ccache, and no sccache that -DALLOW_SCCACHE_FALLBACK=ON has opted into — the module downloads a prebuilt fastcache-cc for the host's OS and architecture from the latest stable release, checks the SHA-256 the release publishes, confirms the binary runs here, and uses it.

It is off by default, because reaching out to the network during cmake is a different thing from using what is installed. It fires only when nothing else is available: a launcher you installed is a decision already made. And it cannot fail a configure — an unreachable network, an unpublished platform, a download that arrives corrupt each end in one status line and the same fall-through to ccache (or sccache, where it was asked for) or plain compilation that a missing launcher has always produced.

The binary is staged per user, under version and platform, so every repository and build tree on the machine shares one copy and later configures neither download nor ask. The release lookup is cached for a day, and honours GITHUB_TOKEN / GH_TOKEN where the unauthenticated limit of 60 requests an hour per address would otherwise be shared out among CI runners. Pinning FASTCACHE_AUTO_INSTALL_VERSION skips the lookup altogether, and pointing FASTCACHE_AUTO_INSTALL_DOWNLOAD_BASE at a mirror installs without reaching GitHub at all.

Auto-installing the launcher alone still leaves a genuinely clean machine uncached, since fastcache-cc has nothing to talk to: -DFASTCACHE_AUTO_START=ON additionally stages and starts a fastcached daemon in the background — from the same release archive, persistently, off by default and independently of FASTCACHE_AUTO_INSTALL — when nothing answers at FASTCACHE_ADDR. cmake/portable/README.md covers it in full, including why it defaults off in CI as well as on a developer machine that has not opted in.

cmake/portable/README.md documents the full option set, and is written for projects vendoring the module rather than building this one.

Environment¶

Configuration is entirely environmental, so a launcher invocation stays a drop-in prefix. An empty value counts as unset.

fastcache-cc --help documents the same set, generated from the table in LauncherCli.cpp that is the launcher's single source of truth for these names. This page is the prose version; if the two ever disagree, --help is right.

Variable	Meaning	Default
`FASTCACHE_ADDR`	`host:port` of the cache — a `fastcached`, or a `fastcache-compile-node`'s `--listen-node`. Hostnames, IPv4 literals, and bracketed IPv6 (`[::1]:6674`) all resolve. Set but empty is the opt-out and means no caching -- but see the PowerShell note below, where that spelling does not reach the launcher.	`127.0.0.1:6674`
`FASTCACHE_SOURCE_DIR`	Checkout source root, used for keying and path canonicalization.	unset — no caching
`FASTCACHE_BINARY_DIR`	Build output root.	unset — no caching
`FASTCACHE_PREFETCH_GROUP`	Prefetch grouping id. Not part of the cache key, so it never partitions the cache.	`default`
`FASTCACHE_VERBOSE`	Print `HIT`/`MISS` and fall-back diagnostics to stderr.	unset (quiet)
`FASTCACHE_NO_STATS`	Do not record invocations to the statistics log.	unset (recording on)
`FASTCACHE_NO_DIRECT`	Disable direct mode, always preprocessing to derive the key.	unset (direct on)
`FASTCACHE_CONNECT_TIMEOUT`	Deadline for opening a connection — name resolution included. Like the three below, a duration: a whole number and one of `ms`, `s`, `min`, `h`, `d`; a bare number is not one and is ignored. `0s` leaves the platform's own, which runs to minutes. Short on purpose: a cache that has not accepted within a second is one the build is better off without, and a wedged resolver would otherwise stall every translation unit.	`1s`
`FASTCACHE_TIMEOUT`	Deadline for one whole exchange with the daemon — or with a scheduler's `LEASE`/`RELEASE` — from the request to the last byte of the reply. `0s` removes the bound. A daemon that accepts and then stalls, or dribbles one byte at a time, would otherwise block the compile forever. Bounds one exchange, not the whole invocation — see below — and not a remote compile.	`10s`
`FASTCACHE_DISPATCH_TIMEOUT`	Deadline for one whole `COMPILE` exchange with a worker. `0s` removes the bound. Far larger than `FASTCACHE_TIMEOUT` because it bounds a different shape of conversation: a worker writes nothing until the compiler has finished, so the client waits out the entire remote compile in one read. Ten minutes because that is the scheduler's own lease timeout — waiting longer means waiting on a lease it has already reclaimed. See Distributed compilation.	`10min`
`FASTCACHE_DISPATCH_IDLE`	Deadline on silence during a `COMPILE` exchange. `0s` removes the bound. A worker writes a five-byte progress frame every few seconds while it is compiling, so this bounds how long it may say nothing rather than how long the compile may take — which is what lets it be seconds while `FASTCACHE_DISPATCH_TIMEOUT` stays minutes. It is the only thing that sees a worker whose machine answers every keepalive probe while the process makes no progress. On expiry the launcher compiles locally and hands the lease back, and its fall-back line (with `FASTCACHE_VERBOSE`) reads stopped reporting progress rather than ran out of budget. See Distributed compilation.	`30s`
`FASTCACHE_MAX_STORE_BYTES`	Largest compiled result the launcher will offer to the daemon; `0` means no limit. A bigger result is simply left uncached. Matches the daemon's `--storage-max-value` default by construction rather than by negotiation — there is no handshake, so raise both or the other keeps refusing.	`268435456` (256 MiB)
`FASTCACHE_SCHEDULER`	`host:port` of a fleet scheduler — the `--listen-node` port of some `fastcache-compile-node` running `--serve-scheduler`. On a miss the launcher asks it for a worker and sends that worker the preprocessed translation unit. Every refusal falls back to a local compile, with one exception: `not-leader` is an instruction rather than an answer about the fleet, so the launcher retries against the endpoint the refusal names (up to two hops, then it compiles locally). This value therefore only has to be a member of the cluster, not the current leader — no launcher needs re-pointing after an election. The workers do the same with their own `--scheduler`: a node follows `not-leader` when it registers and heartbeats, and remembers where the leader answered, so an election re-points the whole fleet rather than just the clients. Both halves are needed — a launcher that followed the redirect while the workers did not would reach a leader whose registry they had all expired out of, and every lease would answer `no-worker`. A cache that is unreachable or refuses counts as a miss for this purpose — it does not disable dispatch. See Distributed compilation.	unset — every miss compiles locally
`FASTCACHE_TOKEN`	Shared secret presented to a daemon started with `--requirepass`. Costs no round trip — it is pipelined ahead of the real command, not awaited. Safe against a daemon that requires none: such a daemon accepts it and ignores it. Not safe with `FASTCACHE_SCHEDULER` — a compile node serves no `AUTH` verb, so the credential is refused and dispatch stops working entirely (#198).	unset — no credential sent
`FASTCACHE_USER`	Username to accompany `FASTCACHE_TOKEN`. Unset (the usual case) authenticates against the secret alone, which is what `--requirepass` configures. Ignored without a token — a username on its own is a misconfiguration, not a request to authenticate, and sending an empty secret would be refused by every server that wants one.	unset
`FASTCACHE_VERIFY`	Verify one hit in every N by compiling the translation unit again and comparing the objects — see Verifying that a hit is the right object. Costs a whole compile per verified hit, so it is for CI, a nightly, or reproducing a report. `1` checks every hit. Which hits are sampled is decided by hashing the key rather than by chance, so the rate holds over a build and a translation unit that verified verifies again. A value that is not a whole number reads as off rather than as an error: this is a diagnostic set by hand, and refusing to compile over a typo in it would break the build it was brought in to investigate.	unset (off)
`FASTCACHE_MSVC_DEPS_PREFIX`	The prefix a dispatched compile's synthesised `/showIncludes` notes carry — that is, this build's `msvc_deps_prefix`. Only dispatch needs it: a worker compiles preprocessed text and reports no dependencies, so the launcher writes the record itself, while a local compile emits the compiler's own notes and needs nothing here. Ninja matches the prefix literally and knows nothing about languages, so an English note against a localized `msvc_deps_prefix` records no dependencies at all for that translation unit and the next header edit does not rebuild it — see A localized MSVC toolchain.	unset — the English `Note: including file:`. Set it to `auto` to have the launcher ASK the compiler instead; see A localized MSVC toolchain for why that is opt-in

The statistics log is located from the usual per-user state variables rather than one of the launcher's own. These are read but never written:

Variable	Meaning	Default
`LOCALAPPDATA`	(Windows) Base for the log directory, `%LOCALAPPDATA%\fastcache-cc`.	unset — no statistics recorded
`XDG_STATE_HOME`	(POSIX) Base for the log directory, `$XDG_STATE_HOME/fastcache-cc`. Preferred over `HOME`.	unset — fall back to `HOME`
`HOME`	(POSIX) Base for `$HOME/.local/state/fastcache-cc`, used when `XDG_STATE_HOME` is unset.	unset — no statistics recorded

With no usable state directory there is nowhere to append to, so statistics are silently disabled. Caching itself is unaffected.

ADDR, SOURCE_DIR and BINARY_DIR must all be non-empty to cache. ADDR has a default and the other two do not, so in practice it is the roots that are missing when nothing caches — the build still succeeds, which is exactly why this is worth checking before concluding the cache does not help. With FASTCACHE_VERBOSE set, that case reports missing FASTCACHE_ADDR/SOURCE_DIR/BINARY_DIR.

The empty opt-out does not work from PowerShell¶

export FASTCACHE_ADDR= is a genuine set-but-empty entry on a POSIX shell, and the launcher reads it as the opt-out. PowerShell cannot express it. $env:FASTCACHE_ADDR = "" leaves the name present in PowerShell's own view -- Test-Path Env:FASTCACHE_ADDR answers True and $env:FASTCACHE_ADDR prints empty, exactly as an operator expects -- while the child process never receives the variable at all. fastcache-cc therefore sees it unset, falls back to the 127.0.0.1:6674 default, and keeps caching. Nothing reports this: the opt-out silently does not happen, and the one place anybody would check agrees that it did.

[Environment]::SetEnvironmentVariable('FASTCACHE_ADDR', '', 'Process') behaves the same way, so it is not a spelling problem with a workaround. From PowerShell, opt out at configure time instead -- -DFASTCACHE_ADDR= (the deliberate form, which wins over the environment) or -DUSE_COMPILER_CACHE=OFF -- both of which travel as command-line arguments rather than through an environment block.

The CMake integration honours the same opt-out, and until #372 it did not. cmake/portable/CompileCache.cmake decides whether the launcher is used at all, and it folded a set-but-empty FASTCACHE_ADDR into the default -- so on a POSIX shell, where the spelling above does travel, export FASTCACHE_ADDR= reached the launcher as an opt-out and reached the build integration as "say nothing", and the build went on being fronted. The two readings of one variable now agree: absent means the default, present-and-empty means no caching, in both. Note this is why the paragraph above still sends a PowerShell user to -D: that platform's problem is that the variable never reaches the child at all, which no predicate on the receiving side can repair.

FASTCACHE_ADDR defaults to 127.0.0.1:6674 rather than to nothing, so the launcher caches with no configuration at all against whichever of fastcached or fastcache-compile-node is running on this machine — the node's --listen-node defaults to the same address for exactly that reason. A remote default would be indefensible, since every translation unit on a machine with nothing listening would pay a connect timeout in silence; a closed loopback port refuses immediately, so a machine running neither pays microseconds per compile.

A localized MSVC toolchain¶

A Visual Studio carrying a language pack prints /showIncludes notes in that language, and CMake records whatever it printed as Ninja's msvc_deps_prefix. Ninja then matches that literal string, so the prefix a note carries has to equal it byte for byte.

That matters here on one path only: a dispatched compile. A worker compiles preprocessed text, which has no #include left in it, so the worker's compiler reports no dependencies and the launcher writes the record itself from what its own probe saw. Until #700 it wrote the English Note: including file: unconditionally. On a localized build Ninja matched none of those lines, recorded no dependencies for the translation unit, printed the notes as ordinary compiler output, and stopped rebuilding it when its headers changed — a wrong build under a zero exit code, which persists until someone cleans.

Set FASTCACHE_MSVC_DEPS_PREFIX to the value your build already holds:

Select-String msvc_deps_prefix build.ninja      # in your build directory
$env:FASTCACHE_MSVC_DEPS_PREFIX = 'Hinweis: Einlesen der Datei:'

The line that comes back is the whole assignment — msvc_deps_prefix = Hinweis: Einlesen der Datei:. What goes in the variable is the text after the =: a prefix that carries the assignment matches no line Ninja ever sees, and fails in exactly the same silence as writing English.

Ninja's behaviour here is measured rather than assumed, and the measurement is re-runnable: scripts/probes/ninja-msvc-deps-prefix.sh drives three cases (matching, mismatched, and localized-matching) and reports the dependency count and whether a header edit rebuilds. It needs no MSVC and no language pack, because the defect is a property of the string, not of any compiler's UI language.

Where the prefix comes from, in order. msvc_deps_prefix is a value the build holds and never exports, so there are three sources and the launcher takes the first that answers:

`FASTCACHE_MSVC_DEPS_PREFIX`	What the launcher does
A prefix, e.g. `Hinweis: Einlesen der Datei:`	Uses it. This is the only source that can be right about a build the launcher cannot see: you copied it out of `build.ninja`, so you are stating a fact.
`auto`	Asks the compiler. Writes a one-header translation unit somewhere temporary, preprocesses it with `/EP /showIncludes`, and reads the prefix back off the line that ends in the header it just wrote. Falls back to English if the compiler will not say.
unset	The English `Note: including file:`, on trust.

With FASTCACHE_VERBOSE the launcher names the prefix it used and where it came from, on every dispatched compile that synthesises notes — the one line that answers why did my build stop rebuilding this file.

Three settings, but four things that line can say, because auto has two outcomes and they need different remedies:

It says	What happened	What to do
`named by FASTCACHE_MSVC_DEPS_PREFIX`	You stated the prefix.	Nothing.
`discovered from the compiler`	`auto`, and the compiler answered.	Nothing.
`the compiler was asked and did not say, so the default; name it with FASTCACHE_MSVC_DEPS_PREFIX`	`auto`, and the probe learned nothing — so this build is on English and your notes may not match.	Copy `msvc_deps_prefix` out of `build.ninja` and set it directly.
`the default; override with FASTCACHE_MSVC_DEPS_PREFIX`	Nobody asked for anything; English on trust.	Nothing, unless your toolchain is localized.

The last two carry the same prefix and are told apart only by that line. They are separate because the remedy differs: override with FASTCACHE_MSVC_DEPS_PREFIX is advice already taken by whoever wrote auto, and telling them to do it again is a confident wrong answer to the one question they asked.

auto is opt-in because it costs a compiler spawn per file. One fastcache-cc process serves one translation unit, and under CMake + Ninja + MSVC /showIncludes is on every compile line — so a probe that ran whenever nobody named a prefix would start a second compiler for every file in your build, including on cache hits, where not running a compiler is the entire point. On an English toolchain it would pay that to rediscover the default. And the answer cannot be remembered to spread the cost: installing a language pack changes what the notes say without changing anything a cache stamp covers, so a remembered answer goes stale in the direction that produces a wrong result which looks right.

The chain, stated so nobody optimises it back into a cache later: the answer cannot be memoized, so the only correct automatic shape is per translation unit; per translation unit is too expensive to impose on every build; therefore opt-in. Each link depends on the one before it. If you find yourself about to add a cache here, the link that has to break first is the first one — and it does not break, because the thing that invalidates a discovered prefix is an installer run that touches no cache stamp.

So auto is for the build that needs it — and if you already know the string, setting it directly is both cheaper and more certain.

The probe runs in your build's own environment rather than the anglicized one the launcher uses for its other spawns, because the question is what your compiles print.

Two things it will not do, both deliberate. It refuses a line with blanks in front of the prefix, because a note begins at column zero — cl puts the inclusion depth after the prefix, not before it. And where two lines suggest two different prefixes it reports nothing rather than picking one: a #pragma message whose text happens to end in a path looks exactly like a note to this rule, and a wrong prefix is worse than none.

The localized case is untested against a real localized compiler, and that is recorded rather than implied: VSLANG selects among the language packs an installation has, so an English-only Visual Studio cannot be made to produce a German note. #878 stays open for that half. If you run a localized toolchain, set the variable.

Sharing a cache across UI languages is safe from generation 3 onward, and this paragraph records what it took, because the hazard was real and an operator running a mixed fleet on an older build still has it.

A note's prefix is now a canonical form, exactly as <SRCROOT> is a canonical form for a checkout root. The launcher rewrites its own prefix to the English Note: including file: before a value is stored, and rewrites it back to this build's prefix when a hit is replayed (NormalizeIncludeNoteMarker and RestoreIncludeNoteMarker — the direction is a name, not an argument position) — so what crosses the wire is locale-free, and neither machine ever sees the other's prefix. That is why nothing on either server changed: they only ever canonicalize the one spelling. It is the launcher that does both rewrites, and it has to be, since only the producing machine knows what language its own notes are in.

Before that, a German machine setting this variable fixed its own builds and began storing values whose notes an English peer replayed and could not match — new breakage on a machine that had none, since the cache key does not fold the prefix (a German and an English Visual Studio of the same toolset key identically, deliberately, because the identity probe is forced to English, #692). Closing it moved CompileValueVersion to 3 (#879), so a value written by a generation-2 build is refused rather than replayed.

The byte has moved twice since: to 4 for #202, and to 5 for #1270, which put back the half of the note anchor #891 gave away. Each of those is its own cold cache, and the paragraph below applies to every one of them unchanged. Expect one cold cache on the upgrade — one, because a refused generation now falls through to the miss path and the STORE that follows overwrites the key with a value of this generation. A bump that refused and then declined to re-store would leave the old value under a key that does not change with the generation, fetched and refused on every later build: not a cold cache but a dead one.

What this still does not cover. A local compile on a localized toolchain that has not set this variable stores a region the launcher cannot normalize, because nothing has told it what prefix that compiler uses — an unmatched marker rewrites nothing, so such a machine is exactly where it was rather than worse. Discovering the prefix from the compiler is #878, and it needs no further generation: the stored form is already locale-free, so #878 only supplies a better value to normalize with. Setting this variable fixes a localized machine's dispatched notes, and for an ASCII prefix it also gets that machine's stored regions canonicalized.

It does not do the second for a non-ASCII prefix, and that is a known gap rather than a claim being made carefully. The value arrives from the environment in this process's narrow text, which this project forces to UTF-8, while a local cl writes its notes in the console output code page — the same mismatch RootReconciler is handed a HostNarrowTextPolicy to deal with for paths. German happens to work because its prefix is ASCII; the Japanese and Chinese catalogues use a full-width colon (U+FF1A) and the Russian one is wholly non-ASCII, so the two byte strings differ, nothing matches, and the region is stored uncanonicalized with no diagnostic. A dispatched compile on the same host is unaffected, because the launcher synthesises those notes from the same UTF-8 value it compares against. Decoding the marker through the same policy as the paths is the repair, and it belongs with #878, which is already the ticket for learning the prefix properly.

How it works¶

Key. Direct mode first: re-hash the project headers a previous compile recorded and look up a manifest — far cheaper than preprocessing (~18 ms versus ~1.4 s on a large translation unit). If that misses, preprocess the TU, relativize checkout-rooted arguments against SOURCE_DIR/BINARY_DIR, and hash (compiler id + preprocessed text + relativized args + dependency paths) into a 128-bit key — MurmurHash3 x64_128, which is 128 bits of strength and not merely 128 bits wide; see the note in .agent/rules/compile-cache.md on why the four-CRC construction it replaced was not. Compiler id is the driver's banner and the target it generates for, where the driver can be asked: a banner alone identifies the driver, and a driver's code generation is not a function of the driver alone — clang-cl takes -fms-compatibility-version from the MSVC install beside it, and one stock g++ banner covers x86_64 and aarch64. A driver that states no target keeps its entries unchanged. Note this is the key, not the toolchain fingerprint a dispatched compile matches workers on, which deliberately omits the target — see Distributed compilation. The dependency set comes from that same preprocess run — -MD into a scratch depfile for GNU drivers, /showIncludes for MSVC ones — which costs about 1.5% of it, because the compiler has already opened every one of those files.
FETCH. On a hit, write the object to the requested output path and replay the cached stdout and stderr on their own channels, with header paths localized to this machine so the build tool's dependency records stay valid.
MISS. Run the real compiler capturing stdout and stderr separately, STORE the canonicalized result, and pass the output through on the true streams.
Any error. Fall back to a plain real compile.

A compile that fails is never cached. The object is stored once, under the preprocessed key; a manifest records that key rather than a second copy of the object, so a direct hit follows one extra fetch instead of doubling the cached volume (which, since the memory tier keeps values uncompressed, would land on RAM where compression cannot help).

Why the dependency paths are in the key¶

A hit reproduces two things: the object file, and the build system's dependency record — a GNU depfile, or the /showIncludes notes Ninja reads as deps = msvc. A cached value that cannot reproduce the depfile a compile names is not served: it is recompiled and stored again. Note that clang-cl under CMake's Ninja generator asks for a GNU depfile (-clang:-MD -clang:-MF<file>, deps = gcc), not for /showIncludes; the launcher reads that spelling, and a separated -clang:-MF -clang:<file> is compiled uncached, since its value cannot be read. Suppressing line markers keeps every path out of the hashed text, which is what makes a key portable across checkouts, and equally what once made it identical after a header moved: same bytes, new path, so the object was still correct and still served while the recorded paths were not. Replaying those makes the build system rebuild that translation unit on every build, forever, with a successful exit code each time.

Naming the dependencies in the key makes a move a different key, so the two layouts hold two entries and moving a header back finds the original one intact. Only machine-independent paths are hashed — those under the source root or build tree, plus relative ones. Toolchain and system paths are left out on purpose: they are the producing machine's spelling, the compiler identity in the key already covers them, and hashing them would stop two machines with the same compiler at different install prefixes from sharing anything at all.

That last exemption leaves one case open, so a hit is still checked before it is written: every dependency path it records that this machine is answerable for must exist. A hit that fails the check is discarded and the compile runs for real, which re-stores the entry with a correct record.

Reading the `dependency set:` line¶

Every preprocessed compile reports, under FASTCACHE_VERBOSE, what became of the paths the driver named:

fastcache-cc: dependency set: 61 of 635 reported path(s) keyed (476 toolchain; 60 filesystem call(s))

M is what the probe reported, N is the key's own dependency set, and the words before the semicolon say why the rest did not reach it.

They will usually not add up, and that is not a fault. M and the drop counts are per reported occurrence; N is the set after deduplication. /showIncludes names a header once per inclusion site, so the 159 occurrences left after the 476 drops above are 61 distinct files. A GNU depfile names each header once, so on those drivers the numbers do sum. What always holds is that the reasons account for every path that did not reach the key:

Reason	What it means
`toolchain`	Under neither root, or inside a vendored tree. The ordinary bulk of any compile: the compiler identity in the key already covers this content collectively. That identity is the compiler's own version banner — `Microsoft (R) C/C++ Optimizing Compiler Version 19.51.36252 for x64`, `clang version 22.1.3 (…)` — so two MSVC toolsets, or the x86 and x64 driver of one, key apart even though their headers are the same files.
`drive-relative`	A Windows `C:foo` path under neither root. It resolves against that drive's own current directory — per-process state on the producing machine that no cache entry can record. One under a drive-relative root is keyed like any other, or counted `toolchain` if it is vendored content.
`unanchored`	Relative, with no working directory to resolve it against.
`no canonical form`	Under no root, yet a character prefix of one: a root spelled almost right, such as `/x/build-other` against `/x/build`. Fix the root rather than looking for the file.
`empty`	The driver named an empty path.

0 of M with M non-zero is the line to act on, and the reason says which repair. Every path counted toolchain means the configured roots match nothing the compiler echoed back — on Windows usually an 8.3 short name in one of them, which is otherwise entirely silent: the key is empty, the replay guard has nothing to check, and the stored value keeps this machine's absolute paths. drive-relative or unanchored point at the build's own spelling of its include paths instead. 0 of 0 is a different fault: the driver reported nothing at all on the preprocess line.

Statistics¶

Each compile is its own short-lived process, so aggregation cannot live in memory. Every invocation appends one line to a per-user log and reporting folds it on demand:

Windows: %LOCALAPPDATA%\fastcache-cc\invocations.log
Elsewhere: $XDG_STATE_HOME/fastcache-cc/invocations.log, falling back to ~/.local/state/fastcache-cc/

Writes use an atomic append (FILE_APPEND_DATA / O_APPEND) so the hundreds of concurrent compilers in one build interleave whole lines instead of shredding each other's. Recording failures are swallowed: statistics never break a build.

$ fastcache-cc --show-stats

all prefetch groups
  compiles     : 4
  hits         : 2  (66.7% of 3 cacheable)
    via direct : 1
  misses       : 1
  unavailable  : 1  (25.0% of all compiles -- CACHE NOT REACHED)
  fall-back reasons
    1x  fetch exchange failed
  distribution
    not attempted   : 2
    unreachable     : 1
    dispatched      : 1  (50.0% of 2 asked of the fleet)
    why distribution did not help
      1x  the fleet could not be reached

  hit latency    2 samples  p50 12ms  p95 70ms  max 70ms
    preprocess   1 samples  all 65ms
    cache i/o    2 samples  p50 0ms   p95 1ms   max 1ms
  miss latency   1 samples  all 200ms

never cached (1 translation units)
  1x  volatile.cpp

The hit rate is computed over cacheable compiles (hits + misses), not all invocations, so an unreachable daemon does not silently look like a low hit rate — it is reported separately as CACHE NOT REACHED.

Latency is shown as a distribution rather than an average because compile latency is routinely multi-modal: a mean of 300 ms tells you nothing about whether that is every TU or a fast majority plus a slow tail. Fall-back reasons are itemized so unavailable is actionable — a refused connection and a rejected STORE call for very different responses.

The distribution section¶

The cache and the fleet are two failure domains, and this report keeps them on two axes. A compile can be a cache miss and a dispatch failure at once: the daemon answered honestly, the object was compiled and stored, and the fleet was still no help. Recording the second as a cache outcome would inflate unavailable and send you to look at the daemon, so it has a section and a reason list of its own — why distribution did not help is never merged into fall-back reasons above it.

The section appears only when there is a fleet to report on. With no FASTCACHE_SCHEDULER set, this launcher does not distribute, and printing dispatched: 0 (0.0%) for it would render an absence as a total failure. So it prints nothing at all — as does a log written before this section existed, which is silent about dispatch rather than evidence of no fleet.

Line	Meaning
`not attempted`	A scheduler is configured and this compile never reached the dispatch decision — the cache served it, it was uncacheable, or the key was never derived. Nothing about the fleet follows from it.
`refused here`	This launcher would not send the compile. The fleet was never asked, so this is fixed on your machine, not in the fleet — see the reason table below for which of the three it was.
`fleet declined`	The scheduler or the worker said no: no worker on this toolchain, no capacity, the key already in flight, or the worker refusing the job. Ordinary; the question is the fleet's size or shape.
`unreachable`	The scheduler or the worker could not be reached, or an exchange broke. Also ordinary, and fixed somewhere else entirely — a machine that is down or a network that is not carrying, rather than a fleet that is merely busy. That difference is why these are two lines and not one.
`crossed reply`	A worker answered about a compile other than the one it was asked for, and the object was refused unread. A defect somewhere in the fleet, not a fleet declining to help — see the entry in the fall-back table above, which is where its reason is ranked.
`result discarded`	A worker ran the compiler and this client did not keep the object: the remote compile exited non-zero and was retried locally, or the object or depfile could not be written here. The exchange worked and the compile was still done twice.
`dispatched`	A worker ran the compiler and the object was used. Distribution worked.

The rate is taken over what was actually asked of the fleet — fleet declined, unreachable, crossed reply, result discarded and dispatched — and not over every compile. not attempted and refused here are excluded because the fleet never saw them, so a build that mostly hit the cache does not read as a fleet that mostly refused. It is the same reasoning as the hit rate being taken over cacheable compiles: an absence must not be counted as a failure. When nothing was asked at all the rate reads n/a rather than 0.0%.

result discarded is a line of its own for the same reason. Folded into dispatched with an explanatory reason underneath, a fleet whose every object was thrown away would headline 100% dispatched — a green number over work done twice. It is also the only place a node that fails compiles which are fine is visible at all, because the local retry succeeds and the build stays green; a rising count here is the signal, and the reason says which of the three causes it was.

The dispatched line prints even at zero. A fleet that dispatched nothing is exactly what this section exists to make visible, and omitting the line would read as "no data" rather than as "none". Every other line is dropped when it is zero.

--html-stats renders the same states as a panel, driven by the same table, so a state cannot appear in one report and be missing from the other.

Why distribution did not help¶

Every reason that appears under why distribution did not help. All of them still produce a correct object — the translation unit is compiled locally instead — and none of them is a caching failure.

Reason	Meaning
`the command line is not dispatchable`	`RemoteCompileArgs` found something on the line it cannot account for, so nothing was sent. Refusing costs one local compile, where stripping an unrecognised argument would change the generated code and hand back an object nobody asked for. `FASTCACHE_VERBOSE` names the offending flag; it is deliberately not in this tally, or you would get one row per command line instead of one per cause.
`the dispatch preprocess failed`	A worker is fed a second preprocess, with `#line` markers that the cache key's copy suppresses, and that run failed. The key's own preprocess had already succeeded, so this is about the marker-emitting form of the command specifically.
`this toolchain has no usable fingerprint`	The toolchain digest does not identify this compiler, so no worker could match it. Deliberately refused here rather than sent: a scheduler asked for an unidentifiable fingerprint answers `NoWorker`, which reads as "the fleet has nobody on your toolchain" and sends you to look at the fleet for a problem on this machine. `fastcache-cc --print-toolchain-fingerprint <compiler>` says what this machine computes and why it is unusable.
`no worker serves this toolchain`	Nothing in the fleet carries your compiler's fingerprint. A machine is missing, or a fingerprint has drifted — upgrading the toolchain on the clients and not the workers looks exactly like this. Never read as a capacity problem: more machines of the wrong compiler change nothing.
`the fleet was full of its own work`	Every matching worker was busy with this fleet's own compiles. The fleet is too small, which is a different purchase from the row above and from the one below.
`matching workers had withdrawn their slots`	Machines that serve your toolchain had slots free on paper and were not offering them — somebody else is using them, a scratch disk is full, or a survey is in flight. The fleet is big enough and unavailable.
`another client was already building this key`	Duplicate-work suppression, and not a failure: another client holds the lease for this exact object and this compile ran locally instead. Sixty clients missing one key after a header change is the ordinary shape of a shared cache. A high count here is the design working.
`the fleet refused this client`	A credential, membership or lease refusal — the fleet would not let this client ask. Fixed where the client is configured or on the scheduler's member list, never by adding machines.
`the worker refused the job`	A lease was granted and the worker then said no: a lease it would not honour, a scratch root it cannot write, a compiler it could not spawn. One machine to go and look at, and `FASTCACHE_VERBOSE` names it.
`the fleet named no leader to ask`	The scheduler chain answered `NotLeader` until the redirect ceiling. Transient during an election and permanent when a fleet is misconfigured; the rate is what separates those, which is why it is not folded in with the credential refusal above.
`this launcher and the fleet disagree about the wire`	A protocol, codec or framing mismatch. Expected and bounded during a staggered upgrade; a count that keeps rising afterwards names a machine that never came back.
`the fleet refused with a reason this launcher does not know`	A refusal code newer than this launcher. Upgrade `fastcache-cc`; the verbose line carries the code the fleet actually sent.
`the fleet could not be reached`	The scheduler or the worker did not answer, broke mid-reply, or ran out of budget. If every compile shows this, check the address in `FASTCACHE_SCHEDULER` before suspecting the fleet — a wrong one looks exactly like a fleet that is entirely down.
`a worker compile failed and was retried locally`	The remote compiler exited non-zero, so the result was discarded and the translation unit compiled here to confirm. Broken code produces this on every machine and is not a fleet problem; a rising count against a build that keeps succeeding is a worker producing failures that are not real, and the verbose line names the machine.
`the dispatched object could not be written`, `the depfile for a dispatched compile could not be written`	The object came back and this machine could not store it — a full disk or a read-only output path. Local, not a fleet problem, and the compile is redone here.

Exit codes¶

Situation	Code
Cache hit	`0`
Miss, fall-back, or non-cacheable line	the real compiler's exit code, verbatim
Compiler could not be spawned	`1`
`--help`, `--version`, `--show-stats`	`0`
`--zero-stats` failed	`1`
No arguments at all, or an unknown option	`2` (diagnostic and usage printed to stderr)

Fall-back reasons¶

Every reason that appears under fall-back reasons, and what to do about it:

Every one of them still produces a correct object. Which of them leave distribution running is a narrower claim than that, and worth reading precisely: the two that describe a daemon failing to serve the lookup — an unreachable one (fetch exchange failed) and one that answered and refused (rejected (…)) — carry on and dispatch, which is the note at the top of this page. The rest end the invocation at a local compile, because each of them is a reason there is nothing to dispatch with: the key is not computed yet (the configuration and path reasons), the preprocess itself failed, the translation unit is one the launcher deliberately steps over, or the object could not be written on this machine. The ones marked uncacheable are the launcher's own refusals and are not about the daemon at all.

One reason in the table is not about the daemon or about a fleet declining to help, and it is the only one worth interrupting somebody for: a worker answered about a different compile. It is printed on stderr whether or not FASTCACHE_VERBOSE is set, because it means a machine in the fleet produced an object for work nobody asked it to do. Every other reason here is quiet unless you ask.

Reason	Meaning
`missing FASTCACHE_ADDR/SOURCE_DIR/BINARY_DIR`	Configuration incomplete — the cache was never contacted, and neither was a scheduler. Distribution is off here deliberately, and not merely for want of a key: `FASTCACHE_ADDR=` (set but empty) is how a build opts out of the launcher altogether, and without the two roots there is no portable key for a scheduler to suppress duplicates on. Set all three to use either.
`preprocess failed`	The compiler rejected the preprocess probe; the line may use an unsupported option form.
`uses __TIME__/__DATE__/__TIMESTAMP__`	Deliberate: the TU is non-deterministic and would never hit. Reported as uncacheable, not as an error.
`a command-line path is drive-relative under no root`, `a reported dependency path is drive-relative under no root`	Deliberate, and Windows-only. A path like `C:foo\bar.hpp` resolves against drive `C:`'s own current directory, which no cache entry records — so the launcher can neither key it (a header moved inside it would not re-key) nor check it on replay (there is no directory to `stat` it against). Caching such a compile could serve a stale dependency record under a zero exit code, so it is not cached at all. Reported as uncacheable, not as an error. Spell the path absolutely (`C:\foo\bar.hpp`), make it relative, or bring it under `FASTCACHE_SOURCE_DIR`/`FASTCACHE_BINARY_DIR`. The first is the rule applied to the command line, the second to what the compiler reported; `FASTCACHE_VERBOSE` names the offending path itself.
`daemon does not support authentication; the configured credential was ignored`	`FASTCACHE_TOKEN` is set but the daemon predates the AUTH verb. Caching works normally — the daemon steps over the verb it does not know and serves the command — but this traffic is not authenticated. Said once per invocation rather than per exchange. Upgrade the daemon, or unset the token if it was not meant to apply here.
`rejected (unauthenticated): ...`	The daemon requires a credential. `authentication required` means none was sent — set `FASTCACHE_TOKEN`. `authentication failed` means one was sent and was wrong. The two are deliberately different messages because they are different mistakes. Either way the build succeeds and only the caching is lost — the compile is still dispatched if `FASTCACHE_SCHEDULER` names a scheduler, and runs locally otherwise.
`fetch exchange failed`, `fetch decoded malformed`	Transport or protocol trouble mid-request. Also how a `FASTCACHE_TIMEOUT` expiry surfaces: a daemon that accepted the connection and then went quiet. If these appear in bulk and each compile stalls for the full timeout first, suspect a wedged daemon rather than a flaky network. `fetch exchange failed` is also what a plainly wrong `FASTCACHE_ADDR` looks like — every compile, at once, with the fleet still doing the work. The two differ in what happens next: `fetch exchange failed` carries on and dispatches, while `fetch decoded malformed` — a daemon that answered with a value this launcher cannot read, which in practice means a mixed install — ends the invocation at a local compile.
`fetch decoded another generation's value`	The daemon served a value that IS a compile value, well formed, written under a canonicalization generation this build does not implement. Ordinary during a rolling upgrade, and not a damaged cache — which is the whole reason it is not `fetch decoded malformed`. Either direction: a producer behind this launcher reads exactly like one ahead of it. The compile happens locally, nothing is stored, and it clears once every server and launcher in the fleet are on one generation. The wire code's side of it is `foreign-value-generation`.
`rejected (unsupported-version): …`	The daemon answered and declined: it does not speak this launcher's wire version. The two binaries ship together, so this means a mixed install — an old daemon still running against a new `fastcache-cc`, or vice versa. The message names the range the daemon does support. Restart the daemon from the same package as the launcher.
`rejected (payload-too-large): …`	The object exceeded the daemon's `--storage-max-value`. Raise it, or accept that this TU will not cache.
`rejected (…)` (other codes)	The daemon refused the command and said why; see the error-code table.
`could not write object on hit`	The object output path was not writable.
`a worker answered about a different compile`	A defect somewhere in the fleet, not a fleet declining to help. The worker's reply did not belong to the request that asked for it — see `correlation`. The object is refused unread and the translation unit is compiled locally, so the build is correct and the caching of it is unaffected (the outcome is still a miss). This is the one reason printed unconditionally rather than only under `FASTCACHE_VERBOSE`, and the line names the worker, the correlation this client expected and the one that arrived. Accepting such a reply would store a wrong object under a correct key and serve it to every other machine that fetches it, so there is no configuration that relaxes this. If it appears at all, find the machine the line names.
`a reported dependency path is not text this host can read`, `a captured region names a path that is not text this host can read`	Deliberate, and Windows-only. `cl.exe` writes the paths in `/showIncludes` in the console output code page, while this launcher's own roots arrive as UTF-8 -- so a header under a non-ASCII directory can reach it as bytes it cannot read as text. Such a path prefix-matches no root, which would key a project header as toolchain content and serve a stale object under a zero exit code, so the compile is not cached at all. Reported as uncacheable, not as an error. The fix is the console: `chcp 65001` makes `cl` emit UTF-8 and this stops appearing.

Verifying that a hit is the right object¶

A wrong object served under a correct key is this cache's worst failure: it links, it usually runs, and nothing says anything. FASTCACHE_VERIFY=<n> makes one hit in every n prove itself — the translation unit is compiled again and the two objects are compared. On a mismatch the freshly compiled object is the one left on disk, so a build that has just caught its cache is still a correct build, and the key is named on stderr whether or not FASTCACHE_VERBOSE is set.

It costs a whole compile per verified hit, so it is off by default.

What "the same object" means is not "the same bytes", and on Windows it cannot be. Every MSVC-family driver stamps the wall clock into the COFF header, and a cached object was compiled earlier than the fresh one it is checked against by construction. Measured on MSVC 14.51 and clang-cl, two compiles of one translation unit to one object path differ in that 4-byte TimeDateStamp and in nothing else, whether two seconds or five minutes apart, with /Z7 or without. That one field is therefore normalised, and nothing else is. On Linux nothing is normalised at all: clang 20.1 and GCC 14.2 objects are byte-identical between two compiles, -g included, so ELF keeps a strict byte comparison.

What that deliberately does not overlook is cl's record of where it compiled — the object's absolute path in .debug$S, and the source hash in .chks64. Those vary with the path rather than with the clock, and verification recompiles to the same path, so a hit produced on this machine differs in neither. A hit that does differ there was built somewhere else — another checkout, or another machine — and is reported, with the message saying so rather than leaving "wrong object" to mean either that or stale code.

Four outcomes reach you, and only one of them is a finding. The last two are different from each other in the way that decides what to do next — one is worth retrying, the other never will be:

Outcome	Meaning
(silence)	The cached object is the object this compiler produces.
`WRONG OBJECT served for key …`	It is not. The message names what differed — a section such as `.text$mn` is stale code; `.debug$S` or `.chks64` is a foreign build path. The fresh object was used, so this build is unaffected. Find the machine that stored it.
`could not verify the hit for key …`	The check did not complete: the fresh compile failed, or a file could not be read. Nothing is known about the cached object either way, and the next hit may well answer.
`cannot verify hits for this toolchain …`	This build cannot lay out the object format its own compiler produced, so it can say nothing about any hit — not this one and not the next. A property of the toolchain, not a statement about your cache.

Debug paths in a replayed object¶

A cache hit replays an object that was built somewhere else, and a compiler with debug information on records where it was built — DWARF's DW_AT_comp_dir is the compile's working directory. That directory is on no command line, so nothing in the cache key can distinguish two checkouts by it, and a debugger opening the replayed object then looks for sources in a tree this machine may not have. Nothing fails; the paths are simply somebody else's.

On GCC and Clang the build does something about it for you. When cmake/portable/CompileCache.cmake enables a launcher it also appends

-fdebug-prefix-map=<source tree>=<relative path from the build tree>
-fdebug-prefix-map=<build tree>=.

which makes the objects byte-identical across checkouts and DW_AT_comp_dir exactly . — both measured, on both drivers. The build tree is mapped last because GCC and Clang honour the last matching rule, and the build tree usually lives inside the source tree; the other order leaves comp_dir naming the build directory, which is still checkout-independent and therefore invisible to an object comparison, while two build trees at one depth under different names then share a key. The source rule is emitted only when the relative path back is a pure ../ chain; for an out-of-tree build it would otherwise carry the checkout's own path, and the configure says so rather than mapping to something that only looks portable:

-- [cache] The source root is NOT mapped: the build tree lies outside it, so the
   relative path back would carry the checkout's own path

Passing your own -fdebug-prefix-map is fine and is keyed correctly: the launcher relativizes the root the flag names, so two checkouts still share, and leaves the replacement literal, so two machines mapping to different replacements miss rather than exchange objects that disagree. The mapping has to be the same everywhere a cache is shared, and that is enforced by the key rather than left to a convention.

-ffile-prefix-map and -fmacro-prefix-map are deliberately not recognised, and passing either costs cross-checkout sharing for that translation unit. They rewrite __FILE__, which lands in the preprocessed text the key hashes, so the launcher cannot relativize them without hashing text the real compile does not produce — and a dispatched compile would then bake the unmapped __FILE__ into an object stored under the same key. Unrecognised, they reach the key verbatim and two checkouts simply miss, which is the safe direction.

A root containing a space is not mapped at all: these rules are spliced into CMAKE_<LANG>_FLAGS, which is space-separated, so one rule would arrive at the driver as two arguments and every compile would fail. The configure says so.

A dispatched compile records the same compilation directory a local one does. The flag itself is never forwarded — it is a path-valued argument, and RemoteCompileArgs drops every one of those, correctly: a worker needs a rule whose left-hand side is a path on the worker, which your machine has never seen. What travels instead is your own compile directory and what your mapping spells it as, and the worker builds the rules. So a fleet-built object and a locally built one both record ., and a debugger resolves sources the same way whichever produced the object in your cache.

The worker maps two directories, both to your replacement, because which one a dispatched object records is a fact about the driver rather than about the fleet. Measured, reading DW_AT_comp_dir:

the preprocess line	what the worker's object records
`g++ -E`	the worker's directory
`g++ -E -g`	your directory
`clang++ -E`, `clang++ -E -g`	the worker's directory

gcc's -fworking-directory is implicit under -g: it puts a line marker naming the preprocessing directory into the text, and the worker's compile adopts it. clang emits no such marker. Mapping both candidates gives one answer either way.

Consequences worth knowing:

A build reached through a symlink is mapped correctly, and that takes care. Both drivers record — and match -fdebug-prefix-map against — $PWD when it is absolute and names the same directory as ., falling back to getcwd(3) otherwise. So if you cd to a build through a link, your rule spells the link and the driver agrees, while the resolved path matches nothing. The launcher predicts from the same value your compiler will use, so the dispatched object and the local one still agree. It is worth stating because getting it wrong is invisible: the mapping simply never travels, the fleet-built object keeps an absolute path, and nothing reports a problem. macOS is where this shows up first — /var is a symlink, so anything under $TMPDIR is.
If your build passes no mapping, a worker adds none. An object built on the fleet then records the worker's directory, exactly as it did before — which is the honest answer, because there is no directory your build would rather see.
A worker that cannot honour the mapping refuses the job rather than returning an object that disagrees, and your compile runs locally. That happens when either directory contains an = (gcc and clang split <from>=<to> at opposite ends, so no unambiguous rule exists) or when the worker's driver has no path-map switch at all. It shows up as a refusal on the worker's metrics and costs one local compile.
A worker running in / maps only your directory, not its own. A prefix-map rule appends the unmatched tail, so a rule whose left-hand side is / would rewrite every absolute path in the object — /usr/include/... becomes .usr/include/.... The shipped fastcache-compile-node.service sets no WorkingDirectory=, so that is the ordinary Linux deployment. Your own rule still lands, which covers gcc completely; under clang such a node's objects keep its working directory, exactly as they did before. Giving the unit a WorkingDirectory= of its own closes that too.

The #line markers a worker is sent still carry the dispatching machine's paths, so a dispatched object's line table names the producing checkout's headers. That half is #506's remainder rather than its subject: on gcc the file names come from those markers and already match a local compile's, and on clang the compilation unit's own name is the worker's scratch file (#660).

On Windows there is no equivalent and the paths stay. cl has no path-map switch, and -ffile-prefix-map does not reach the records that matter for clang-cl — CodeView's S_OBJNAME and the embedded -cc1 line are not remapped, measured. So a replayed object on Windows names the checkout that produced it in its debug records. That is an accepted cost of cross-checkout sharing, which on that platform is most of the value: closing it means keying on the compile's location, and then no two checkouts ever share an entry. It affects debug information only — .text, .data, .xdata, .pdata, the relocations and the symbol table are byte-identical.

Two things this is not:

It is not __FILE__. The preprocessor expands __FILE__ and the key hashes preprocessed output raw, so a translation unit whose __FILE__ differs between checkouts also keys differently and the two never share an entry. There is no arrangement in which a program reports a source location from another checkout.
It is not coverage. Coverage mapping has the same defect and is guarded outright: configuring a coverage build with a launcher imposed is a hard error, not a warning.

Known limitations¶

__TIME__ / __DATE__ / __TIMESTAMP__ detection scans the source file text. Direct use is caught; use reached only through a header is not, so such a TU stays a permanent miss. Never incorrect — just never cached.
A Windows drive-relative path (C:foo) reaching a compile takes that translation unit out of the cache entirely, rather than being ignored. Only clang-cl echoes such a path back unresolved (cl resolves it through the filesystem), and no common generator emits one, so in practice this costs nothing — but where it does fire it costs the whole TU rather than one path, deliberately: a partial answer here is the stale-dependency serve it exists to prevent. The refusal is silent unless FASTCACHE_VERBOSE is set, and shows in --show-stats as uncacheable.
Diagnostics-stream paths outside the include grammar are not yet localized.
Toolchain and system dependency paths are in neither the key nor the existence check, by design: they are the producing machine's spelling, the compiler identity in the key already covers them, and including them would make two machines with different system include prefixes share nothing. So a cache shared between machines whose compilers print the same version banner and target the same triple, from different prefixes, can still replay a dependency record naming a path the consumer lacks. Project headers — the ones that actually move — are covered by the key, so this is now confined to the toolchain.
The cache key normalization is deliberately young (objkey-v6). Tune it against real developer↔CI hit rates before relying on it broadly. Bumping the schema re-keys the cache: existing entries miss once and are rewritten.
Localized path separators may be normalized to / in some segments. Ninja matches dependencies separator-insensitively, so this is cosmetic.
/showIncludes note paths are emitted with their .. segments collapsed, because Ninja refuses a note path longer than _MAX_PATH (260) by length, before it canonicalizes one — a fixed-buffer check no long-path policy reaches. A note path that is over the limit with nothing to collapse is still refused, and no lexical rule can change that; the remedy there is a shorter build root. The bytes stored in the cache keep the driver's own spelling.
Non-C/C++ inputs (for example Windows .rc resource files) are correctly classified as non-cacheable and passed through.
FASTCACHE_TIMEOUT bounds one exchange, not a whole invocation. Direct mode makes a separate manifest round-trip before the object fetch, so a compile against a daemon that accepts and then goes silent can wait up to twice the timeout before falling back.
A dispatched compile exceeding FASTCACHE_DISPATCH_TIMEOUT is abandoned by the client, which hands the lease back and compiles locally — so the build still succeeds and the key is not pinned for the scheduler's lease timeout. The worker notices the client has gone and skips writing the object back, so the transfer is not paid for; the compile itself still runs to completion, so that CPU is spent twice.
That deadline is no longer how long a dead worker goes unnoticed. The compile exchange dials with TCP keepalive armed, so a worker whose machine has vanished — powered off, unplugged, suspended, or cut off — is noticed in about 16 seconds on Linux and macOS and about 30 seconds on Windows, whatever the deadline is set to. Raising FASTCACHE_DISPATCH_TIMEOUT for a slow translation unit therefore does not make hard-failure detection slower.
Nor is it how long a WEDGED worker goes unnoticed. Keepalive cannot see a machine whose kernel answers every probe while the process above it makes no progress, so the worker says so itself: a running compile writes a Status::Progress frame every five seconds, carrying nothing, and FASTCACHE_DISPATCH_IDLE bounds the silence between them at thirty seconds (#245). The launcher's fall-back line reports it as stopped reporting progress, which is a different remedy from ran out of budget.
Those pulses are why the 0xFC protocol version is 3, and why version 3 is also the oldest a server accepts: a reply carries a status byte and no kind, so a client built before this cannot step over a frame it does not know the way it can step over an unknown request verb. An older launcher meeting a pulse would abandon the compile minutes in, as a transport failure naming nothing; refused at the request instead, it is told unsupported-version and which range would have worked.

Measured behaviour¶

Validated against a large real C++ codebase by replaying its compile_commands.json (4223 translation units) through the launcher twice — once to populate, once to measure. On a 400-TU sample the second pass reached a 99.7% hit rate (393 of 394 cacheable), with the classification of the non-cacheable remainder identical across both passes: stable and correct.