Skip to content

Distributed compilation

A cache makes the second build of a commit fast. It can do nothing about the first one: on a miss, fastcache-cc runs the real compiler locally and stores the result.

Distributed compilation absorbs those misses. Machines that would otherwise sit idle register as workers, and a client that misses the cache hands the translation unit to one of them instead of compiling it itself.

It is opt-in at both ends and cannot break a build: every refusal, every unreachable worker, every mismatch falls back to the local compile that would have happened anyway.

It also does not depend on the cache being up: the setup below deliberately puts the shared cache and the scheduler on different machines, and a cache that is unreachable or that refuses this client costs the lookup and the store, nothing more. Why that is worth stating is in How it works.

This page is how to set it up and run it. If you want the model first — what the pieces are and how one compile flows through them — read How it works, which is a shorter read and makes everything below easier to place.

Architecture

Three roles across two programs, with the shared cache behind them. Scheduler and worker are the same binary — fastcache-compile-node — because being the one that hands out capacity is a role a node takes, not a different program.

Role Program What it does
Client fastcache-cc Fronts each compile. Checks the cache, and on a miss asks for a worker.
Scheduler fastcache-compile-node --serve-scheduler Tracks the fleet's workers and hands one out. Exactly one node at a time.
Worker fastcache-compile-node Compiles what it is sent. Also holds a cache tier of its own, unless you turn it off.
Shared cache fastcached Optional, and not the scheduler. Where objects end up so other machines get them. 
   ┌──────────────┐          ┌───────────────────────────────┐
   │ fastcache-cc │          │ fastcache-compile-node        │
   │  (client)    │          │  (this one leads: scheduler)  │
   └──────┬───────┘          └───────────────┬───────────────┘
          │                                  │
          │  1. FETCH ──────────────────────►│ :6674  its cache tier,
          │     ◄──────────── hit: done      │        reading through to
          │                                  │        fastcached upstream
          │  2. miss: "give me a worker" ───►│ :6675  scheduler
          │     ◄──────── endpoint + lease   │
          │                                  ▲
          │                                  │ register + heartbeat
          │                   ┌──────────────┴─────────┐
          │  3. compile this ►│ fastcache-compile-node │ :6674
          │     ◄──── object  │      (a worker)        │
          │                   └────────────────────────┘
          │
          └─ 4. STORE the object ─► :6674  ← the CLIENT stores, not the worker

fastcached is behind all of it as the fleet-wide store a node's --upstream points at. It serves the cache verbs and nothing else: a scheduling verb arriving at one of its listeners is refused with a typed dispatch-not-permitted naming where the scheduler went.

The picture above is the compile path. For the whole topology — consensus, discovery and the dashboard history alongside it — plus a directional table of every connection and what to open on a firewall, see Cluster communication.

Why the scheduler knows what is already being compiled

A LEASE request names the object key, not just the toolchain — so the one node handing out capacity sees the whole fleet's misses as they arrive, which is something neither distcc nor sccache-dist is positioned to know.

When a header changes and sixty parallel clients miss the same key — the ordinary shape of a miss on a shared cache, not an exotic one — the scheduler dispatches one job and tells the other fifty-nine to compile locally. Without that, sixty machines compile the same translation unit and fifty-nine of the results are thrown away.

Why the client stores the result, never the worker

A STORE is trusted today because whoever stored it compiled it themselves — they could only poison their own key space with something they would have gotten anyway. If workers stored, one rogue worker could poison the shared cache for everyone.

Routing the result back through the client keeps that model exactly as it was. Workers are given no cache credentials at all.

How it works

1. The client preprocesses, and that is the part that cannot be distributed

fastcache-cc must preprocess to compute the cache key before it can know whether there is a miss. That work is unavoidable and always local — roughly 45 ms against compiles of 300 ms–2 s, which is where the ceiling of about 10–40× comes from. Distribution is not linear scaling and should not be sized as if it were.

2. On a miss, the client asks for a worker

The request names a toolchain fingerprint and the object key. The scheduler matches the fingerprint byte-identically and picks the matching worker with the most free slots — ties broken by utilization, so between two machines with four slots free the one with proportionally more of itself left takes the job.

Free slots rather than fewest running jobs, because absolute counts treat every machine as an identical box: a 64-slot server running 8 jobs looks busier than a 4-slot laptop running 2, when the server has 56 slots free and the laptop has none. Across a fleet of mixed machines — the ordinary case — counting jobs sends work to the smallest machines first and leaves the big ones idle.

The scheduler refuses rather than queues — no-worker, no-capacity, withdrawn, already-in-flight — because the client is holding the source and can simply compile it. Queueing would buy latency and nothing else. Each names a different operator problem, which is why they are counted apart; see the sizing notes.

3. The client sends preprocessed text, not files

The worker receives the translation unit already preprocessed, so there are no headers to ship and no sysroot to replicate. This is distcc's non-pump model, i.e. the one that works.

One subtlety with real consequences: the text sent to a worker is not the text the cache key hashed. The key's copy has #line markers suppressed so no checkout path can reach the key; a compiler needs those markers to know which lines came from a system header. Without them every warning inside libc++ or the CRT is re-reported against your own file, and under -Werror that is a failed compile. So the client preprocesses a second time for dispatch.

4. The worker compiles it — with a compiler it chose

The job names a fingerprint, never a program. The worker maps that fingerprint to a compiler it serves and refuses one it does not have.

Which compilers those are is a fact the worker establishes for itself: it surveys the machine at startup, so a package install is the whole setup. --toolchain narrows that set rather than supplying it.

This is the difference between a build accelerator and a remote shell, and it is why there is still deliberately no default compiler. "No default" and "no discovery" are different claims — a default is how a job ends up running against something nobody chose, while which compilers a machine holds is simply a fact.

The worker also re-checks the argument list, because the client's check protects an honest client from dispatching something that would not work, while the worker's protects it from a client that is not honest.

5. The client writes the object, reproduces the dependency record, and stores

The worker compiled preprocessed text, which has no #include left in it, so it reports no dependencies. The client already knows them — its own preprocess pass opened every one — so it writes the depfile (or /showIncludes notes) itself. Skipping that would leave the build with no header dependencies for that translation unit, so it would stop rebuilding when those headers change.

A dispatched compile is then shaped to look exactly like a local one, so the STORE, the manifest and the statistics all take one path.

6. And the client checks that the reply belongs to its request

Every step above is upstream of the reply. The cache key covers the inputs, the fingerprint covers the toolchain, the lease covers the authorization — and none of them says anything about whether this object answers this job. So the worker digests what it actually compiled and sends that digest back, and the client recomputes it from what it asked for and refuses a reply that does not match, before the object is even decompressed.

Getting that wrong in the other direction is the worst failure this system has: a crossed reply accepted as an answer is a wrong object under a correct key, which succeeds, gets stored, and is then served to every other machine that fetches that key. There is nothing else in the pipeline that would notice. The refusal is therefore a refusal — the translation unit is compiled locally and the launcher prints a line naming the worker, whether or not FASTCACHE_VERBOSE is set.

It is integrity against accident — a connection reused wrongly, a scratch collision, a worker answering out of order — not against a hostile worker: the digest is unkeyed, so a worker that can return a wrong object can return a wrong digest as well.

Setting it up

The scheduler

The scheduler is a compile node, not the cache. Pick one machine and give it --serve-scheduler:

fastcache-compile-node \
    --serve-scheduler --listen-node=0.0.0.0:6675 \
    --listen-raft=6680 --raft-self=scheduler.internal \
    --fleet-member=worker-01.internal \
    --fleet-member=worker-02.internal \
    --fleet-member=dev-01.internal \
    --scheduler=127.0.0.1:6675 \
    --advertise=scheduler.internal:6675

A scheduler is a cluster, even of one (#178). It signs every lease with its own identity key, and it hands its workers a roster — the cluster's voters and their keys — that a majority of those voters certify, so it holds replicated state and runs consensus to keep it: --listen-raft turns consensus on, and --raft-self states the host another member would dial it at (127.0.0.1 when none ever will). One started without --listen-raft is refused, by name. Its identity is minted into its state directory on first start, and the same command line with --print-identity added prints it without serving — the public-key line is what every worker's --voter-key names.

There is no shared secret to provision: the pre-shared --cluster-key-file a fleet used to copy to every member is gone (#178). Every machine proves its own identity key instead -- a member to the other members, a worker to its scheduler -- and removing one machine is revoking that one key. What that buys is under Security.

It used to be fastcached --listen-dispatch=..., and that flag is gone rather than deprecated. The two jobs have opposite deployment shapes: a cache is shared infrastructure somebody operates, while handing out capacity is a decision only one node may make at a time — and nothing in the cache daemon can establish which node that is. So scheduling moved to where cluster leadership lives.

A scheduling verb arriving at a fastcached listener is refused with a typed error naming where the scheduler went, so a client configured for the old layout tells you what to fix instead of failing mysteriously.

No --toolchain is needed: the node surveys the machine at startup and serves what it finds. A scheduler is therefore also a worker unless told otherwise — every node is a peer, and being the one that schedules is a role rather than a different program.

To keep a machine out of the work — a small always-on box, a VM whose cores belong to something else — give it --slots=0. It then runs no worker at all: it surveys no compilers, claims no scratch directory, registers nothing and is never sent a compile, so a setting only a worker reads — --toolchain or --node-class, say — refuses to start it, by name. --scheduler it may keep: on such a node it registers nothing and only tells the --cluster-* and --enroll-* commands where to ask. What it can still run is the scheduler, consensus and a cache tier; a node running none of those is refused too. Until #1440 its /metrics and history still read 0 slots.

fastcache-compile-node --serve-scheduler --slots=0 \
    --listen-node=0.0.0.0:6674 --fleet-member=10.0.0.21 ...

Such a machine shows among the cluster's members when it runs consensus, and not among the fleet page's machines, which are built from worker registrations; its own history is not handed to a leader either (#1440).

Once several nodes schedule, exactly one of them may at a time, which is what the same consensus decides once it has more than one member — --raft-peer naming each of them, each with its identity key. See a cluster, and who leads it.

Who may use the fleet

--fleet-member lists the hosts this node answers at all, and it is matched by host: a peer dials from an ephemeral source port, so an endpoint is not something a connection can be compared against. It is not only the worker list — every scheduler verb is gated, LEASE included, so a client machine that is not on it is refused a lease and compiles locally. List the machines that ask as well as the machines that answer.

--fleet-open admits everybody, for one machine or a network that is already your boundary. Give one of them — or admit each client machine with --cluster-admit-client, which the cluster replicates: a scheduler with none of the three refuses every caller but its own machine, which is the right default and not a working configuration. The startup rules cannot see the third route, so they no longer refuse a scheduler naming neither flag; its /metrics shows the refusals instead.

Membership gates two of a node's surfaces — its scheduler and its compile port — and this machine is always a member of its own fleet whatever the list says. What membership pays for on a node is CPU time, and those two are where it is spent.

It does not gate the node's own cache tier, which serves this machine and nothing else however either list reads (#287): that tier is a developer's whole build output, and contributing capacity does not entitle another machine to read it. Nor does it gate the shared fastcached, where a non-member reads and writes objects exactly as before — that one is infrastructure somebody operates, and it is what a cache several machines share looks like.

Because it gates the compile port, every node needs one of the two flags, not only the scheduler. A worker without one admits its own machine and refuses the network, which is the right default for a developer's laptop and cannot receive a dispatched compile.

The workers

On each machine that should take work:

fastcache-compile-node \
    --scheduler=build-cache.internal:6675 \
    --listen-node=0.0.0.0:6674 \
    --advertise=worker-01.internal:6674 \
    --fleet-open \
    --voter-key=<scheduler-public-key> \
    --cluster-dir=/var/lib/fastcache-node

--cluster-dir is where a worker keeps WHO IT IS (#178). On its first start it mints an identity key there, and from then on it proves that key on every connection to its scheduler; the scheduler refuses every verb a machine joins the fleet with -- registering, announcing, heartbeating -- on a connection that proved nothing, so a worker started without --cluster-dir is refused at startup instead. The cluster admits the key once, and either way works:

# on the worker: ask a member to enrol it, then approve it there with --enroll-approve
fastcache-compile-node --cluster-dir=/var/lib/fastcache-node --enroll-from=scheduler.internal:6675

# or print the worker's identity, and admit it from anywhere
fastcache-compile-node --cluster-dir=/var/lib/fastcache-node --print-identity
fastcache-compile-node --scheduler=scheduler.internal:6675 --cluster-admit-worker=<id>@<key>

An address still admits a client -- a developer's fastcache-cc asking for a lease needs no identity -- but never a machine joining the fleet.

--voter-key is how a worker knows whose leases to honour (#178). Every lease is signed by the scheduler that issued it, with that machine's own identity key, and the worker checks the signature against a roster: the cluster's voters and their keys, endorsed by a strict majority of those voters, re-endorsed every 15 minutes and good for an hour at a time. --voter-key names the voters it trusts before it holds one — paste each voter's public-key line from --print-identity, one flag per voter. The first roster a majority of them endorses is adopted from the scheduler's reply, and from then on only the roster held certifies its successor, so a key typed here never outvotes the cluster's own revocation. The roster is kept in --cluster-dir across restarts, and a worker enrolled with --enroll-from is handed one at admission -- it then needs no --voter-key at all. A worker another machine can reach will not start with no way to check a lease; one reachable only from its own machine needs none.

A worker cut off from the leader goes on honouring grants for as long as its roster stays certified, and then refuses every one roster-expired: that is the bound on how long a scheduler the cluster has since removed can go on leasing it out. fastcache_node_roster_expires_in_seconds says how long is left.

--listen-node is not optional on a worker that serves a fleet. It defaults to loopback, which is right for the single-machine install and unreachable for everybody else — so a worker that leaves it alone advertises an address no client can dial. The node refuses to start rather than registering one, but the flag is the fix and it is easy to leave off.

--scheduler names a DNS name or a VIP, never a scheduler's literal address. Every worker carries that value for as long as it is installed, so decide it before the rollout: Name the scheduler by something that outlives one machine says why, and how several --scheduler values fall back to one another.

That is the whole of it — but a membership flag is not optional, and it is the line people leave off. (--fleet-open is the one a build network that is already your boundary wants; --fleet-member is the narrower alternative, below.) Membership gates this node's compile verbs, so a worker with neither admits its own machine and nothing else: the scheduler leases it out, the client dials it, and the compile is refused not-a-member and run locally instead. The scheduler's counters stay flat and correct while that happens, because the lease was granted (#235; before it was fixed, neither flag could be given to a worker at all). The worker says which it is in its own startup line:

[INFO] compile node ready on 0.0.0.0:6674, advertising worker-01.internal:6674, 16 slot(s) as a … node, identifying 2 toolchain(s), every caller admitted
[INFO] compile node ready on 0.0.0.0:6674, … , this machine only -- give --fleet-member or --fleet-open to admit peers

Use --fleet-open where the build network is already your boundary. Where it is not, swap it for a repeated --fleet-member=dev-01.internal naming the machines whose clients may dispatch here, and everyone else stays refused. The list is matched by host: a client dials from an ephemeral source port, so there is no port for an endpoint to be compared against.

On a node running consensus, the cluster's members are admitted as well

The agreed member set adds to --fleet-member rather than replacing it — see membership at runtime. A cluster member is a peer; a client machine is not and never will be, so --fleet-member stays the way to admit a laptop or a CI runner on a clustered fleet just as on a node running no consensus. --fleet-open remains the answer where the build network is already your boundary.

The worker then surveys the machine at startup and serves every compiler it finds, naming each one and the layout it came from:

[INFO] found /usr/bin/g++ (usr)
[INFO] found /usr/bin/clang++ (usr)
[INFO] discovered 2 toolchain(s) on this machine; pass --toolchain to serve a narrower set

Add --toolchain=<compiler> to serve a narrower set — a build farm pinned to a curated toolchain — or --no-toolchain-discovery to stop the survey entirely. Naming any --toolchain pins the worker to exactly those.

A node started this way is also a cache, which is the part that surprises people: --listen-node defaults to 127.0.0.1:6674 and --cache-memory to a quarter of host RAM within [512m, 8g], so the machine gets a local tier without asking for one. Point it at the shared cache with --upstream and a local rebuild stops reaching the wire at all. --cache-memory=0 with no --cache-dir turns the tier off, which is what a machine that only compiles for others wants. The whole of it is on the node's own page.

On Linux the package ships a socket-activated unit; put the settings in /etc/fastcached/fastcache-compile-node.yaml — every key is one flag, spelled with underscores, and a worker needs cluster_dir: /var/lib/fastcache-node, the state directory the unit creates for its identity key — and enable the socket:

sudo systemctl enable --now fastcache-compile-node.socket

--advertise is the flag to get wrong

The scheduler hands your string to clients verbatim. A worker advertising 127.0.0.1 is leased and then never answers — and the symptom is a build that falls back to local compiles on every machine except the one running the worker.

The clients

export FASTCACHE_ADDR=build-cache.internal:6674
export FASTCACHE_SCHEDULER=scheduler.internal:6675
cmake -DCMAKE_CXX_COMPILER_LAUNCHER=fastcache-cc ...

FASTCACHE_ADDR is the cache — a fastcached, or the local node's own --listen-node. It defaults to 127.0.0.1:6674 when unset, which is where a node on this machine already answers, so a developer running a node needs only the scheduler line. FASTCACHE_ADDR= (set but empty) is the opt-out -- on a POSIX shell. PowerShell cannot express a set-but-empty variable: $env:FASTCACHE_ADDR = "" looks set from inside PowerShell and reaches the child process as unset, so the launcher takes its default and goes on caching. Use -DFASTCACHE_ADDR= or -DUSE_COMPILER_CACHE=OFF there; see the launcher's note.

Getting it wrong costs the cache and nothing else: a daemon that cannot be reached, or that answers and refuses, is reported (cache unavailable (…), counted under unavailable by --show-stats) and the translation unit is dispatched anyway. The FASTCACHE_ADDR= opt-out above is different, and turns both off — it is how a build says it wants no launcher at all.

FASTCACHE_SCHEDULER is the scheduler, which is the --listen-node port of some node running --serve-scheduler. Unset it and every miss compiles locally again — the behaviour without this feature, and the way to turn it off for one build.

FASTCACHE_DISPATCH_TIMEOUT is how long the client will wait for one remote compile, measured from the request to the last byte of the reply — the dial has its own FASTCACHE_CONNECT_TIMEOUT. It defaults to 10min and is deliberately not the cache's FASTCACHE_TIMEOUT: a worker writes nothing until the compiler has finished, so the client waits out the whole compile in a single read, and while the two shared one number every translation unit taking longer than ten seconds was abandoned and rebuilt locally — precisely the ones worth distributing (#223). Ten minutes because that is LeaseTable's own lease timeout: waiting longer means waiting on a lease the scheduler has already reclaimed.

FASTCACHE_DISPATCH_IDLE is the companion knob and the one that answers the other question: how long the worker may say nothing. It defaults to 30s. A worker writes a five-byte progress frame every few seconds while a compile is running, so the client measures silence rather than duration — which is what lets this be seconds while the total above stays minutes (#245). Setting it to 0s turns it off and restores one flat deadline. All four are durations — a whole number and a unit (ms, s, min, h, d); a bare number is not one and is ignored, so the launcher runs under the default.

Both are runtime settings: the launcher is one process per translation unit, so exporting a new value is all it takes and the next compile uses it. Nothing reloads and nothing restarts. Raise the total if a single translation unit legitimately compiles for longer than ten minutes; lowering it buys you nothing against a worker that has stopped, for the reason in the next box.

A worker that stops is noticed in seconds, not at the deadline

A flat deadline cannot tell a worker that is still compiling from one that has stopped, so it is not asked to. Two mechanisms answer the second question instead, and they cover different failures:

the worker detected by when
process exits, crashes, is killed the client's own parked read — FIN or RST immediately
host vanishes: powered off, unplugged, suspended, VPN dropped TCP keepalive on the compile dial ~16 s on Linux and macOS, ~30 s on Windows
host answers while the process makes no progress the missing progress frames — FASTCACHE_DISPATCH_IDLE ~30 s
compiler runs longer than anybody is prepared to wait FASTCACHE_DISPATCH_TIMEOUT at the deadline

In every one of them the client hands the lease back and compiles locally, so nothing is lost and the key is not pinned for the scheduler's lease timeout.

The Windows keepalive figure is not a typo: the OS fixes the probe count at 10 and offers no way to set it.

Row 3 is the one keepalive cannot see — the kernel answers every probe, so nothing below the protocol knows anything is wrong. The worker therefore says so itself: while a compile is running it writes a Status::Progress frame every five seconds, carrying nothing at all, and the client gives up after thirty seconds of silence. The launcher's own fall-back line (FASTCACHE_VERBOSE) then reads stopped reporting progress rather than ran out of budget, because "that machine has wedged" and "that compile was slow" are fixed in different places by different people. On the --show-stats axis both are tallied as unavailable, which is deliberate: that tally is bounded by cause and does not grow a row per socket condition.

This is a wire feature, not a client-side heuristic: it is why the 0xFC protocol version is 3, and a launcher or node from before it is refused with unsupported-version rather than being left to meet a frame it cannot parse.

What a client that runs out of budget costs the fleet

The worker notices the client has gone and skips writing the object back, so the transfer is not paid for. The compile itself still runs to completion, so that CPU is spent twice and fastcache_worker_jobs_completed_total still counts the job; fastcache_worker_jobs_abandoned_client_gone_total counts the delivery that was skipped.

FASTCACHE_TOKEN against a node is accepted and ignored

A node's scheduler, compile and cache ports serve no AUTH verb, so there is nothing for a credential to authenticate against. All three refuse it unknown-opcode, which the launcher steps over before carrying on unauthenticated, so leases, compiles and registrations all work normally.

The launcher says credential ignored for its cache exchanges. It does not say it for a lease, a compile, a release or a worker's registration: those paths carry the same fact back and drop it, so an operator who set a token there is told nothing at all.

Two of the three used to answer dispatch-not-permitted instead, which the launcher treats as fatal and returns in place of the answer to the request it actually sent — so every LEASE was declined and every compile happened locally, behind a green build (#340).

What a credential would mean here is still open (#198); see Security for what does protect a fleet today.

Confirming it works

With FASTCACHE_VERBOSE=1 the launcher says what happened to each translation unit:

fastcache-cc: HIT key=…                                          served from cache
fastcache-cc: DISPATCHED to worker-01.internal:6674 key=…        compiled remotely
fastcache-cc: not dispatched (rejected (no-worker)); compiling locally
fastcache-cc: cache unavailable (fetch exchange failed); compiling this translation unit anyway

The last of those is a cache problem, not a fleet one, and the line says so by not ending at the compiler: a DISPATCHED line normally follows it.

The scheduler is a compile node, so its /metrics endpoint is the node's: start it with --admin-listen and these count the outcomes.

Counter Rising means
fastcached_dispatch_leases_granted_total Work is being distributed.
fastcached_dispatch_leases_released_total Clients are reporting their jobs done. Granted minus released is what is outstanding; flat while granted climbs means clients are dying mid-job, or predate the verb.
fastcached_dispatch_leases_no_worker_total The fleet is misconfigured — workers are up but nobody matches.
fastcached_dispatch_leases_no_capacity_total The fleet is too small — full of your own build.
fastcached_dispatch_leases_withdrawn_total The fleet is unavailable — slots free on paper, machines busy elsewhere or out of scratch space.
fastcached_dispatch_leases_duplicate_total Duplicate-work suppression is doing its job. Not a problem.
fastcached_dispatch_worker_registrations_total Workers registering. A steady rise means heartbeats are not arriving.
fastcached_dispatch_worker_registrations_malformed_total A peer named its toolchain, endpoint or version in bytes that are not UTF-8 and was refused. Any rise names a machine that is not in the fleet.
fastcached_dispatch_worker_endpoint_mismatch_total A worker was admitted while advertising an endpoint whose host is not the address it connected from. Not a fault by itself — DNS names, a node dialling itself, NAT, a VPN and multi-homing all land here. See the endpoint a registration names is not verified.
fastcached_dispatch_workers_expired_total A machine stopped heartbeating and was dropped. Rising beside a rising registration count is a fleet whose heartbeats are not arriving, not one that is growing.
fastcached_dispatch_workers_withdrawn_total A machine re-surveyed, found it no longer serves a toolchain, and retired that registration itself rather than letting it time out. Never sum it with the expiry counter above: an expiry is a machine that stopped answering, this is one that said so, and adding them makes every routine toolchain upgrade read as a heartbeat failure.
fastcached_dispatch_leases_reclaimed_total A machine went away mid-job, or restarted, and the keys it was building were freed. Work nobody will report done — a build that lost part of its distribution, which is a different thing to fix from a fleet that is merely full.
fastcached_dispatch_leases_unauthorized_total A client handed back a lease token this cluster never signed. Never sum it with the unknown-lease refusals beside it: those name a lease this scheduler did issue and has since forgotten, while a rise here is a forged release — or, far more likely, a launcher predating signed leases, in which case it tracks a rollout and stops when the rollout finishes.
fastcached_dispatch_leases_released_late_total A client reported a job whose lease had already expired: a real compile outran the lease timeout. Read it as a fraction of released_total — a steady fraction means the lease bound is shorter than this site's slowest translation unit. Not a count of leases that expired: a client that never reports back at all reaches this nowhere, and is reclaimed or nothing.

The first three refusals are different operator problems and are deliberately counted apart: summing them hides a misconfiguration behind a busy fleet, and hides an unavailable fleet behind one that merely looks undersized.

Those count what the scheduler decided. Four more count what it refused before any verb reached it — a frame turned away at the wire, which until #494 answered a client and moved nothing, so a scheduler being probed and a scheduler nobody was talking to drew the same flat line.

Counter Rising means
fastcached_dispatch_frames_refused_unsupported_version_total A peer built against another release of the wire. During a rollout this tracks the rollout; afterwards it names a machine nobody upgraded.
fastcached_dispatch_frames_refused_unknown_opcode_total A frame naming a verb no build has. A scanner, or a client speaking something else entirely at this port.
fastcached_dispatch_frames_refused_not_permitted_total A frame naming a verb that exists and is served on a different port — typically a cache or compile request sent to the scheduler's address. Any rise names a client pointed at the wrong one.
fastcached_dispatch_frames_refused_truncated_total A frame whose header disagreed with what arrived: a framing or transport fault, before any verb was routed. Never sum it with a malformed_payload series — those are payloads that decoded wrong, and share only the wire code.

A non-member caller is not in this table: that refusal is decided one layer down, in SchedulerService, which deliberately counts it nowhere so a client retrying past a momentarily full surface cannot bury the credential series. Whether it should get a series of its own is #592.

On each worker, start it with --admin-listen and the same endpoint reports what that machine is doing:

Counter Rising means
fastcache_worker_jobs_started_total Jobs accepted.
fastcache_worker_jobs_completed_total Jobs that ran to an exit code — including a non-zero one, which is the client's answer rather than a worker failure.
fastcache_worker_compile_milliseconds_total Compile wall time. Divide by ..._jobs_completed_total for the mean; both are counters, so a rate over a window gives you the current one.
fastcache_worker_jobs_abandoned_client_gone_total The client had disconnected before the object could be written back, so it was not sent. The compile still counts in ..._jobs_completed_total — it ran and this machine paid for it, and only the delivery found nobody there. A rise is a client-side story: cancelled builds, Ctrl-C, a CI runner reclaimed mid-job. What it saves is the transfer; the CPU is spent either way until #661. A client that half-closes its write side after sending and still waits for its object is read as gone and counted here — nothing shipped in this project does that, but a third-party client speaking the protocol over a raw socket could.
fastcache_worker_jobs_refused_no_slot_total This worker is full. Pair it with the scheduler's ..._no_capacity_total.
fastcache_worker_jobs_refused_unknown_fingerprint_total Somebody is dispatching a toolchain this worker does not have.
fastcache_worker_jobs_refused_rejected_argument_total A command line carrying something that could name a file.
fastcache_worker_jobs_refused_scratch_unavailable_total The scratch disk is full or unwritable.
fastcache_worker_jobs_refused_spawn_failed_total The toolchain is configured but cannot be executed.
fastcache_worker_jobs_refused_compiler_unclassified_total The toolchain runs, and this build cannot tell which driver it is, so no command line can be built for it. The remedy is the --toolchain on that node, not a path. The node also says this once at survey time, so check its startup log before this counter.
fastcache_worker_jobs_refused_not_a_member_total A caller this worker does not admit tried to compile on it. Check your own fleet first: if this worker has no --fleet-member / --fleet-open, it admits its own machine and nothing else, and this counter is your clients being turned away one hop after a lease was granted. Once it names a policy, a rise here is somebody with no claim on the machine — check who can reach --listen-node.
fastcache_worker_jobs_refused_envelope_declared_too_large_total A request declared its payload expands past this worker's ceiling. Nothing honest does that by accident: a probe of the port, or a client with a larger ceiling than the worker.
fastcache_worker_jobs_refused_envelope_unsupported_codec_total A client compressed with a codec this worker was not built with. A packaging difference between two honest machines; each one cost a local compile.
fastcache_worker_jobs_refused_envelope_malformed_total A payload envelope that did not parse: a version skew, or something on the port that is not this protocol.
fastcache_worker_jobs_refused_envelope_corrupt_total Bytes that parsed and then did not expand to their declared size. The one refusal here that implicates the transport.
fastcache_worker_jobs_refused_lease_unauthorized_total The lease presented was not signed by any voter the worker's roster names. The one counter here that is unambiguously a security signal rather than a capacity or configuration one — or a launcher predating this lease format, which is told apart by whether the rise tracks a rollout.
fastcache_worker_jobs_refused_lease_signer_revoked_total The lease verifies — under a key the cluster has revoked. The removed machine itself, still leasing out work: it should read zero forever, and a rise names a machine somebody removed and nobody stopped.
fastcache_worker_jobs_refused_lease_wrong_cluster_total An authentic lease was signed by a different fleet. Never sum it with ..._unauthorized_total beside it: that one is a bad signature, this one is a good signature from somebody else. A rise here is a provisioning mistake — one identity key voting in two clusters, which is what copying a --cluster-dir to a second site produces — and the fix is a fresh state directory on one of them, not a firewall.
fastcache_worker_jobs_refused_lease_replayed_total An authentic, unexpired lease that this worker had already run. A lease authorizes exactly one compile — one grant per lease, presented once, with no retry — so nothing honest produces this and it should read zero forever. Any rise is somebody presenting a captured grant a second time. Do not sum it with ..._wrong_cluster_total either: they share a wire code and nothing else, and they send you to opposite places.
fastcache_worker_jobs_refused_lease_endpoint_mismatch_total An authentic lease named a different worker. Almost never a replay and almost always a worker registered under an address clients do not dial — a NAT, or a hostname where clients resolve an address.
fastcache_worker_jobs_refused_lease_expired_total An authentic lease had expired. A rise on one machine and nowhere else is that machine's clock, not the fleet's leases — which is why the check carries skew slack and why this is worth seeing per node.
fastcache_worker_jobs_refused_lease_no_roster_total The worker holds no roster to check any lease against yet: none its --voter-key voters endorse has arrived. A few at startup are ordinary; a rise that does not stop means no leader it reaches is endorsed by the keys it was given.
fastcache_worker_jobs_refused_lease_roster_expired_total The worker's roster was not re-certified within its lifetime, so it can no longer tell a live voter from a revoked one. It is cut off from the leader, or reaches only an ex-leader withholding newer rosters — fastcache_node_roster_expires_in_seconds reached 0 first.
fastcache_worker_scratch_roots_reclaimed_total This worker took over a scratch root left behind by a node that exited without cleaning up. The work itself is correct — a root is only reclaimed once its previous owner's exclusive claim is free, which the OS releases however that process died. A rise means nodes are dying rather than stopping, which is worth knowing and is visible nowhere else.
fastcache_worker_bytes_received_total / ..._returned_total Link volume, counted at the socket.

The refusals are split by reason for the same reason the scheduler's two are: a full worker and a misconfigured one are different problems with different fixes, and one number covering both tells you neither.

Retired series. fastcache_worker_jobs_refused_lease_stale_epoch_total no longer exists, as of #614. If a dashboard or alert of yours scrapes it, it will now find nothing — that is the change, not a fault. The refusal it counted can no longer happen: a worker adopts a scheduler term that went backwards instead of refusing it, because replay is closed by grants being spendable once. Point those alerts at fastcache_worker_jobs_refused_lease_replayed_total (somebody presenting a captured grant twice) and fastcache_worker_scheduler_term_regressions_total (a scheduler term that went backwards). The reasoning is under "A grant is spendable once" below. It was retired rather than left exported at zero because a series that reads zero for ever because its event is impossible looks exactly like one reading zero because the event has not happened, and only one of those means your fleet is healthy.

The ..._lease_* counters move only on a worker that checks leases: a consensus member, against the state it applies, or a worker holding a roster its --voter-key voters certified. A worker that checks nothing says so at startup rather than leaving these at zero and looking healthy — and one another machine could dial is refused outright (#282, #178).

Four of them — ..._unauthorized_total, ..._signer_revoked_total, ..._wrong_cluster_total and ..._replayed_total — share the wire code lease-unauthorized, because a client's answer to all four is the same, compile it locally, and a code it does not recognise would be worse than one it does. Your answer is different for each, which is why they are four series rather than one: a bad signature is a security question, a revoked signer is a machine somebody removed and nobody stopped, a good signature from another fleet is a provisioning one, and a grant presented twice is somebody replaying a credential.

The rest carry codes of their own — ..._endpoint_mismatch_total answers lease-endpoint-mismatch, ..._expired_total answers lease-expired, and ..._no_roster_total and ..._roster_expired_total both answer roster-expired, which says the worker can check nobody's lease right now rather than that this one is bad — so an alert written against lease-unauthorized will not see them. That is deliberate and it is LeaseRefusalTable in LeaseToken.hpp that decides it: a client can act differently on those two, and a code it can act on is worth minting.

A grant is spendable once, and the scheduler's term is only a diagnostic

A lease authorizes exactly one compile. The scheduler mints one grant per lease, the launcher presents it in exactly one request with no retry, and it hands the lease back to the scheduler rather than to the worker — so a grant arriving twice at one worker is a replay and never an honest client. Since #614 a worker enforces that: the second presentation is refused and fastcache_worker_jobs_refused_lease_replayed_total moves. Before it, a captured grant could be replayed at its worker until it expired.

A grant refused for anything else — a wrong toolchain, a wrong endpoint, an expiry — is not consumed, so a client whose job was declined for a configuration mistake still holds a usable lease.

The one thing this does not cover is a worker restart, which forgets what it has spent. A grant captured and withheld across a restart is usable once afterwards, for whatever is left of its expiry. That is the same window the lease expiry has always bounded.

The scheduler term inside a grant no longer decides anything. It is carried, it is signed, and a worker adopts it — but it is a diagnostic rather than a check. Until

614 a worker refused any grant naming a term below the one it had learned, and three

ordinary operator actions break that: wiping a scheduler's Raft directory, re-bootstrapping the cluster, or turning consensus off all drop it to a lower term truthfully. Every worker then refused every grant until its process was restarted, with nothing but a climbing counter to say why — which read like an election storm.

Worse, the rule was backwards in exactly that case. With a worker at term 7 and a scheduler honestly reset to term 0, it refused the fresh grant and accepted a token captured under term 7. The check written to stop replay refused every legitimate grant and admitted the replayed one. Grants being spendable once is what actually answers that question, so the term check is gone and fastcache_worker_jobs_refused_lease_stale_epoch_total was retired — the refusal it named can no longer happen, and a series reading zero because its event is impossible looks exactly like one reading zero because the event did not happen.

What you watch instead is fastcache_worker_scheduler_term_regressions_total, which counts a worker adopting a term that went backwards, alongside one WARN line naming both terms.

It does not mean somebody reset a cluster, and you have to read the rate to know which it is. Two things produce a lower term and a worker cannot tell them apart from the grant:

  • A grant that arrived late. A client asks for a lease, then preprocesses and uploads a translation unit — seconds — and an election inside that window puts its term-N grant behind somebody else's term-N+1 grant at the same worker. Completely ordinary. Expect occasional single counts that line up with leadership changes.
  • A scheduler that lost its state — Raft directory wiped, cluster re-bootstrapped, consensus turned off. This one repeats, because every grant that scheduler mints now carries the lower term.

So: occasional counts tracking elections are the first and need nothing from you. A sustained rise, especially across the fleet at once and against no election, is the second.

The trade, stated plainly because it is real: a grant captured before it ever reached its worker used to stop being good at the next election, and now stops being good at its expiry. That is the weaker of the two bounds. It could not be kept — a scheduler that was legitimately reset and a replayed older grant look identical inside a token, and refusing them together is what stopped the fleet.

The worker also reports what the machine is — fastcache_node_logical_cores, fastcache_node_memory_total_bytes, fastcache_node_disk_capacity_bytes, fastcache_node_disk_free_bytes, fastcache_node_slots_configured and fastcache_node_slots_busy. Those are gauges: "is this node pulling its weight" is not answerable without knowing how big it is. Beside them is what the machine is doing — fastcache_node_memory_available_bytes, and the CPU as two tick counters, fastcache_node_cpu_busy_ticks_total over fastcache_node_cpu_ticks_total, whose rates divide into the busy share.

And what its cache tier is doing — fastcache_node_cache_hits_total and ..._misses_total for the tier itself, ..._upstream_hits_total for what the shared cache answered after a local miss, and ..._fill_failures_total, ..._store_failures_total, ..._upstream_stores_total and ..._upstream_store_failures_total for the writes. A high upstream-hit rate against a low local one means the tier is too small for this machine's working set, which is a different problem from a fleet that is missing a lot.

Beside them, what it is holding — fastcached_items, fastcached_bytes_used and fastcached_bytes_limit for the cache as a whole, plus a fastcached_tier_*{tier="memory"|"disk"} set for the split. A node whose cache is effectively empty sends every rebuild to the wire; one whose evictions climb while its hit rate falls has a budget too small for the tree it builds. Both are invisible from the scheduler's own counters, which only see the work that reached it. See the node's own page for why these must not be summed across tiers, and why a node running no tier reports the series as absent rather than as zero.

The same figures reach the leader, on each node's REGISTER and HEARTBEAT, so they are readable from one place rather than by scraping every machine. Note that a node started with two --toolchain flags is two registry entries against one machine and one cache: both carry the same figures, and anything summing them counts that cache twice.

The whole fleet on one page

--dashboard, on a leader that already has --admin-listen and --serve-scheduler, serves /fleet and /fleet.json — every member's hostname, endpoint, software version, capacity and cache, plus charts of the last 24 hours or 7 days:

fastcache-compile-node ... \
    --serve-scheduler --listen-node=6675 --fleet-member=10.0.0.2 \
    --admin-listen=6677 \
    --dashboard --dashboard-token-file=/etc/fastcached/dashboard.token

Only the leader answers it in full; anyone else replies 503 naming the leader, because a follower's registry holds whatever registered against it rather than the fleet. The credential is required off loopback and is deliberately its own file rather than --requirepass. /metrics and /healthz stay outside it, so a scraper or a probe is unaffected — and /metrics remains the source of truth for anything you alert on. The full account, including what each column means and why a value nobody reported renders as – rather than 0, is under looking at the whole fleet.

"No worker matches this toolchain"

The commonest setup failure. A worker is matched only if its toolchain fingerprint is byte-identical to the client's, so ask both ends:

fastcache-cc --print-toolchain-fingerprint /usr/bin/g++    # on a client
journalctl -u fastcache-compile-node | grep serving        # on a worker

If they differ, the machines really do have different toolchains — different patch releases, different SDKs, a vendored header that is not the same.

The fingerprint is a digest of the compiler's version banner and the include tree that belongs to it. That is what lets two machines with the same toolchain at different install prefixes match, while two machines whose headers differ do not. It cannot be loosened: an over-strict match costs one local compile, an over-loose one produces a silently wrong object that is then stored under a key every other machine fetches.

Belongs to it is the operative phrase on Windows. cl carries the VC\Tools\MSVC\<version> toolset it lives inside and the newest Windows SDK; clang-cl carries only the resource directory it names when asked (-print-resource-dir), because it borrows the VC toolset and the SDK rather than owning them — so two clang-cl machines with different SDKs installed still match. Neither derives its answer from INCLUDE, which is set per developer command prompt and never inherited by a service — so a worker installed as a service matches the launchers that talk to it.

The fingerprint and the cache key are not the same string

They answer different questions, so one of them carries the compiler's target and the other deliberately does not:

Decides Carries the target?
Toolchain fingerprint which worker may serve a client no
Object cache key which object may be served yes

A driver's code generation is not a function of the driver alone. clang-cl detects the MSVC installation beside it and sets -fms-compatibility-version from that, and clang's Microsoft C++ ABI gates version-specific code generation on the value — so the same clang-cl.exe in a developer prompt and under a service generates differently. Stock g++ on x86_64 and aarch64 is the same shape with no MSVC anywhere near it: one --version banner, two code generators.

The key therefore folds the target in, so those cases stop sharing entries. The fingerprint must not, or a developer-prompt launcher would stop matching a service-run worker — the mismatch that keeps a fleet answering no-worker for reasons nobody can see. Dispatch closes the gap on the line instead: a dispatched compile states --target=<triple> ahead of the build's own arguments, so the worker generates for the client's target rather than re-deriving one from its own machine, while a --target= or -m32 the build states itself still wins.

Only a driver that can be told a target is pinned this way. gcc is fixed-target and rejects the flag, so its target is identified for the key and never stated on a dispatched line; cl is neither, because which code generator runs is decided by which cl.exe you invoked and no command line can restate it.

The practical consequence is a hit-rate one: after this change two machines whose compilers report one banner but generate for different targets no longer share cache entries. That is a correction — they were sharing objects they should not have — but it looks like a cold cache the first time a fleet upgrades past it.

Sizing and operations

  • Misses are bursty. With a warm shared cache, hit rates run 90 %+, so a fleet is idle most of the time and the value is concentrated in the first build of a commit, developer branches and toolchain bumps. Socket activation exists for exactly this shape.
  • --slots defaults to derived: hardware threads, clamped by what the memory supports, less what --node-class reserves (two cores on a workstation, none on a dedicated node). A number you give it overrides all three. Whatever it resolves to is advertised and enforced locally from one calculation, so a worker cannot end up fuller and slower than the scheduler believes at the same moment. --slots=0 is not a count: it runs no worker at all (see The scheduler above). See the worker's own page.
  • --node-class defaults to workstation, which is the safe answer rather than the common one: a node nobody classified is somebody's desktop until proven otherwise. Set --node-class dedicated on build servers, or the fleet quietly runs two cores short on every one of them.
  • The scheduler picks by free slots, not by fewest running jobs. Absolute counts make a 64-slot server running 8 jobs look busier than a 4-slot laptop running 2, which sends work to the smallest machines first.
  • Workers withdraw capacity while their machines are busy elsewhere. Each heartbeat carries host CPU, available memory and free scratch space; the fleet's own jobs are subtracted, so what is left is load that belongs to somebody else. A worker whose scratch filesystem fills up stops being picked entirely, and starts again when it drains.
  • Three lease refusals, never summed. no-worker means a fingerprint nobody serves, no-capacity means the fleet is too small, and withdrawn means the machines are there and unavailable. Each has its own fastcached_dispatch_leases_*_total, because the fixes are different and folding withdrawn into no-capacity sends an operator to buy hardware they already own.
  • A node's own cache tier is subtracted from what it can compile. Capacity is one job per gigabyte of RAM, and a resident cache is memory that will not yield — so a 64-thread host with 32 GiB holding 8 GiB of cache offers 24 slots, not 32. What is subtracted is what the tier actually holds: a node with --cache-memory=0 and no --cache-dir reserves nothing and offers the whole machine, and so does one whose cache never started.
  • Workers can come and go. A worker heartbeats every 20 s and is dropped after 90 s without one; a client that leases one in the gap finds it unreachable and compiles locally.

Security

Two things protect a fleet: who a node admits, and whether the job was granted.

Every machine that joins the fleet proves which one it is. A worker proves its own identity key on every connection to its scheduler, and every frame after that proof is sealed under a key only the two ends hold -- so something on the path can neither speak for a worker nor slip a verb into a connection the worker's proof admitted. A machine the cluster forgets with --cluster-forget is refused on its key, from any address. See A node proves which machine it is.

A node is closed by default. Its compile port and its scheduler admit this machine and --fleet-member peers only (or every caller, once you say --fleet-open); once a cluster exists the agreed member set is added to that list rather than replacing it, and both surfaces follow the union — so a host stops being admitted only when the operator stops listing it. That matters most for the compile port, which binds 0.0.0.0 because peers have to dial it — anybody who could route to it would otherwise have your machine run their compiler on source they chose.

Its own cache tier admits this machine and nothing else, whatever those lists say and whatever it is bound to (#287). A peer that may spend this machine's CPU is not thereby entitled to read everything it has ever compiled, and the tier is exactly that. A cache several machines share is a fastcached behind --upstream.

Membership alone was never enough, because admitted to the fleet is not granted this compile: an admitted machine could spend any worker's CPU without ever asking the scheduler for a slot. So a lease grant is a signed capability. The scheduler signs the granted worker's endpoint, the toolchain, the object key and an expiry with its own Ed25519 identity key, and the worker checks that signature — against the cluster's roster of voters, unrevoked — before it decompresses anything (#281, #282, #178). The endpoint is inside the signature, so a token captured on the way to one machine cannot be replayed against the rest of the fleet; and a signature is one machine's, so a scheduler the cluster removes stops being able to lease anything out the moment its workers hold the roster that revokes it — or, if one withholds that roster, once the roster they hold lapses.

Give every worker a --cluster-dir, and a way to check a lease

Every worker names a scheduler, so every worker will not start without --cluster-dir: it has no identity to prove, and every verb it would join the fleet with would be refused. And a worker another machine could dial — anything with --fleet-member or --fleet-open on a bind that is not loopback — will not start with no way to check a lease: it runs consensus, keeps a roster in its --cluster-dir, or names the voters with --voter-key. Both refusals are deliberate and both are startup ones, not per-request fallbacks: a worker that quietly skipped the check would serve whoever reached its port while every refusal counter read zero, which is a fleet that looks healthy from both ends.

A node nothing else can dial — the ordinary one-machine install — needs no roster and logs a warning saying the check is off. A process on that host already has that host's compiler.

Upgrade the fleet together. A lease format and a roster are new in the same release, and a worker and a scheduler from either side of it cannot agree on either; nor can consensus members from either side of the authenticated consensus wire.

A node checks no inbound credential

None of a node's three framed surfaces — scheduler, compile port, cache tier — serves the AUTH verb, so there is nothing for a credential to authenticate against. --requirepass on a node is only the secret it presents when it dials somebody else, and it works in exactly one direction: against a fastcached named by --upstream, which does serve AUTH.

Setting a credential no longer breaks anything. All three surfaces refuse AUTH with unknown-opcode, which is the one refusal fastcache-cc steps over before carrying on unauthenticated — the right outcome against a surface with no credential to check. So a client with FASTCACHE_TOKEN set leases, compiles, registers, reads and writes normally. On its cache exchanges it also reports credential ignored, so the fact is not silent there; on a lease, a compile, a release or a registration it is — those callers receive the same flag and discard it.

That code is a wire contract between binaries that do not link each other, and the three surfaces once answered it three different ways. Two of them said dispatch-not-permitted, which the launcher treats as fatal and returns in place of the answer to the request actually sent — #283 corrected the cache tier and #340 the other two. Until then, a FASTCACHE_TOKEN client had every LEASE declined behind a green build, and a --requirepass worker never joined the fleet at all.

What is still open is the credential itself, and it is a gap rather than a design: what an inbound credential should be spelled, and what AUTH should mean against a node that has none configured, are the questions #976 is open on. (#198 and #289 asked the worker's and the scheduler's halves separately and are both closed; #976 is the live ticket, and the questions did not change.) Until it closes, a fleet's boundary is network reachability plus membership, not a secret — so keep these surfaces on a network you would run a compiler for.

Beyond the lease and consensus — whose every connection proves each member's own identity key, see Raft peer authentication — the fleet's own traffic is unauthenticated, so treat its boundary as network reachability plus membership and size the network accordingly.

So: keep --serve-scheduler off any network you would not run a compiler for, and put mTLS in front of every port for anything beyond a trusted build network. The two remaining credentials in this system are real and unaffected — --dashboard-token-file guards the fleet page and the live-stats fleet stream, and fastcached's own --requirepass guards the shared cache.

Also worth knowing: the node's inter-node gate matches on the peer's source address alone (#180), so a build LAN where addresses can be spoofed is not a boundary this can hold.

Limits worth knowing before you adopt it

  • Preprocessing does not distribute (see step 1). The ceiling is ~10–40×.
  • -g embeds the worker's scratch path in DWARF. Use -fdebug-prefix-map, which this launcher recognises and folds into the cache key, so two checkouts still share. Not -ffile-prefix-map or -fmacro-prefix-map: they are deliberately not recognised, and passing either costs cross-checkout sharing for that translation unit — they rewrite __FILE__ into the preprocessed text the key hashes, so the launcher cannot relativize them without hashing text the real compile never produced. See the launcher's own account.
  • Diagnostics from a failed remote compile are not shown. A worker reporting a non-zero exit is retried locally and the local result is what you see — which also regenerates the diagnostics with correct line numbers.
  • --install-service is Windows and macOS. It registers an SCM service or a launchd job. On Linux the packages ship a socket-activated systemd unit instead, so there is nothing for the flag to do and it reports as much. A worker that needs --requirepass cannot be registered on any platform — a supervisor records launch arguments where every local account can read them — so on Linux that token goes in the worker's configuration file as requirepass:, and the registration carries --config=<path> rather than the token.
  • On Windows a dispatched object is not byte-identical to a local one — the code in it is. Every MSVC-family driver stamps the clock into the COFF header, and cl also records the absolute path of the object file (in .debug$S) and a hash of the source file it opened (in .chks64), with no debug flag asked for. A worker compiles its own scratch file, so those three differ; every section carrying code or data is byte-identical, which is what the end-to-end test asserts. If your build compares object bytes across machines, compare sections.

  • Some compiles are never distributed, by design. A C++ module interface unit and any compile that writes a precompiled header produce a second artefact beside the object, and only the object travels — so those are compiled locally and are not cached either. So is a command line that names its input language itself (/TP, -x c++), because the launcher has to state the language of the preprocessed text it sends and would otherwise silently override yours. Run with FASTCACHE_VERBOSE=1 to see which of these applied.

  • A node's --requirepass cannot reach another node (see Security). Until #198 closes, the fleet's own traffic is unauthenticated and setting a token on it is worse than leaving it unset.

Reference