Distributed compilation¶

A cache makes the second build of a commit fast. It can do nothing about the first one: on a miss, pace-cc runs the real compiler locally and stores the result.

Distributed compilation absorbs those misses. Machines that would otherwise sit idle register as workers, and a client that misses the cache hands the translation unit to one of them instead of compiling it itself.

It is opt-in at both ends and cannot break a build: every refusal, every unreachable worker, every mismatch falls back to the local compile that would have happened anyway.

It also does not depend on the cache being up: the setup below deliberately puts the shared cache and the scheduler on different machines, and a cache that is unreachable or that refuses this client costs the lookup and the store, nothing more. Why that is worth stating is in How it works.

This page is how to set it up and run it. If you want the model first — what the pieces are and how one compile flows through them — read How it works, which is a shorter read and makes everything below easier to place.

Architecture¶

Three roles across two programs, with the shared cache behind them. Scheduler and worker are the same binary — pace-node — because being the one that hands out capacity is a role a node takes, not a different program.

Role	Program	What it does
Client	`pace-cc`	Fronts each compile. Checks the cache, and on a miss asks for a worker.
Scheduler	`pace-node`	Tracks the fleet's workers and hands one out. Exactly one node at a time: whichever leads the cluster.
Worker	`pace-node`	Compiles what it is sent. Also holds a cache tier of its own, unless you turn it off.
Shared cache	`pace-kv`	Optional, and not the scheduler. Where objects end up so other machines get them.

   ┌──────────────┐          ┌───────────────────────────────┐
   │ pace-cc │          │ pace-node        │
   │  (client)    │          │  (this one leads: scheduler)  │
   └──────┬───────┘          └───────────────┬───────────────┘
          │                                  │
          │  1. FETCH ──────────────────────►│ :6674  its cache tier,
          │     ◄──────────── hit: done      │        reading through to
          │                                  │        pace-kv upstream
          │  2. miss: "give me a worker" ───►│ :6675  scheduler
          │     ◄──────── endpoint + lease   │
          │                                  ▲
          │                                  │ register + heartbeat
          │                   ┌──────────────┴─────────┐
          │  3. compile this ►│ pace-node │ :6674
          │     ◄──── object  │      (a worker)        │
          │                   └────────────────────────┘
          │
          └─ 4. STORE the object ─► :6674  ← the CLIENT stores, not the worker

pace-kv is behind all of it as the fleet-wide store a node's --upstream points at. It serves the cache verbs and nothing else: a scheduling verb arriving at one of its listeners is refused with a typed dispatch-not-permitted naming where the scheduler went.

The picture above is the compile path. For the whole topology — consensus, discovery and the dashboard history alongside it — plus a directional table of every connection and what to open on a firewall, see Cluster communication.

Why the scheduler knows what is already being compiled¶

A LEASE request names the object key, not just the toolchain — so the one node handing out capacity sees the whole fleet's misses as they arrive, which is something neither distcc nor sccache-dist is positioned to know.

When a header changes and sixty parallel clients miss the same key — the ordinary shape of a miss on a shared cache, not an exotic one — the scheduler dispatches one job and tells the other fifty-nine to compile locally. Without that, sixty machines compile the same translation unit and fifty-nine of the results are thrown away.

Why the client stores the result, never the worker¶

A STORE is trusted today because whoever stored it compiled it themselves — they could only poison their own key space with something they would have gotten anyway. If workers stored, one rogue worker could poison the shared cache for everyone.

Routing the result back through the client keeps that model exactly as it was. Workers are given no cache credentials at all.

How it works¶

1. The client preprocesses, and that is the part that cannot be distributed¶

pace-cc must preprocess to compute the cache key before it can know whether there is a miss. That work is unavoidable and always local — roughly 45 ms against compiles of 300 ms–2 s, which is where the ceiling of about 10–40× comes from. Distribution is not linear scaling and should not be sized as if it were.

2. On a miss, the client asks for a worker¶

The request names a toolchain fingerprint and the object key. The scheduler matches the fingerprint byte-identically and picks the matching worker with the most free slots — ties broken by utilization, so between two machines with four slots free the one with proportionally more of itself left takes the job.

Free slots rather than fewest running jobs, because absolute counts treat every machine as an identical box: a 64-slot server running 8 jobs looks busier than a 4-slot laptop running 2, when the server has 56 slots free and the laptop has none. Across a fleet of mixed machines — the ordinary case — counting jobs sends work to the smallest machines first and leaves the big ones idle.

The scheduler refuses rather than queues — no-worker, no-capacity, withdrawn, all-excluded, already-in-flight — because the client is holding the source and can simply compile it. Queueing would buy latency and nothing else. Each names a different operator problem, which is why they are counted apart; see the sizing notes.

3. The client sends preprocessed text, not files¶

The worker receives the translation unit already preprocessed, so there are no headers to ship and no sysroot to replicate. This is distcc's non-pump model, i.e. the one that works.

One subtlety with real consequences: the text sent to a worker is not the text the cache key hashed. The key's copy has #line markers suppressed so no checkout path can reach the key; a compiler needs those markers to know which lines came from a system header. Without them every warning inside libc++ or the CRT is re-reported against your own file, and under -Werror that is a failed compile. So the client preprocesses a second time for dispatch.

4. The worker compiles it — with a compiler it chose¶

The job names a fingerprint, never a program. The worker maps that fingerprint to a compiler it serves and refuses one it does not have.

Which compilers those are is a fact the worker establishes for itself: it surveys the machine at startup, so a package install is the whole setup. --toolchain narrows that set rather than supplying it.

This is the difference between a build accelerator and a remote shell, and it is why there is still deliberately no default compiler. "No default" and "no discovery" are different claims — a default is how a job ends up running against something nobody chose, while which compilers a machine holds is simply a fact.

The worker also re-checks the argument list, because the client's check protects an honest client from dispatching something that would not work, while the worker's protects it from a client that is not honest.

5. The client writes the object, reproduces the dependency record, and stores¶

The worker compiled preprocessed text, which has no #include left in it, so it reports no dependencies. The client already knows them — its own preprocess pass opened every one — so it writes the depfile (or /showIncludes notes) itself. Skipping that would leave the build with no header dependencies for that translation unit, so it would stop rebuilding when those headers change.

A dispatched compile is then shaped to look exactly like a local one, so the STORE, the manifest and the statistics all take one path.

6. And the client checks that the reply belongs to its request¶

Every step above is upstream of the reply. The cache key covers the inputs, the fingerprint covers the toolchain, the lease covers the authorization — and none of them says anything about whether this object answers this job. So the worker digests what it actually compiled and sends that digest back, and the client recomputes it from what it asked for and refuses a reply that does not match, before the object is even decompressed.

Getting that wrong in the other direction is the worst failure this system has: a crossed reply accepted as an answer is a wrong object under a correct key, which succeeds, gets stored, and is then served to every other machine that fetches that key. There is nothing else in the pipeline that would notice. The refusal is therefore a refusal — the translation unit is compiled locally and the launcher prints a line naming the worker, whether or not PACE_VERBOSE is set.

It is integrity against accident — a connection reused wrongly, a scratch collision, a worker answering out of order — not against a hostile worker: the digest is unkeyed, so a worker that can return a wrong object can return a wrong digest as well.

What it is NOT left to the digest is which machine answered. A worker's address can come to belong to another machine between its heartbeat and your compile — a VPN reconnect is enough — and that machine could return a perfectly formed object. So the grant names the identity key the worker proved when it registered, the worker signs its reply under that key, and the client checks the signature before it stores anything. A reply signed by anybody else, or by nobody, is refused the same way: the translation unit compiles locally, and --show-stats counts it as UNAUTHENTICATED.

Setting it up¶

The scheduler¶

The scheduler is a compile node, not the cache. Pick one machine and start it:

pace-node \
    --listen-node=0.0.0.0:6675 \
    --raft-self=scheduler.internal \
    --advertise=scheduler.internal:6675

A scheduler is a cluster, even of one (#178). It signs every lease with its own identity key, and its workers check that signature against the cluster's voters and their keys as their own consensus applied them, so it holds replicated state and runs consensus to keep it. No flag asks for either: its first start mints a cluster of one into its state directory, which it leads, and whose mode serves the scheduler. Consensus is on by default (--listen-raft defaults to 6680), and --raft-self states the host another member would dial it at (127.0.0.1 when none ever will). Both names above are optional pins: left out, the node is dialled at the address this machine routes from and advertises that address on its --listen-node port, re-deriving both when the machine changes network. A node whose consensus is turned off with an empty --listen-raft= serves no scheduler and joins no fleet, so it may run only a cache tier (--slots=0): one running a worker is refused at startup, since its worker would have nowhere to register. Its identity is minted into its state directory on first start, and the same command line with --print-identity added prints it without serving. Its own worker registers with the scheduler it serves, on this machine: no flag names where, since a node's formation record says that.

There is no shared secret to provision: the pre-shared --cluster-key-file a fleet used to copy to every member is gone (#178). Every machine proves its own identity key instead -- a member to the other members, a worker to its scheduler -- and removing one machine is revoking that one key. What that buys is under Security.

It used to be pace-kv --listen-dispatch=..., and that flag is gone rather than deprecated. The two jobs have opposite deployment shapes: a cache is shared infrastructure somebody operates, while handing out capacity is a decision only one node may make at a time — and nothing in the cache daemon can establish which node that is. So scheduling moved to where cluster leadership lives.

A scheduling verb arriving at a pace-kv listener is refused with a typed error naming where the scheduler went, so a client configured for the old layout tells you what to fix instead of failing mysteriously.

No --toolchain is needed: the node surveys the machine at startup and serves what it finds. A scheduler is therefore also a worker unless told otherwise — every node is a peer, and being the one that schedules is a role rather than a different program.

To keep a machine out of the work — a small always-on box, a VM whose cores belong to something else — give it --slots=0. It then runs no worker at all: it surveys no compilers, claims no scratch directory, registers nothing and is never sent a compile, so a setting only a worker reads — --toolchain or --node-class, say — refuses to start it, by name. What it can still run is the scheduler, consensus and a cache tier; a node running none of those is refused too. Until #1440 its /metrics and history still read 0 slots.

pace-node --slots=0 \
    --listen-node=0.0.0.0:6674 ...

Such a machine shows among the cluster's members when it runs consensus, and not among the fleet page's machines, which are built from worker registrations; its own history is not handed to a leader either (#1440).

Once several nodes schedule, exactly one of them may at a time, which is what the same consensus decides once it has more than one member: one machine founds the cluster, every other one joins it as a learner and is admitted by its identity key. See a cluster, and who leads it.

Who may use the fleet¶

A node admits a machine by its key, never by its address. A worker proves the identity key in its --cluster-dir; a machine that only asks — a laptop, a CI runner — presents a machine ticket its own node mints for each exchange, signed with that node's key. Either way the machine's key is admitted to the cluster once, and from then on every node admits it from whatever address it dials — which is what makes a machine on a VPN or DHCP usable at all. It is not only the worker list: every scheduler verb is gated, LEASE included, so a machine whose key the cluster does not hold is refused a lease and compiles locally. Admit the machines that ask as well as the machines that answer: see a machine that only asks.

--fleet-open admits every caller, for one machine or a network that is already your boundary. A node with neither a roster nor --fleet-open refuses every caller but its own machine, which is the right default and not a working configuration; its /metrics shows the refusals.

Membership gates two of a node's surfaces — its scheduler and its compile port — and this machine is always a member of its own fleet, with nothing to present. What membership pays for on a node is CPU time, and those two are where it is spent.

It does not gate the node's own cache tier, which serves this machine and nothing else however either list reads (#287): that tier is a developer's whole build output, and contributing capacity does not entitle another machine to read it. Nor does it gate the shared pace-kv, where a non-member reads and writes objects exactly as before — that one is infrastructure somebody operates, and it is what a cache several machines share looks like.

Because it gates the compile port, every worker needs a roster or --fleet-open, not only the scheduler. A worker with neither admits its own machine and refuses the network, which is the right default for a developer's laptop and cannot receive a dispatched compile. A worker in a fleet has the roster already: it runs consensus, as every serving node does, and the state it applies is the roster it checks leases and admission against.

The workers¶

On each machine that should take work:

pace-node \
    --fleet-seed=scheduler.internal:6675 \
    --listen-node=0.0.0.0:6674 \
    --advertise=worker-01.internal:6674 \
    --fleet-open \
    --cluster-dir=/var/lib/pace-node

A worker joins the scheduler's fleet as a LEARNER, and that is the only way a machine joins. Its first start founds a cluster of one in --cluster-dir, minting its identity key there; it then finds the scheduler's fleet -- by the LAN beacon, or at --fleet-seed where no beacon reaches -- and asks to be admitted under that key. An operator on the scheduler compares the key and approves it:

# on the scheduler: what is waiting, each row with the key to compare and the line to approve it
pace-node --enroll-list
pace-node --enroll-approve=<id>@<key>

# or, while a known batch of machines is being added, admit whoever asks for a while
pace-node --enroll-auto-approve=30min

The approved machine becomes a learner of the fleet: it applies the fleet's replicated state, counts towards no quorum, and its worker registers with the fleet's scheduler, which its formation record names. No flag says where the scheduler is: --scheduler names only where a one-shot verb like the ones above is sent (this machine's own node when unset, which follows the fleet to its leader), and a node that serves is refused it.

A learner checks every lease against the state it applies. Every lease is signed by the scheduler that issued it, with that machine's own identity key, and the worker checks the signature against the voters its fleet's replicated state records -- so there is no key to type, and a voter the fleet revokes stops verifying the moment the learner applies the revocation. A worker another machine can reach will not start with no way to check a lease; one reachable only from its own machine needs none.

An address still admits a client -- a developer's pace-cc asking for a lease needs no identity -- but never a machine joining the fleet.

--listen-node and --advertise may both be left alone on a worker that serves a fleet. --listen-node binds every interface on port 6674 by default, and --advertise defaults to auto: the address this machine routes from, on that port, re-derived when an interface or address changes, so a laptop that moves between networks keeps taking work. Name --advertise only where clients have to dial something else — a DNS name, or an address in front of a NAT — and prefer a name to a literal address, which stops the node following the network.

--fleet-seed names a DNS name or a VIP, never a scheduler's literal address, where it is needed at all: every worker carries that value for as long as it is installed, so decide it before the rollout.

That is the whole of it — but who the worker admits is not optional, and it is the part people leave off. Membership gates this node's compile verbs, so a worker that admits its own machine and nothing else is leased out, dialled, and the compile is refused not-a-member and run locally instead. The scheduler's counters stay flat and correct while that happens, because the lease was granted (#235). The worker says what it admits in its own startup line:

[INFO] compile node ready on 0.0.0.0:6674, advertising worker-01.internal:6674, 16 slot(s) as a … node, identifying 2 toolchain(s), admits this machine, and machines the roster admits by key or ticket, and every other caller (--fleet-open)

--fleet-open is for a build network that is already your boundary. Where it is not, drop it: the worker's roster — the fleet's state, which it applies — admits exactly the machines the cluster admitted, by the key each proves or the ticket each presents, and everyone else stays refused, whatever address they dial from. A client machine is admitted by its key too, as a learner: see a machine that only asks.

The worker then surveys the machine at startup and serves every compiler it finds, naming each one and the layout it came from:

[INFO] found /usr/bin/g++ (usr)
[INFO] found /usr/bin/clang++ (usr)
[INFO] discovered 2 toolchain(s) on this machine; pass --toolchain to serve a narrower set

Add --toolchain=<compiler> to serve a narrower set — a build farm pinned to a curated toolchain — or --no-toolchain-discovery to stop the survey entirely. Naming any --toolchain pins the worker to exactly those.

A node started this way is also a cache, which is the part that surprises people: --listen-node defaults to 127.0.0.1:6674 and --cache-memory to a quarter of host RAM within [512m, 8g], so the machine gets a local tier without asking for one. Point it at the shared cache with --upstream and a local rebuild stops reaching the wire at all. --cache-memory=0 with no --cache-dir turns the tier off, which is what a machine that only compiles for others wants. The whole of it is on the node's own page.

On Linux the package ships a socket-activated unit; put the settings in /etc/pace/pace-node.yaml — every key is one flag, spelled with underscores, and a worker needs cluster_dir: /var/lib/pace-node, the state directory the unit creates for its identity key — and enable the socket:

sudo systemctl enable --now pace-node.socket

--advertise is the flag to get wrong

The scheduler hands your string to clients verbatim. A worker advertising 127.0.0.1 is leased and then never answers — and the symptom is a build that falls back to local compiles on every machine except the one running the worker.

The clients¶

export PACE_ADDR=build-cache.internal:6674
export PACE_SCHEDULER=scheduler.internal:6675
cmake -DCMAKE_CXX_COMPILER_LAUNCHER=pace-cc ...

PACE_ADDR is the cache — a pace-kv, or the local node's own --listen-node. It defaults to 127.0.0.1:6674 when unset, which is where a node on this machine already answers, so a developer running a node needs only the scheduler line. PACE_ADDR= (set but empty) is the opt-out -- on a POSIX shell. PowerShell cannot express a set-but-empty variable: $env:PACE_ADDR = "" looks set from inside PowerShell and reaches the child process as unset, so the launcher takes its default and goes on caching. Use -DPACE_ADDR= or -DUSE_COMPILER_CACHE=OFF there; see the launcher's note.

Getting it wrong costs the cache and nothing else: a daemon that cannot be reached, or that answers and refuses, is reported (cache unavailable (…), counted under unavailable by --show-stats) and the translation unit is dispatched anyway. The PACE_ADDR= opt-out above is different, and turns both off — it is how a build says it wants no launcher at all.

PACE_SCHEDULER is the scheduler, which is the --listen-node port of some node that serves it. Once this machine runs a node that has joined the fleet, 127.0.0.1:6674 — this machine's own node — is the value to use: a node that runs no scheduler answers with the leader's address, and the launcher follows it there, wherever the leader moves. Unset it and every miss compiles locally again — the behaviour without this feature, and the way to turn it off for one build.

PACE_DISPATCH_TIMEOUT is how long the client will wait for one remote compile, measured from the request to the last byte of the reply — the dial has its own PACE_CONNECT_TIMEOUT. It defaults to 10min and is deliberately not the cache's PACE_TIMEOUT: a worker writes nothing until the compiler has finished, so the client waits out the whole compile in a single read, and while the two shared one number every translation unit taking longer than ten seconds was abandoned and rebuilt locally — precisely the ones worth distributing (#223). Ten minutes because that is LeaseTable's own lease timeout: waiting longer means waiting on a lease the scheduler has already reclaimed.

PACE_DISPATCH_IDLE is the companion knob and the one that answers the other question: how long the worker may say nothing. It defaults to 30s. A worker writes a five-byte progress frame every few seconds while a compile is running, so the client measures silence rather than duration — which is what lets this be seconds while the total above stays minutes (#245). Setting it to 0s turns it off and restores one flat deadline. All four are durations — a whole number and a unit (ms, s, min, h, d); a bare number is not one and is ignored, so the launcher runs under the default.

Both are runtime settings: the launcher is one process per translation unit, so exporting a new value is all it takes and the next compile uses it. Nothing reloads and nothing restarts. Raise the total if a single translation unit legitimately compiles for longer than ten minutes; lowering it buys you nothing against a worker that has stopped, for the reason in the next box.

A worker that stops is noticed in seconds, not at the deadline

A flat deadline cannot tell a worker that is still compiling from one that has stopped, so it is not asked to. Two mechanisms answer the second question instead, and they cover different failures:

the worker	detected by	when
process exits, crashes, is killed	the client's own parked read — FIN or RST	immediately
host vanishes: powered off, unplugged, suspended, VPN dropped	TCP keepalive on the compile dial	~16 s on Linux and macOS, ~30 s on Windows
host answers while the process makes no progress	the missing progress frames — `PACE_DISPATCH_IDLE`	~30 s
compiler runs longer than anybody is prepared to wait	`PACE_DISPATCH_TIMEOUT`	at the deadline

In every one of them the client hands the lease back and compiles locally, so nothing is lost and the key is not pinned for the scheduler's lease timeout.

The Windows keepalive figure is not a typo: the OS fixes the probe count at 10 and offers no way to set it.

Row 3 is the one keepalive cannot see — the kernel answers every probe, so nothing below the protocol knows anything is wrong. The worker therefore says so itself: while a compile is running it writes a Status::Progress frame every five seconds, carrying nothing at all, and the client gives up after thirty seconds of silence. The launcher's own fall-back line (PACE_VERBOSE) then reads stopped reporting progress rather than ran out of budget, because "that machine has wedged" and "that compile was slow" are fixed in different places by different people. On the --show-stats axis both are tallied as unavailable, which is deliberate: that tally is bounded by cause and does not grow a row per socket condition.

This is a wire feature, not a client-side heuristic: it is why the 0xFC protocol version is 3, and a launcher or node from before it is refused with unsupported-version rather than being left to meet a frame it cannot parse.

What a client that runs out of budget costs the fleet

The worker notices the client has gone and skips writing the object back, so the transfer is not paid for. The compile itself still runs to completion, so that CPU is spent twice and pace_worker_jobs_completed_total still counts the job; pace_worker_jobs_abandoned_client_gone_total counts the delivery that was skipped.

PACE_TOKEN is the cache's password, and a node never receives it

PACE_TOKEN is presented to the cache at PACE_ADDR and to nothing else. Every exchange with another machine — the scheduler, a worker, a remote cache — presents a machine ticket instead, minted for that exchange by this machine's own node, so a password is never sent where it was not configured.

A node checks no password. A password AUTH that does reach one — an older launcher, a hand-written client — is answered Ok and establishes nothing, so the request behind it is served or refused exactly as if nothing had been presented.

A node's scheduler once answered a credential dispatch-not-permitted, which the launcher treats as fatal and returns in place of the answer to the request it actually sent — so every LEASE was declined and every compile happened locally, behind a green build (#340).

Confirming it works¶

With PACE_VERBOSE=1 the launcher says what happened to each translation unit:

pace-cc: HIT key=…                                          served from cache
pace-cc: DISPATCHED to worker-01.internal:6674 key=…        compiled remotely
pace-cc: not dispatched (rejected (no-worker)); compiling locally
pace-cc: cache unavailable (fetch exchange failed); compiling this translation unit anyway

The last of those is a cache problem, not a fleet one, and the line says so by not ending at the compiler: a DISPATCHED line normally follows it.

The scheduler is a compile node, so its /metrics endpoint is the node's: start it with --admin-listen and these count the outcomes.

Counter	Rising means
`pace_dispatch_leases_granted_total`	Work is being distributed.
`pace_dispatch_leases_released_total`	Clients are reporting their jobs done. Granted minus released is what is outstanding; flat while granted climbs means clients are dying mid-job, or predate the verb.
`pace_dispatch_leases_no_worker_total`	The fleet is misconfigured — workers are up but nobody matches.
`pace_dispatch_leases_no_capacity_total`	The fleet is too small — full of your own build — or, where a worker reads `limited-by: compile-budget`, its compiles hold its `--compile-inflight-max` instead: raise that on it.
`pace_dispatch_leases_withdrawn_total`	The fleet is unavailable — slots free on paper, machines busy elsewhere or out of scratch space.
`pace_dispatch_leases_all_excluded_total`	The fleet is unreachable from its clients — every matching worker was on the asking client's exclusion list. A network problem, not a missing toolchain. Check the VPN and the workers' advertised names.
`pace_dispatch_leases_duplicate_total`	Duplicate-work suppression is doing its job. Not a problem.
`pace_dispatch_worker_registrations_total`	Workers registering. A steady rise means heartbeats are not arriving.
`pace_dispatch_worker_registrations_malformed_total`	A peer named its toolchain, endpoint or version in bytes that are not UTF-8 and was refused. Any rise names a machine that is not in the fleet.
`pace_dispatch_worker_endpoint_mismatch_total`	A worker was admitted while advertising an endpoint whose host is not the address it connected from. Not a fault by itself — DNS names, a node dialling itself, NAT, a VPN and multi-homing all land here. See the endpoint a registration names is not verified.
`pace_dispatch_workers_expired_total`	A machine stopped heartbeating and was dropped. Rising beside a rising registration count is a fleet whose heartbeats are not arriving, not one that is growing.
`pace_dispatch_workers_withdrawn_total`	A machine re-surveyed, found it no longer serves a toolchain, and retired that registration itself rather than letting it time out. Never sum it with the expiry counter above: an expiry is a machine that stopped answering, this is one that said so, and adding them makes every routine toolchain upgrade read as a heartbeat failure.
`pace_dispatch_leases_reclaimed_total`	A machine went away mid-job, or restarted, and the keys it was building were freed. Work nobody will report done — a build that lost part of its distribution, which is a different thing to fix from a fleet that is merely full.
`pace_dispatch_leases_unauthorized_total`	A client handed back a lease token this cluster never signed. Never sum it with the `unknown-lease` refusals beside it: those name a lease this scheduler did issue and has since forgotten, while a rise here is a forged release — or, far more likely, a launcher predating signed leases, in which case it tracks a rollout and stops when the rollout finishes.
`pace_dispatch_leases_released_late_total`	A client reported a job whose lease had already expired: a real compile outran the lease timeout. Read it as a fraction of `released_total` — a steady fraction means the lease bound is shorter than this site's slowest translation unit. Not a count of leases that expired: a client that never reports back at all reaches this nowhere, and is `reclaimed` or nothing.
`pace_dispatch_leases_malformed_total`	A client asked for a lease naming its key, toolchain or toolchain label in bytes that are not UTF-8, and was refused before anything it named was kept. `pace-cc` checks its label before sending it, so any rise names a client that does not.
`pace_dispatch_leases_field_too_long_total`	A client asked for a lease with a key, toolchain fingerprint or toolchain label longer than a scheduler records, and was refused before any of it was kept. Each is a few dozen bytes from a real client, and `pace-cc` sends none longer, so any rise names a client that does.
`pace_dispatch_worker_registrations_field_too_long_total`	A worker registered with a fingerprint, endpoint, version, toolchain label, display name or codec list longer than a scheduler records, and was refused before any of it was kept. Each is kept in the worker's entry, and this project's nodes send none that long -- a compiler whose name would overrun its ceiling registers without the name -- so any rise names a peer that does.
`pace_dispatch_node_announcements_field_too_long_total`	A machine announced itself with an endpoint or version longer than a scheduler records, and was refused before any of it was kept. A condition field has a ceiling too, but an overlong one is refused earlier, when the frame is decoded, as a malformed frame no counter moves for -- so this series never sees it from the network. This project's nodes send none that long, so any rise names a peer that does.

The first three refusals are different operator problems and are deliberately counted apart: summing them hides a misconfiguration behind a busy fleet, and hides an unavailable fleet behind one that merely looks undersized.

Those count what the scheduler decided. Four more count what it refused before any verb reached it — a frame turned away at the wire, which until #494 answered a client and moved nothing, so a scheduler being probed and a scheduler nobody was talking to drew the same flat line.

Counter	Rising means
`pace_dispatch_frames_refused_unsupported_version_total`	A peer built against another release of the wire. During a rollout this tracks the rollout; afterwards it names a machine nobody upgraded.
`pace_dispatch_frames_refused_unknown_opcode_total`	A frame naming a verb no build has. A scanner, or a client speaking something else entirely at this port.
`pace_dispatch_frames_refused_not_permitted_total`	A frame naming a verb that exists and is served on a different port — typically a cache or compile request sent to the scheduler's address. Any rise names a client pointed at the wrong one.
`pace_dispatch_frames_refused_truncated_total`	A frame whose header disagreed with what arrived: a framing or transport fault, before any verb was routed. Never sum it with a `malformed_payload` series — those are payloads that decoded wrong, and share only the wire code.

A non-member caller is not in this table: that refusal is decided one layer down, in SchedulerService, which deliberately counts it nowhere so a client retrying past a momentarily full surface cannot bury the credential series. Whether it should get a series of its own is #592.

On each worker, start it with --admin-listen and the same endpoint reports what that machine is doing:

Counter	Rising means
`pace_worker_jobs_started_total`	Jobs accepted.
`pace_worker_jobs_completed_total`	Jobs that ran to an exit code — including a non-zero one, which is the client's answer rather than a worker failure.
`pace_worker_compile_milliseconds_total`	Compile wall time. Divide by `..._jobs_completed_total` for the mean; both are counters, so a rate over a window gives you the current one.
`pace_worker_jobs_abandoned_client_gone_total`	The client had disconnected before the object could be written back, so it was not sent. The compile still counts in `..._jobs_completed_total` — it ran and this machine paid for it, and only the delivery found nobody there. A rise is a client-side story: cancelled builds, `Ctrl-C`, a CI runner reclaimed mid-job. What it saves is the transfer; the CPU is spent either way until #661. A client that half-closes its write side after sending and still waits for its object is read as gone and counted here — nothing shipped in this project does that, but a third-party client speaking the protocol over a raw socket could.
`pace_worker_jobs_refused_no_slot_total`	This worker is full. Pair it with the scheduler's `..._no_capacity_total`.
`pace_worker_jobs_refused_unknown_fingerprint_total`	Somebody is dispatching a toolchain this worker does not have.
`pace_worker_jobs_refused_rejected_argument_total`	A command line carrying something that could name a file.
`pace_worker_jobs_refused_scratch_unavailable_total`	The scratch disk is full or unwritable.
`pace_worker_jobs_refused_spawn_failed_total`	The toolchain is configured but cannot be executed.
`pace_worker_jobs_refused_compiler_unclassified_total`	The toolchain runs, and this build cannot tell which driver it is, so no command line can be built for it. The remedy is the `--toolchain` on that node, not a path. The node also says this once at survey time, so check its startup log before this counter.
`pace_worker_jobs_refused_not_a_member_total`	A caller this worker does not admit tried to compile on it. Check your own fleet first: if this worker holds no roster and is not `--fleet-open`, it admits its own machine and nothing else, and this counter is your clients being turned away one hop after a lease was granted; a client whose machine the cluster has not admitted lands here too, and `pacectl explain-admission <machine>` says so. Once the policy is right, a rise here is somebody with no claim on the machine — check who can reach `--listen-node`.
`pace_worker_jobs_refused_envelope_declared_too_large_total`	A request declared its payload expands past this worker's ceiling. Nothing honest does that by accident: a probe of the port, or a client with a larger ceiling than the worker.
`pace_worker_jobs_refused_envelope_unsupported_codec_total`	A client compressed with a codec this worker was not built with. A packaging difference between two honest machines; each one cost a local compile.
`pace_worker_jobs_refused_envelope_malformed_total`	A payload envelope that did not parse: a version skew, or something on the port that is not this protocol.
`pace_worker_jobs_refused_envelope_corrupt_total`	Bytes that parsed and then did not expand to their declared size. The one refusal here that implicates the transport.
`pace_worker_jobs_refused_lease_unauthorized_total`	The lease presented was not signed by any voter the worker's roster names. The one counter here that is unambiguously a security signal rather than a capacity or configuration one — or a launcher predating this lease format, which is told apart by whether the rise tracks a rollout.
`pace_worker_jobs_refused_lease_signer_revoked_total`	The lease verifies — under a key the cluster has revoked. The removed machine itself, still leasing out work: it should read zero forever, and a rise names a machine somebody removed and nobody stopped.
`pace_worker_jobs_refused_lease_wrong_cluster_total`	An authentic lease was signed by a different fleet. Never sum it with `..._unauthorized_total` beside it: that one is a bad signature, this one is a good signature from somebody else. A rise here is a provisioning mistake — one identity key voting in two clusters, which is what copying a `--cluster-dir` to a second site produces — and the fix is a fresh state directory on one of them, not a firewall.
`pace_worker_jobs_refused_lease_replayed_total`	An authentic, unexpired lease that this worker had already run. A lease authorizes exactly one compile — one grant per lease, presented once, with no retry — so nothing honest produces this and it should read zero forever. Any rise is somebody presenting a captured grant a second time. Do not sum it with `..._wrong_cluster_total` either: they share a wire code and nothing else, and they send you to opposite places.
`pace_worker_jobs_refused_lease_endpoint_mismatch_total`	An authentic lease named a different worker. Almost never a replay and almost always a worker registered under an address clients do not dial — a NAT, or a hostname where clients resolve an address.
`pace_worker_jobs_refused_lease_expired_total`	An authentic lease had expired. A rise on one machine and nowhere else is that machine's clock, not the fleet's leases — which is why the check carries skew slack and why this is worth seeing per node.
`pace_worker_jobs_refused_lease_isolated_total`	This worker has heard from no leader its fleet counts for longer than it may trust the state it applied (65 minutes), so a voter the fleet forgot meanwhile could still be in it, and every lease is refused until one speaks again. Should read zero; a rise means the worker is cut off from its fleet's consensus while something still hands it leases.
`pace_worker_jobs_refused_lease_no_roster_total`	The state this worker applied records no voter's key yet, so no lease from anybody can verify. A few while a node starts or joins are ordinary; a rise that does not stop means this node never applied its fleet's state.
`pace_worker_scratch_roots_reclaimed_total`	This worker took over a scratch root left behind by a node that exited without cleaning up. The work itself is correct — a root is only reclaimed once its previous owner's exclusive claim is free, which the OS releases however that process died. A rise means nodes are dying rather than stopping, which is worth knowing and is visible nowhere else.
`pace_worker_bytes_received_total` / `..._returned_total`	Link volume, counted at the socket.

The refusals are split by reason for the same reason the scheduler's two are: a full worker and a misconfigured one are different problems with different fixes, and one number covering both tells you neither.

Retired series. pace_worker_jobs_refused_lease_stale_epoch_total no longer exists, as of #614. If a dashboard or alert of yours scrapes it, it will now find nothing — that is the change, not a fault. The refusal it counted can no longer happen: a worker adopts a scheduler term that went backwards instead of refusing it, because replay is closed by grants being spendable once. Point those alerts at pace_worker_jobs_refused_lease_replayed_total (somebody presenting a captured grant twice) and pace_worker_scheduler_term_regressions_total (a scheduler term that went backwards). The reasoning is under "A grant is spendable once" below. It was retired rather than left exported at zero because a series that reads zero for ever because its event is impossible looks exactly like one reading zero because the event has not happened, and only one of those means your fleet is healthy.

Retired with the certified roster. pace_worker_jobs_refused_lease_roster_expired_total, pace_node_roster_expires_in_seconds, pace_worker_rosters_refused_uncertified_total, pace_worker_rosters_refused_expired_total and pace_scheduler_roster_endorsements_refused_total no longer exist. Every node verifies a grant against the state its own consensus applied, which nothing certifies and nothing lapses, so none of those events can happen. Point those alerts at pace_worker_jobs_refused_lease_no_roster_total (a node whose applied state names no voter yet) and pace_worker_jobs_refused_lease_signer_revoked_total (a machine the cluster removed, still minting).

The ..._lease_* counters move only on a worker that checks leases: a consensus member -- a learner included -- against the state it applies. A worker that checks nothing says so at startup rather than leaving these at zero and looking healthy — and one another machine could dial is refused outright (#282, #178).

Four of them — ..._unauthorized_total, ..._signer_revoked_total, ..._wrong_cluster_total and ..._replayed_total — share the wire code lease-unauthorized, because a client's answer to all four is the same, compile it locally, and a code it does not recognise would be worse than one it does. Your answer is different for each, which is why they are four series rather than one: a bad signature is a security question, a revoked signer is a machine somebody removed and nobody stopped, a good signature from another fleet is a provisioning one, and a grant presented twice is somebody replaying a credential.

The rest carry codes of their own — ..._endpoint_mismatch_total answers lease-endpoint-mismatch, ..._expired_total answers lease-expired, and ..._no_roster_total and ..._isolated_total both answer grant-unverifiable (0x31; 0x28, once roster-expired, is retired and never reused), which says the worker can check nobody's lease right now rather than that this one is bad — so an alert written against lease-unauthorized will not see them. That is deliberate and it is LeaseRefusalTable in LeaseToken.hpp that decides it: a client can act differently on those two, and a code it can act on is worth minting.

A grant is spendable once, and the scheduler's term is only a diagnostic¶

A lease authorizes exactly one compile. The scheduler mints one grant per lease, the launcher presents it in exactly one request with no retry, and it hands the lease back to the scheduler rather than to the worker — so a grant arriving twice at one worker is a replay and never an honest client. Since #614 a worker enforces that: the second presentation is refused and pace_worker_jobs_refused_lease_replayed_total moves. Before it, a captured grant could be replayed at its worker until it expired.

A grant refused for anything else — a wrong toolchain, a wrong endpoint, an expiry — is not consumed, so a client whose job was declined for a configuration mistake still holds a usable lease.

The one thing this does not cover is a worker restart, which forgets what it has spent. A grant captured and withheld across a restart is usable once afterwards, for whatever is left of its expiry. That is the same window the lease expiry has always bounded.

The scheduler term inside a grant no longer decides anything. It is carried, it is signed, and a worker adopts it — but it is a diagnostic rather than a check. Until

614 a worker refused any grant naming a term below the one it had learned, and three¶

ordinary operator actions break that: wiping a scheduler's Raft directory, re-bootstrapping the cluster, or turning consensus off all drop it to a lower term truthfully. Every worker then refused every grant until its process was restarted, with nothing but a climbing counter to say why — which read like an election storm.

Worse, the rule was backwards in exactly that case. With a worker at term 7 and a scheduler honestly reset to term 0, it refused the fresh grant and accepted a token captured under term 7. The check written to stop replay refused every legitimate grant and admitted the replayed one. Grants being spendable once is what actually answers that question, so the term check is gone and pace_worker_jobs_refused_lease_stale_epoch_total was retired — the refusal it named can no longer happen, and a series reading zero because its event is impossible looks exactly like one reading zero because the event did not happen.

What you watch instead is pace_worker_scheduler_term_regressions_total, which counts a worker adopting a term that went backwards, alongside one WARN line naming both terms.

It does not mean somebody reset a cluster, and you have to read the rate to know which it is. Two things produce a lower term and a worker cannot tell them apart from the grant:

A grant that arrived late. A client asks for a lease, then preprocesses and uploads a translation unit — seconds — and an election inside that window puts its term-N grant behind somebody else's term-N+1 grant at the same worker. Completely ordinary. Expect occasional single counts that line up with leadership changes.
A scheduler that lost its state — Raft directory wiped, cluster re-bootstrapped, consensus turned off. This one repeats, because every grant that scheduler mints now carries the lower term.

So: occasional counts tracking elections are the first and need nothing from you. A sustained rise, especially across the fleet at once and against no election, is the second.

The trade, stated plainly because it is real: a grant captured before it ever reached its worker used to stop being good at the next election, and now stops being good at its expiry. That is the weaker of the two bounds. It could not be kept — a scheduler that was legitimately reset and a replayed older grant look identical inside a token, and refusing them together is what stopped the fleet.

The worker also reports what the machine is — pace_node_logical_cores, pace_node_memory_total_bytes, pace_node_disk_capacity_bytes, pace_node_disk_free_bytes, pace_node_slots_configured and pace_node_slots_busy. Those are gauges: "is this node pulling its weight" is not answerable without knowing how big it is. Beside them is what the machine is doing — pace_node_memory_available_bytes, and the CPU as two tick counters, pace_node_cpu_busy_ticks_total over pace_node_cpu_ticks_total, whose rates divide into the busy share.

And what its cache tier is doing — pace_node_cache_hits_total and ..._misses_total for the tier itself, ..._upstream_hits_total for what the shared cache answered after a local miss, and ..._fill_failures_total, ..._store_failures_total, ..._upstream_stores_total and ..._upstream_store_failures_total for the writes. A high upstream-hit rate against a low local one means the tier is too small for this machine's working set, which is a different problem from a fleet that is missing a lot.

Beside them, what it is holding — pace_items, pace_bytes_used and pace_bytes_limit for the cache as a whole, plus a pace_tier_*{tier="memory"|"disk"} set for the split. A node whose cache is effectively empty sends every rebuild to the wire; one whose evictions climb while its hit rate falls has a budget too small for the tree it builds. Both are invisible from the scheduler's own counters, which only see the work that reached it. See the node's own page for why these must not be summed across tiers, and why a node running no tier reports the series as absent rather than as zero.

The same figures reach the leader, on each node's REGISTER and HEARTBEAT, so they are readable from one place rather than by scraping every machine. Note that a node started with two --toolchain flags is two registry entries against one machine and one cache: both carry the same figures, and anything summing them counts that cache twice.

The whole fleet on one page¶

--dashboard, on a leader that already has --admin-listen, serves /fleet and /fleet.json — every member's hostname, endpoint, software version, capacity and cache, plus charts of the last 24 hours or 7 days:

pace-node ... \
    --listen-node=6675 \
    --admin-listen=6677 \
    --dashboard --dashboard-token-file=/etc/pace/dashboard.token

Only the leader answers it in full; anyone else replies 503 naming the leader, because a follower's registry holds whatever registered against it rather than the fleet. The credential is required off loopback and is deliberately its own file rather than --requirepass. /metrics and /healthz stay outside it, so a scraper or a probe is unaffected — and /metrics remains the source of truth for anything you alert on. The full account, including what each column means and why a value nobody reported renders as – rather than 0, is under looking at the whole fleet.

"No worker matches this toolchain"¶

The commonest setup failure. A worker is matched only if its toolchain fingerprint is byte-identical to the client's, so ask both ends:

pace-cc --print-toolchain-fingerprint /usr/bin/g++    # on a client
journalctl -u pace-node | grep serving        # on a worker

If they differ, the machines really do have different toolchains — different patch releases, a different Xcode SDK or sysroot, a vendored header that is not the same.

The fingerprint is a digest of the compiler's version banner, its argument grammar and the include tree that belongs to it. That is what lets two machines with the same toolchain at different install prefixes match, while two machines whose headers differ do not. It cannot be loosened: an over-strict match costs one local compile, an over-loose one produces a silently wrong object that is then stored under a key every other machine fetches.

Belongs to it is the operative phrase on Windows. cl carries no tree: its banner names its toolset version and target, and the toolset and SDK headers are ones a worker never opens, since it compiles text the client already preprocessed. Folding them in split two machines with the same toolset and SDK on nothing but an installed Visual Studio language pack (#1656). clang-cl, and clang/clang++ targeting *-windows-msvc*, carry only the resource directory the driver names when asked (-print-resource-dir), because they borrow the VC toolset and the SDK rather than owning them — so two machines with different SDKs or VS languages installed still match. A clang targeting anything else carries every directory its -v search list names, as GCC does. None of them derives its fingerprint from INCLUDE, which is set per developer command prompt and never inherited by a service — so a worker installed as a service matches the launchers that talk to it.

The fingerprint and the cache key are not the same string¶

They answer different questions, so one of them carries the compiler's target and the other deliberately does not:

	Decides	Carries the target?
Toolchain fingerprint	which worker may serve a client	no
Object cache key	which object may be served	yes

A driver's code generation is not a function of the driver alone. clang-cl detects the MSVC installation beside it and sets -fms-compatibility-version from that, and clang's Microsoft C++ ABI gates version-specific code generation on the value — so the same clang-cl.exe in a developer prompt and under a service generates differently. Stock g++ on x86_64 and aarch64 is the same shape with no MSVC anywhere near it: one --version banner, two code generators.

The key therefore folds the target in, so those cases stop sharing entries. The fingerprint must not, or a developer-prompt launcher would stop matching a service-run worker — the mismatch that keeps a fleet answering no-worker for reasons nobody can see. Dispatch closes the gap on the line instead: a dispatched compile states --target=<triple> ahead of the build's own arguments, so the worker generates for the client's target rather than re-deriving one from its own machine, while a --target= or -m32 the build states itself still wins.

Only a driver that can be told a target is pinned this way. gcc is fixed-target and rejects the flag, so its target is identified for the key and never stated on a dispatched line; cl is neither, because which code generator runs is decided by which cl.exe you invoked and no command line can restate it.

The practical consequence is a hit-rate one: after this change two machines whose compilers report one banner but generate for different targets no longer share cache entries. That is a correction — they were sharing objects they should not have — but it looks like a cold cache the first time a fleet upgrades past it.

Sizing and operations¶

Misses are bursty. With a warm shared cache, hit rates run 90 %+, so a fleet is idle most of the time and the value is concentrated in the first build of a commit, developer branches and toolchain bumps. Socket activation exists for exactly this shape.
--slots defaults to derived: hardware threads, clamped by what the memory supports, less what --node-class reserves (two cores on a workstation, none on a dedicated node). A number you give it overrides all three. Whatever it resolves to is advertised and enforced locally from one calculation, so a worker cannot end up fuller and slower than the scheduler believes at the same moment. --slots=0 is not a count: it runs no worker at all (see The scheduler above). See the worker's own page.
--node-class defaults to workstation, which is the safe answer rather than the common one: a node nobody classified is somebody's desktop until proven otherwise. Set --node-class dedicated on build servers, or the fleet quietly runs two cores short on every one of them.
The scheduler picks by free slots, not by fewest running jobs. Absolute counts make a 64-slot server running 8 jobs look busier than a 4-slot laptop running 2, which sends work to the smallest machines first.
Workers withdraw capacity while their machines are busy elsewhere. Each heartbeat carries host CPU, available memory and free scratch space; the fleet's own jobs are subtracted, so what is left is load that belongs to somebody else. A worker whose scratch filesystem fills up stops being picked entirely, and starts again when it drains.
Four lease refusals, never summed. no-worker means a fingerprint nobody serves, no-capacity means the fleet is too small, withdrawn means the machines are there and unavailable, and all-excluded means the machines are there and the client itself could not reach them. Each has its own pace_dispatch_leases_*_total, because the fixes are different and folding withdrawn into no-capacity sends an operator to buy hardware they already own, while all-excluded folded into no-worker sends them chasing a toolchain that is not missing.
A node's own cache tier is subtracted from what it can compile. Capacity is one job per gigabyte of RAM, and a resident cache is memory that will not yield — so a 64-thread host with 32 GiB holding 8 GiB of cache offers 24 slots, not 32. What is subtracted is what the tier actually holds: a node with --cache-memory=0 and no --cache-dir reserves nothing and offers the whole machine, and so does one whose cache never started.
Workers can come and go. A worker heartbeats every 20 s and is dropped after 90 s without one; a client that leases one in the gap finds it unreachable and compiles locally.

Security¶

Two things protect a fleet: who a node admits, and whether the job was granted.

Every machine that joins the fleet proves which one it is. A worker proves its own identity key on every connection to its scheduler, and every frame after that proof is sealed under a key only the two ends hold -- so something on the path can neither speak for a worker nor slip a verb into a connection the worker's proof admitted. A machine the cluster forgets with --cluster-forget is refused on its key, from any address. See A node proves which machine it is.

A node is closed by default. Its compile port and its scheduler admit this machine, and machines whose key the cluster admitted — by a proof, or by a machine ticket — or every caller, once you say --fleet-open. A machine stops being admitted everywhere at once when the cluster forgets it, since its key is what every node asks about on every request. That matters most for the compile port, which binds 0.0.0.0 because peers have to dial it — anybody who could route to it would otherwise have your machine run their compiler on source they chose.

Its own cache tier admits this machine and nothing else, whatever the roster says and whatever it is bound to (#287). A peer that may spend this machine's CPU is not thereby entitled to read everything it has ever compiled, and the tier is exactly that. A cache several machines share is a pace-kv behind --upstream.

Membership alone was never enough, because admitted to the fleet is not granted this compile: an admitted machine could spend any worker's CPU without ever asking the scheduler for a slot. So a lease grant is a signed capability. The scheduler signs the granted worker's endpoint, the toolchain, the object key and an expiry with its own Ed25519 identity key, and the worker checks that signature — against the cluster's roster of voters, unrevoked — before it decompresses anything (#281, #282, #178). The endpoint is inside the signature, so a token captured on the way to one machine cannot be replayed against the rest of the fleet; and a signature is one machine's, so a scheduler the cluster removes stops being able to lease anything out the moment its workers hold the roster that revokes it — or, if one withholds that roster, once the roster they hold lapses.

Give every worker a --cluster-dir, and a way to check a lease

Every worker registers with a scheduler, so every worker keeps its identity in a state directory: --cluster-dir, or the platform's default. And a worker another machine could dial — one that admits other machines, on a bind that is not loopback — will not start with no way to check a lease: it runs consensus, or keeps a roster in its --cluster-dir. Both refusals are deliberate and both are startup ones, not per-request fallbacks: a worker that quietly skipped the check would serve whoever reached its port while every refusal counter read zero, which is a fleet that looks healthy from both ends.

A node nothing else can dial — the ordinary one-machine install — needs no roster and logs a warning saying the check is off. A process on that host already has that host's compiler.

Upgrade the fleet together. A lease format and a roster are new in the same release, and a worker and a scheduler from either side of it cannot agree on either; nor can consensus members from either side of the authenticated consensus wire.

A node verifies machines, and holds no shared secret

A compile node's framed surfaces — scheduler, compile port, cache tier — take one inbound credential, and it identifies a machine, not a person: a machine ticket, signed by a machine's own identity key and verified against the roster the node holds. There is no password and no key shared across the fleet, so nothing a node could leak lets another machine speak for it. --requirepass on a node is only the secret it presents when it dials a pace-kv named by --upstream, which is the one surface that takes one.

A launcher sends PACE_TOKEN to the cache at PACE_ADDR and to nothing else; every exchange with another machine presents a ticket its own node minted for that exchange. A password AUTH that does reach a node is answered Ok and establishes nothing, so a client configured with a token is never broken by a node — the node's scheduler once answered it dispatch-not-permitted instead, which the launcher treats as fatal (#340).

A ticket travels in the clear, so it is bound to ONE endpoint, lives a minute and is spent once at the node that accepts it; a node proof is sealed from its first frame. A network where a ticket can be captured costs a replay at no other node and no later moment. The cache tier admits this machine only, whatever the caller presents.

Beyond the lease, consensus — whose every connection proves each member's own identity key, see Raft peer authentication — and the machine ticket a client presents, the fleet's own traffic is not encrypted, so treat its boundary as network reachability plus membership and size the network accordingly.

So: keep a scheduling node's port off any network you would not run a compiler for, and put mTLS in front of every port for anything beyond a trusted build network. The two remaining credentials in this system are real and unaffected — --dashboard-token-file guards the fleet page and the live-stats fleet stream, and pace-kv's own --requirepass guards the shared cache.

Also worth knowing: no source address admits anybody but a node's own machine (#180) — a machine is admitted by its key — so a spoofed address buys a stranger nothing. A captured ticket buys one replay at no other node and no later moment, since each names one endpoint, lives a minute and is spent once.

Limits worth knowing before you adopt it¶

Preprocessing does not distribute (see step 1). The ceiling is ~10–40×.
-g embeds the worker's scratch path in DWARF. Use -fdebug-prefix-map, which this launcher recognises and folds into the cache key, so two checkouts still share. -ffile-prefix-map and -fmacro-prefix-map are keyed the same way and share across checkouts too. -fmacro-prefix-map travels to the worker verbatim, which applies it to the client's paths; a line carrying -ffile-prefix-map compiles locally, since it also rewrites debug records a worker maps by its own rules. See the launcher's own account.
Diagnostics from a failed remote compile are not shown. A worker reporting a non-zero exit is retried locally and the local result is what you see — which also regenerates the diagnostics with correct line numbers.
--install-service is Windows and macOS. It registers an SCM service or a launchd job. On Linux the packages ship a socket-activated systemd unit instead, so there is nothing for the flag to do and it reports as much. A worker that needs --requirepass cannot be registered on any platform — a supervisor records launch arguments where every local account can read them — so on Linux that token goes in the worker's configuration file as requirepass:, and the registration carries --config=<path> rather than the token.
On Windows a dispatched object is not byte-identical to a local one — the code in it is. Every MSVC-family driver stamps the clock into the COFF header, and cl also records the absolute path of the object file (in .debug$S) and a hash of the source file it opened (in .chks64), with no debug flag asked for. A worker compiles its own scratch file, so those three differ; every section carrying code or data is byte-identical, which is what the end-to-end test asserts. If your build compares object bytes across machines, compare sections.
Some compiles are never distributed, by design. A C++ module interface unit, any compile that writes a precompiled header, and cl /Zi or /ZI (a PDB shared across the target) produce a second artefact beside the object, and only the object travels — so those are compiled locally and are not cached either. So is a cl compile that uses a precompiled header (/Yu): its object names the pch.obj it was compiled against by absolute path, and with cl 19.51 a hit replayed into another checkout failed to link (LNK2011), while one replayed after a PCH rebuild linked without the translation unit's debug info (LNK4206). clang-cl's /Yu object carries no such tie, so it is cached, but it is still compiled locally: the header it reads is on your machine, not the worker's. So is a command line whose language selector the launcher cannot restate for the preprocessed text it sends — an -x value with no preprocessed form such as -x assembler, or /Tc<file> / /Tp<file>, which name a file. A plain /TP, /TC or -x c++ — CMake puts /TP on every C++ source it hands MSVC — is folded into the language the launcher states, and dispatched. Run with PACE_VERBOSE=1 to see which of these applied.
A node's --requirepass is for its --upstream alone. It is the secret of the shared pace-kv, presented there and nowhere else. The fleet's own traffic is authenticated by machine keys instead (see Security): a node proves its identity key on every connection to its scheduler, and any other machine presents a ticket its own node minted. So there is no fleet password to set.

Reference¶

pace-node — every flag, the node's cache tier, the cluster, and the fleet dashboard.
Cluster communication — one compile as the fleet sees it, every connection in one table, and firewall rules.
Cluster discovery — how nodes find each other, and why finding one no longer admits it.
Compile cache protocol — the wire format.