Deployment¶

pace-kv ships the pieces needed to run it beyond a trusted LAN: authentication, TLS, a Prometheus metrics endpoint, a health probe, and a container image. This page shows how they fit together.

systemd¶

The .deb and .rpm packages install a unit that runs the daemon as a dedicated pace-kv system user and starts it on boot. See Install for the package-level workflow; this section covers what the unit does and how to adapt it.

The unit runs the daemon in the foreground under Type=simple and does not pass --daemon. That is deliberate: the POSIX daemonize path double-forks and redirects stdout/stderr to /dev/null, so journald would capture nothing, and its pidfile is written by the grandchild only after both parents exit, which races Type=forking with PIDFile=. Running in the foreground avoids both and gives journald the daemon's stderr directly.

journalctl -u pace-kv -f

Two things about --daemon changed in the release that fixed #784, and neither affects a machine installed from the packages, since no shipped unit passes the flag:

A relative --pidfile now resolves against the directory the command was run in. It used to resolve against /, because the host changed directory before writing it and only root can write there. An absolute --pidfile — which is what every documented invocation uses — is unaffected.
pace-kv --daemon still changes directory to /; it executes nothing, so that costs it nothing but a busy mount point. pace-node --daemon now changes to a directory of its own instead, because it spawns a compiler and a worker running in / records rewritten paths in every object it produces.

Leave log_timestamps off — journald timestamps every line already, and off is the default on Linux.

A start the daemon refuses — a flag it does not know, a configuration file it read that does not load, a combination of settings a rule forbids, and for pace-node also a credential file it read and found empty, or an identity key in its state directory it read and cannot use — exits 78 (EX_CONFIG; systemctl status shows status=78/CONFIG) and is not restarted: the unit says RestartPreventExitStatus=78, because the next start would refuse the same configuration the same way. A start that finds a defect in the binary's own build — a component it was built without — exits 78 as well, for the same reason: no configuration causes it, and every start meets it. The unit stays failed with the reason in the journal until you fix it and start it. Every other failure — a port still held by a process that was exiting, a store another process holds, and any named file that could not be read, whatever the reason: a configuration file, a credential or a key that is not there, or that the service account may not read, or that lives on a share or a mount that is not back yet — exits 1 and is restarted after a second (Restart=on-failure), within systemd's default start limit of five starts in ten seconds. A file that could not be read says nothing about its contents, and the next start may read it; if it cannot — a path that is simply wrong — the limit stops the retries.

Measured on systemd 259:

What happened	What systemd did
a refusal, at once or after three seconds	ran it once; `failed`, status 78
a failure, at once	five runs, then `failed` by the start limit
a failure after three seconds	restarted every time: nine restarts in forty seconds, still going
`systemctl restart` by hand, eight times three seconds apart	every one honoured
`systemctl restart` by hand, eight times 0.2 seconds apart	the fifth refused, "start of the service was attempted too often", and the unit left stopped

The last row is the default start limit, which counts your starts as well as systemd's: five starts inside ten seconds are honoured, and a sixth — the fifth systemctl restart right after a start — is refused and leaves the service down until systemctl reset-failed pace-kv && systemctl start pace-kv.

Only a start answers 1 or 78. A command that answers and exits — --install-service, --uninstall-service, --migrate-storage and --seed-config on either binary, and pace-node's --print-surfaces, --print-identity, --migrate-cache, cluster, enrollment and cordon commands — is not supervised, and answers what a script calling it should do next:

Exit	Meaning	When
0	done	it did what it was asked
2	a decision — do not retry	a command line that did not parse (wherever the command was typed on it), a configuration file that did not parse or names a setting that does not exist, an install the startup rules refuse, a service name the registration will not use, access denied, a service that already exists or is not installed, a store in a format it cannot convert or that another process holds, a refusal the scheduler or seed replied with — except the ones listed under 1
1	transient — a retry may help	a scheduler, seed or node that could not be reached or broke off the exchange, a reply that decided nothing: another cluster change still committing, an endpoint or fleet at its capacity, a fleet with no worker registered for the toolchain yet (one that is still starting), a cluster with no leader right now, leaders that went on redirecting to each other, a machine that gave up waiting to be approved (run it again once an operator approves it), a file that is not there or could not be read or written (a named configuration file included, as at a start), a service-manager call that failed on its own, a conversion that stopped part-way (run it again to finish it)

pace-kv --healthcheck is read by a container runtime, and Docker documents 0 and 1 and reserves 2, so it answers 0 for healthy and 1 for everything else — unhealthy, a refused command line, or a configuration file that did not load.

Reloading¶

systemctl reload pace-kv sends SIGHUP, which re-reads the config file in use — /etc/pace/pace-kv.yaml for the packaged unit. Only part of the configuration is reloadable:

Reloadable	Requires a restart
`log_level`, `max_memory`, `requirepass`, `auth_username`, `notify_keyspace_events`	`bind`, `port`, `listen`, `listen_tls`, `storage_path`, `storage_shards`, `storage_durability`, `storage_max_value`, `threads`, `active_expiry_interval`, `active_expiry_scan`

The right-hand column is live-wired at startup — listeners are bound and the storage backend is constructed once — so a reload that changes any of it is rejected in full, the previous configuration is kept, and the reason is logged. Reload therefore either applies everything or nothing.

Hardening¶

pace-kv never drops privileges on its own, so every restriction comes from the unit: User=/Group=, StateDirectory=, and the usual Protect*/Restrict* set. Check it with:

systemd-analyze security pace-kv.service

Two consequences worth knowing. ProtectSystem=strict and ProtectHome=yes mean TLS certificates must live somewhere the unit can still read — /etc/pace/ is the natural place. And binding a port below 1024 needs capabilities the unit drops by default:

# systemctl edit pace-kv
[Service]
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE

Persistent storage¶

StateDirectory=pace-kv provides /var/lib/pace-kv, owned by the service user. Point storage_path at it to enable the on-disk tier:

storage_path: /var/lib/pace-kv/cache

The path's shape selects the layout: a path with no file extension (as above) becomes a sharded directory, while one with an extension (cache.cow) is a single file.

launchd¶

On macOS the .pkg registers a launchd job. The same registration is available from the command line on any installation, and --service-scope picks the domain:

pace-kv --install-service --service-scope=user            # LaunchAgent
sudo pace-kv --install-service --service-scope=system      # LaunchDaemon

The system scope needs the _pace-kv service account, which the .pkg creates; installing from a tarball or a source build fails with a message saying so rather than registering a job launchd cannot spawn.

The .pkg creates a second account, pace-node, for pace-node. It is deliberately not shared with _pace-kv: a worker runs a compiler on input that arrived over the network while pace-kv owns the cache storage, so one account would let a compromised compile rewrite every cached object. It comes from the package's Runtime component, so it is present whichever launchd choice you made here, and the uninstaller removes both.

Every flag you pass on the command line alongside --install-service is baked into the job's ProgramArguments, exactly as on Windows. Values read from a --config file are not: they stay in the file, so editing it and restarting the job keeps working.

sudo pace-kv --install-service --service-scope=system \
    --config=/opt/pace/etc/pace-kv.yaml --threads=4

Prefer the config file for anything you might want to change later. A flag on this command line outranks the same key in YAML for the life of the registration, so --storage=… here would quietly pin the cache location and make a later storage_path: edit a no-op.

--requirepass is the one flag that cannot be baked in, and the install is refused rather than silently dropping it: ProgramArguments lands in a world-readable plist, so the secret belongs in the config file passed with --config. Passing both is refused too — nothing can tell from the command line whether that file actually carries requirepass:, so accepting the combination would be the silent drop under another name.

The package sets the system daemon's config to mode 0640, owned root:_pace-kv, so it is readable by the account the daemon drops to and by nobody else — which is the whole reason to keep the secret there rather than in the plist. If you point the daemon at a config of your own, the install checks that _pace-kv can read it and tells you how to fix it if not; without that check an unreadable config shows up only as a job that exits at every start, with the EACCES visible nowhere.

--service-scope=user must not run under sudo: the agent belongs to the invoking account, and sudo would resolve that to root — registering an agent under /var/root that your own login never starts and your own --uninstall-service cannot see. The install refuses rather than guess.

	`user`	`system`
Plist	`~/Library/LaunchAgents/`	`/Library/LaunchDaemons/`
Runs as	the invoking user	`_pace-kv`
Starts at	login	boot
Privileges	none	root to install
`KeepAlive`	on crash only	on crash only

The generated plist deliberately does not pass --daemon. launchd, like systemd, supervises the process it started; a job that double-forks is reaped immediately as exited and the service silently never runs.

The two scopes are alternatives — both bind the same address, and there is no unix-socket endpoint to separate them.

Both use KeepAlive={Crashed:true}: a process that crashed is restarted, no more often than every thirty seconds (ThrottleInterval), and one that exited — whatever its code — stays stopped. launchd cannot tell a refusal (exit 78) from any other exit: launchd.plist(5)'s KeepAlive conditions ask whether an exit was successful or a crash, never which code it had, and nothing in it stops after N. So every shape that restarts a clean non-zero exit restarts a refused start forever, once per ThrottleInterval, writing the same refusal each time; the system daemon's KeepAlive=true did exactly that. Crash-only is the least bad choice, and it costs two things the other supervisors do not:

a failure is not retried either — a port still held by a process that was exiting, or a per-user agent that lost the race for the port to another user's, is one line in the log and a stopped job, where systemd and the Windows service manager would start it again;
a process that crashes at every start is restarted forever, every thirty seconds.

Status, restart, logs:

launchctl print gui/$UID/software.lastrada.pace-kv
launchctl kickstart -k gui/$UID/software.lastrada.pace-kv
tail -f ~/Library/Logs/pace-kv/software.lastrada.pace-kv.err.log

The system daemon logs to /opt/pace/var/log/<label>/ instead — a directory per job, because that directory is owned by the account the job runs as and a machine may run both pace-kv and pace-node system-wide:

tail -f /opt/pace/var/log/software.lastrada.pace-kv/software.lastrada.pace-kv.err.log

Timestamps¶

Those are plain files, and launchd writes nothing into them itself — unlike journald, which stamps every line it captures, and unlike the Windows event log, where the time is part of the record. A log with no times is close to useless the one time it is read.

So log_timestamps defaults to on when running on macOS, in both binaries and in every deployment, not only under launchd: a terminal run stamps too. Turn it off with no_log_timestamps: true in either binary's configuration file, or with the --no-log-timestamps flag, which exists for this — a bare --log-timestamps can only ask for the default that is already in force.

Each key is its flag, so log_timestamps: false means "do not pass --log-timestamps" and leaves the platform default — which on macOS is on. Say off with the negative key, and on (for Linux) with log_timestamps: true. The command line outranks the file, as everywhere else.

There is no reload equivalent to systemctl reload: send SIGHUP to the pid launchctl print reports, or kickstart the job.

To remove it: pace-kv --uninstall-service --service-scope=<scope>, or sudo /opt/pace/bin/pace-uninstall to remove the whole install.

Windows service¶

The MSI registers pace-kv as an auto-start service — a disabled one, left stopped, when the pace-node feature is installed too, since both answer on 6674 (the MSI's service table) — and seeds C:\ProgramData\pace\pace-kv.yaml from the template it ships, unless a config is already there. The registration passes no --config: the service resolves that path itself at every start, so editing the file is all it takes to reconfigure it, and a machine without one still starts on the built-in defaults rather than failing.

The same registration is available from the command line on any installation:

pace-kv.exe --install-service
sc.exe start pace-kv
pace-kv.exe --uninstall-service

Pass --config=C:\path\to\pace-kv.yaml only to point the service at a file other than the default location.

When a start is refused¶

A configuration the service refuses at startup — a flag it does not know, a file that does not parse, a flag combination a startup rule rejects — is reported to the SCM as a stop with a service-specific code, and the reason is written to the Application event log under the service's name:

> sc.exe query pace-kv
        STATE              : 1  STOPPED
        WIN32_EXIT_CODE    : 1066  (0x42a)
        SERVICE_EXIT_CODE  : 78  (0x4e)

1066 is ERROR_SERVICE_SPECIFIC_ERROR: the reason is the process's own exit code, 78 for a start either binary refused on its configuration (or on a defect in its own build) and 1 for one that failed for a reason the next start may not meet. It is not error 1053, did not respond to the start or control request in a timely fashion — that is what a refusal used to produce, because the process exited before connecting to the SCM.

--install-service is held to the same rules as a start, so a line every start would refuse is refused while you are watching rather than registered. --uninstall-service and --healthcheck are not: removing a service must not be blocked by the mistake it was reached to undo, and a probe of a daemon that is serving must not call it unhealthy over a rule that decides only whether one may start.

The restart policy the registration sets is finite: after a failure the SCM restarts the service after one second, again after one second, once more after thirty seconds, and then leaves it stopped (sc qfailure lists the four actions). Ten minutes without a failure starts the count over. It counts failures rather than starts, so the SCM never refuses a start you make yourself.

A refused start is restarted here too, unlike under systemd, because the SCM cannot be told apart from a failure without losing what sc query shows: it runs its recovery actions for every stop whose WIN32_EXIT_CODE is not zero, and the only refusal that would escape them is one reported as zero — with the 1066 and the 78 above gone. So a configuration that stays wrong is refused four times, each writing its reason to the event log, and then the service stays stopped until you fix it and start it. systemd does not restart a refusal at all, and launchd restarts a crash only (see launchd).

What it runs as¶

The service logs on as the virtual account NT SERVICE\pace-kv. The SCM derives that from the service name and manages it itself, so there is no account to create and no password to keep. Told nothing, CreateService would use LocalSystem, which has unrestricted access to every local resource and is a member of the local Administrators group — more than a cache daemon listening on a socket has any use for.

It can still read C:\ProgramData\pace\pace-kv.yaml, because seeding grants NT AUTHORITY\SERVICE read on that file and a service logon is a member of it. The file does not inherit the directory's BUILTIN\Users read, so a requirepass: written there is not readable by every local account. It deliberately cannot write there either: a service that cannot rewrite its own configuration cannot be talked into loading a different one.

If you set storage_path, the account needs access to it. --install-service hands over whatever storage_path is configured at the time it runs, so seeding the config first and installing second needs nothing extra.

One exception: a storage_path that names a single cache file — an existing file, or a path with a file extension such as D:\pace\cache.cow — is only created by the installer if it is already there. Creating cache.cow as a directory would make the very next start treat it as a directory of shards and fan out inside it, so the installer refuses and says so. An upgrade therefore needs nothing extra (the file exists, and is handed over); a first install against a not-yet-created cache file prints that refusal as a warning: on the install line and needs the icacls command below, run against the directory that will hold it.

If you add or move storage_path afterwards, grant it from an elevated prompt:

icacls "D:\pace\cache" /grant "NT SERVICE\pace-kv":(OI)(CI)F

Re-running --install-service from an elevated prompt repairs it too: registering a service that already exists re-applies the registration — start type, command line, account, restart policy — and runs the handover again for the storage_path that install resolves. It does not restart a running service, so a changed command line takes effect at the next start.

A daemon that cannot open its storage says so at startup and prints this command with your own paths and service name filled in; without storage_path it is memory-only and needs no directory at all. The advice appears only when the daemon is running as the service (--daemon) — a foreground run is not the virtual account, so naming it there would send you after an identity that is not the one being refused.

Renaming the service with --service-name renames the account with it, since the SCM derives one from the other.

A storage path belongs to one daemon. The store is claimed exclusively while it is open, so a second daemon started against the same storage_path refuses rather than sharing it:

failed to open storage 'D:\pace\cache': StorageError(code=InUse system=0 context=FilePageStore::Open)
Another process already has this store open. A storage path belongs to one daemon:
stop the other one, or give this daemon a path of its own.

Running two daemons on one machine is fine — give each its own path. Nothing is written to the store to enforce this, so its files stay readable by any build.

InUse is one of four codes an open failure can carry, and they call for different things. UnsupportedFormatVersion means a healthy store of another vintage — convert it, never delete it. Corrupt means the bytes really are damaged: When a store reports Corrupt says whether the process starts, what deleting costs, and what to keep first.

--install-service records the flags it was given on the command line into the service's command line (values from a --config file stay in the file) and makes path arguments absolute — a service starts with its working directory set to C:\Windows\System32, so a relative path captured at install time would resolve elsewhere at boot. It creates the service already set to auto-start but leaves it stopped, so the first start is explicit; --service-start=manual registers it to wait for somebody to start it instead. The start mode belongs to the registration, not to the command line the service runs with.

What it opens in the firewall¶

Registering also opens the Windows Firewall for every cache listener that binds beyond loopback and, with --metrics, for a metrics endpoint that does — nothing at all for the default 127.0.0.1:6674. Each gets one inbound rule, admitting pace-kv.exe running as this service, on every network profile, in the group Pace: pace-kv; the install line lists them. --firewall-allow=<address[/prefix]> (IPv4 or IPv6, repeatable) limits which remote addresses they admit, and /0 is refused: leave the flag out to allow any address.

Running --install-service again replaces the group rather than adding to it, and --uninstall-service removes it once Windows accepts the delete — at once, even for a service that had not stopped within a minute and is only marked for deletion until it exits. A firewall that refuses never fails either command: it is reported as a warning, and an uninstall whose delete was refused leaves the rules in place, since the service may still be running. A listener bound to the name localhost is given a rule too, and the install says why — a name is whatever the resolver answers; bind 127.0.0.1 or ::1 to need none.

Container¶

A multi-stage Dockerfile builds a Release binary with TLS enabled and copies it onto a slim Debian runtime:

docker build -t pace-kv .
docker run --rm -p 6674:6674 -p 9259:9259 pace-kv \
    --bind=0.0.0.0 --metrics --metrics-bind=0.0.0.0 --requirepass=secret

The image's default CMD binds 0.0.0.0 (so the cache is reachable from outside the container) and enables the metrics endpoint. Binding 0.0.0.0 without --requirepass exposes the cache to anyone who can reach the port — that is exactly the deployment this bundle exists to secure, so pair it with --requirepass and, across untrusted networks, --tls.

Health check¶

The image's HEALTHCHECK runs pace-kv --healthcheck, which probes http://127.0.0.1:<metrics-port>/healthz and exits 0 (healthy) or 1. It is self-contained — no curl/wget in the image — but requires the daemon to run with --metrics.

/healthz answers 503 once any bind's accept loop has ended while the daemon was not stopping, naming the bind and what its last accept answered: such a port still listens and refuses every client, so a probe that only saw the process alive would call it healthy. Only a closed listener ends a loop; a client that reset its queued connection is logged at Warn and accepted past.

Authentication¶

Set a shared secret to require clients to authenticate before any data command:

pace-kv --requirepass=secret            # redis AUTH + memcached binary SASL PLAIN
pace-kv --requirepass=secret --auth-username=alice

Redis: AUTH secret or AUTH alice secret → +OK; data commands before auth get -NOAUTH.
memcached binary: SASL PLAIN against the same secret.
memcached text: has no auth handshake, so it rejects data commands while a secret is configured — use the binary or RESP protocol with auth.

The secret is compared in constant time and never logged.

TLS¶

TLS is compiled in with -DPACE_ENABLE_TLS=ON (the image enables it) and turned on at runtime:

pace-kv --tls --tls-cert=/etc/pace/server.crt --tls-key=/etc/pace/server.key
# clients:
redis-cli --tls --insecure -a secret ping

--tls on a build without TLS support exits with a clear error. Client certificate (mutual TLS) auth is not yet implemented.

Metrics¶

pace-kv --metrics --metrics-bind=0.0.0.0 --metrics-port=9259
curl http://host:9259/metrics    # Prometheus text exposition
curl http://host:9259/healthz    # 200 OK; 503 naming any bind that stopped accepting

If the endpoint cannot be bound the daemon carries on serving the cache without it, and says so at Warn. That is deliberate: the admin surface is a scrape and probe convenience rather than the service. pace-node answers the same failure by refusing to start, because an operator who asked a worker for an endpoint is almost always wiring a probe to it.

The series `pace-kv` exports¶

Every counter in the build is exported by every process that serves this endpoint, without exception — a per-counter "does this apply?" predicate is exactly the mechanism that once left seven of nine live counters unexported. So a pace-kv scrape also carries the pace_worker_*, pace_node_* and pace_scheduler_* series, and on a daemon they are flat at zero: nothing in this binary moves them. They belong to pace-node, which documents them. Read a zero there as this process produces no such events, never as this process has no such subsystem — a cumulative counter cannot tell you which, which is why the things that genuinely can be absent (pace_items and the whole storage block on a process with no cache, a tier a cache does not have) are omitted from the exposition instead of reported as 0.

The same reading applies inside the daemon's own namespace: the pace_dispatch_* block below is the fleet scheduler's, and the scheduler is pace-node. The pace_ prefix on it is historical.

Command traffic¶

Present only when the process has a cache, which for pace-kv is always.

Series	Says
`pace_cmd_get_total`	GET commands processed.
`pace_cmd_set_total`	SET-family commands processed.
`pace_cmd_touch_total`	TOUCH commands processed.
`pace_cmd_flush_total`	FLUSH commands processed.
`pace_get_hits_total`	GETs that found a live entry. With the misses below, this is the hit ratio.
`pace_get_misses_total`	GETs that found nothing.
`pace_delete_hits_total`	DELETEs that removed an entry.
`pace_delete_misses_total`	DELETEs with no matching key.
`pace_incr_hits_total`	INCRs against a present key.
`pace_incr_misses_total`	INCRs with no matching key.
`pace_decr_hits_total`	DECRs against a present key.
`pace_decr_misses_total`	DECRs with no matching key.
`pace_touch_hits_total`	TOUCHes against a present key.
`pace_touch_misses_total`	TOUCHes with no matching key.
`pace_cas_hits_total`	CAS requests that matched and stored.
`pace_cas_misses_total`	CAS requests with no matching key.
`pace_cas_badval_total`	CAS requests refused on a token mismatch — a real write conflict, not a missing key.

Capacity and what the cache is holding¶

Series	Says
`pace_items`	Live entries currently stored. A gauge.
`pace_bytes_used`	Bytes currently stored. A gauge.
`pace_bytes_limit`	The configured byte budget; `0` means unbounded. A gauge.
`pace_evictions_total`	Entries dropped to stay inside that budget. Sustained evictions with a falling hit ratio mean the working set no longer fits.
`pace_evicted_unfetched_total`	Entries evicted before ever being read — capacity spent on values nobody wanted.
`pace_expired_unfetched_total`	Entries that lapsed before ever being read.
`pace_expirations_total`	Entries removed because their TTL lapsed, whichever path removed them: a lookup or a write that met one, or the expiry cycle. The unfetched series above is the part of this nobody read, and `pace_expiry_keys_reclaimed_total` is the cycle's share alone.
`pace_write_errors_total`	Value writes that failed to persist: a full disk, an I/O error, a read-only mount, a damaged store. This one is about the disk. See When a store reports Corrupt.

Per tier¶

Emitted once per tier the cache actually has, each sample carrying a tier="memory" or tier="disk" label. A tier this cache does not have contributes no line at all rather than a zero: pace_tier_items{tier="disk"} 0 says a disk tier is standing empty, and a memory-only daemon has no such tier to be empty.

Nothing here is a total waiting to be summed. The memory tier mirrors what it reads out of the disk tier, so adding the item counts double-counts; the disk tier is consulted only when memory missed, so the hit/miss split is deliberately not published per tier and stays on the unlabelled series above.

Series	Says
`pace_tier_items`	Live entries this tier holds.
`pace_tier_bytes_used`	What this tier's budget counts: stored (compressed) value bytes for `memory`, the on-disk footprint (pages in use × page size) for `disk`.
`pace_tier_bytes_limit`	This tier's own byte budget; `0` means unbounded.
`pace_tier_file_bytes`	Length of the file backing a disk tier, free pages included; runs ahead of `bytes_used` by its free pages until a commit cuts them off the end or later writes reuse them. Absent for a tier with no file.
`pace_tier_evictions_total`	Entries this tier dropped to stay within its budget.
`pace_tier_index_bytes`	Resident memory this tier spends on its key index — always RAM, even for a disk tier.

Expiry¶

Series	Says
`pace_expiry_cycles_total`	Sweeps the active expiry cycle has run. Flat on a daemon serving traffic means the cycle is disabled (`--expiry-interval=0s`) or wedged, which otherwise looks exactly like nothing having expired.
`pace_expiry_keys_reclaimed_total`	Entries that cycle reclaimed — keys that lapsed and that nothing would have touched again, so no other path would ever have freed them.
`pace_keyspace_reclaim_events_dropped_total`	Reclaimed keys whose `expired`/`evicted` keyspace event was never published, because one call reclaimed more at once than the notification buffer holds. Any rise means a subscriber's view is incomplete.

Connections¶

Series	Says
`pace_connections_total`	Connections accepted since start.
`pace_connections_rejected_total`	Connections refused by admission control (`--max-connections`).
`pace_connections_total_tls`	The subset of `pace_connections_total` that arrived on a TLS-flagged bind. Plaintext traffic is the difference between the two.
`pace_connections_rejected_tls`	The TLS subset of the rejections.

The compile-cache (`0xFC`) surface¶

What pace-cc and pace-node talk to this daemon with. These are the daemon's own counters, and they are not the node's identically-shaped pace_node_cache_* series: a mixed fleet moves both, and watching only the node's reads a fleet mid-upgrade as one that has converged.

Series	Says
`pace_cache_stores_refused_foreign_generation_total`	A `STORE` whose value names a canonicalization generation this build does not implement. A rise names a rolling upgrade in progress; it stopping names one that finished. Flat at zero unless the fleet spans a `CompileValueVersion` bump, so any rise is a real event. The node's twin is `pace_node_cache_requests_refused_foreign_generation_total` — watch both.
`pace_cache_stores_refused_not_a_compile_value_total`	A `STORE` whose value is not a compile value at all. Distinct from `pace_cache_malformed_values_total`, which is the memcached and Redis path's answer for a set or stream contradicting its own flags.
`pace_cache_stores_failed_total`	A `STORE` that reached the engine and could not be written: a full disk, a read-only mount, a backend refusing writes. The only arm of this surface that is about this machine rather than its clients.
`pace_cache_malformed_values_total`	A stored value that did not decode as the type its flags claim. Not a disk signal — the store is intact and every record still verifies.
`pace_cache_frames_refused_unsupported_version_total`	A frame naming a protocol version this build does not serve. During a rollout this tracks the rollout; afterwards it names a machine nobody upgraded.
`pace_cache_frames_refused_unknown_opcode_total`	A frame naming an opcode this build has no row for. A scanner, or a client speaking something else entirely at this port.
`pace_cache_frames_refused_payload_too_large_total`	A frame refused before its payload was read, for declaring more than the session cap or the verb's own ceiling allows. The cheapest probe there is.
`pace_cache_frames_refused_malformed_payload_total`	A payload that did not decode as the verb it named — two ends that agree on the framing and disagree about what goes inside it.
`pace_cache_frames_refused_unauthenticated_total`	A verb attempted on a connection that never authenticated: a client that has not learned it needs a credential, rather than one presenting a wrong credential.
`pace_cache_frames_refused_malformed_credential_total`	An `AUTH` payload that did not decode. Counted apart from ordinary malformed payloads because garbage aimed at the credential verb is what a scanner produces.
`pace_cache_credentials_rejected_total`	A credential presented and refused. Any sustained rise names somebody guessing.

Fleet dispatch¶

The fleet scheduler's series. pace-kv runs no scheduler, so on a daemon these are flat at zero; they move on a pace-node that serves the scheduler, and Distributed compilation is where they are explained one by one.

Series	Says
`pace_dispatch_leases_granted_total`	Work is being distributed.
`pace_dispatch_leases_released_total`	Clients are reporting their jobs done.
`pace_dispatch_leases_no_worker_total`	The fleet is misconfigured — workers are up but nobody matches.
`pace_dispatch_leases_no_capacity_total`	The fleet is too small -- or, where a worker reads `limited-by: compile-budget`, its compiles hold its `--compile-inflight-max` instead: raise that on it.
`pace_dispatch_leases_withdrawn_total`	The fleet is unavailable — slots free on paper, machines busy elsewhere.
`pace_dispatch_leases_all_excluded_total`	The fleet is unreachable from its clients — every matching worker was on the asking client's exclusion list. A network problem, not a missing toolchain.
`pace_dispatch_leases_duplicate_total`	Duplicate-work suppression is doing its job.
`pace_dispatch_leases_reclaimed_total`	A machine went away mid-job and the keys it was building were freed.
`pace_dispatch_leases_unauthorized_total`	A lease token this cluster never signed was handed back.
`pace_dispatch_leases_released_late_total`	A compile outran the lease timeout and reported back too late. Read as a fraction of `released_total`; a steady fraction means the lease bound is too short for this site's slowest translation unit.
`pace_dispatch_leases_malformed_total`	A client asked for a lease naming its key, toolchain or toolchain label in bytes that are not UTF-8, and was refused.
`pace_dispatch_leases_field_too_long_total`	A client asked for a lease with a key, toolchain fingerprint or toolchain label longer than a scheduler records, and was refused.
`pace_dispatch_worker_registrations_total`	Workers registering. A steady rise means heartbeats are not arriving.
`pace_dispatch_worker_registrations_malformed_total`	A peer named its toolchain, endpoint or version in bytes that are not UTF-8 and was refused.
`pace_dispatch_worker_registrations_field_too_long_total`	A peer registered with a fingerprint, endpoint, version, toolchain label, display name or codec list longer than a scheduler records, and was refused.
`pace_dispatch_node_announcements_field_too_long_total`	A machine announced itself with an endpoint or version longer than a scheduler records, and was refused. An overlong condition field is refused earlier, when the frame is decoded, and is not counted here.
`pace_dispatch_worker_endpoint_mismatch_total`	A worker was admitted while advertising an endpoint whose host is not the address it connected from.
`pace_dispatch_workers_expired_total`	A machine stopped heartbeating and was dropped.
`pace_dispatch_workers_withdrawn_total`	A machine re-surveyed, found it no longer serves a toolchain, and retired that registration itself.
`pace_dispatch_frames_refused_unsupported_version_total`	A peer built against another release of the wire.
`pace_dispatch_frames_refused_unknown_opcode_total`	A frame naming a verb no build has.
`pace_dispatch_frames_refused_not_permitted_total`	A verb that exists and is served on a different port.
`pace_dispatch_frames_refused_truncated_total`	A frame whose header disagreed with what arrived. Never sum it with a malformed-payload series.

Process¶

Series	Says
`pace_uptime_seconds`	Seconds since this process started. A gauge; a reset is a restart.
`pace_build_info`	The build this process runs, as `pace_build_info{version="0.4.1"} 1`. The value is always 1 and the version is the label, so a query joins on it rather than reading a number.
`pace_metrics_surface_absent`	How many counters this build's catalogue names that belong to surfaces this process does not serve, so nothing here can ever write them. A gauge, and a nonzero reading is the healthy state of any binary that serves only some surfaces. Those series are omitted from the scrape rather than rendered as a zero. Do not alert on this; it is the row above that is a fault. It moves with the configuration: every catalogue row is attributed to a surface, and `pace-node` derives its surfaces from the components it was told to run -- so a node started `--slots=0` reports the worker's refusal counters here rather than as zeroes claiming nothing was ever refused, and one with no `--listen-raft` does the same for the Raft peer wire. A reading that CHANGES after a flag change is therefore expected, and the daemon reports `0` because it claims every surface.
`pace_metrics_catalogue_skew`	How many counters this build's metrics catalogue names that its sink has no slot for. A gauge, and zero on any correctly built process -- a nonzero reading means the catalogue and the sink were compiled against different versions of the counter enum, so the binary is inconsistent and every counter at the affected ordinals is missing from this scrape rather than reading zero. Alert on `> 0`.

This table is checked against the exposition the daemon actually renders — see MetricsDocumentation_test.cpp, which fails in both directions.

Kubernetes¶

A minimal Deployment with probes and a Prometheus scrape annotation:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pace-kv
spec:
  replicas: 1
  selector:
    matchLabels: { app: pace-kv }
  template:
    metadata:
      labels: { app: pace-kv }
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9259"
        prometheus.io/path: "/metrics"
    spec:
      containers:
        - name: pace-kv
          image: pace-kv:latest
          args: ["--bind=0.0.0.0", "--metrics", "--metrics-bind=0.0.0.0", "--requirepass=$(CACHE_SECRET)"]
          env:
            - name: CACHE_SECRET
              valueFrom: { secretKeyRef: { name: pace-kv, key: requirepass } }
          ports:
            - { containerPort: 6674, name: cache }
            - { containerPort: 9259, name: metrics }
          livenessProbe:
            httpGet: { path: /healthz, port: metrics }
            periodSeconds: 30
          readinessProbe:
            httpGet: { path: /healthz, port: metrics }
            periodSeconds: 10

sccache¶

The original use case still works — point sccache at the cache port:

export SCCACHE_REDIS=redis://:secret@host:6674   # with auth
export SCCACHE_MEMCACHED=tcp://host:6674         # binary protocol, no auth

On MSVC and clang-cl, one sccache cache must not be shared between checkouts

sccache preprocesses MSVC and clang-cl with /EP, which emits no line markers, so the text it hashes to find a cache hit carries no paths at all — while the /showIncludes stream it replays on that hit carries the absolute paths of the checkout that stored it. Two checkouts sharing one cache therefore record dependencies pointing into each other, after which editing a header in the checkout you are building rebuilds nothing: the build stays green and the objects are stale.

Measured on this project — a second worktree recorded 1097 dependency edges into the first and none into itself — and reproducible in two compiles: with sccache 0.14.0 and MSVC 14.51, a second directory took a cache hit from the first, and its /showIncludes named the first directory's header.

What is exposed is narrower than "sharing a cache". It is an incremental build across checkouts at different absolute paths. A clean build has no dependency graph to corrupt, and checkouts that all sit at the same path replay paths that are correct — a CI fleet is normally both, and this is not a reason to take the cache away from one. GCC and Clang are not exposed at all: their preprocessed output carries the paths, so two checkouts never share an entry to begin with — the same two compiles under g++ 14 were 0 hits and 2 misses.

A pace-kv-backed sccache is definitionally one cache shared by every checkout and every machine pointed at it, so the developer machines behind it are exposed even where CI is not. On MSVC and clang-cl, use pace-cc instead: it rewrites a hit's paths into the consuming checkout before replaying them, and refuses a hit whose replayed dependency is not there — which is why it does not have this failure mode, and why its entries are portable across checkout paths on purpose.