Fork every worker from one warm evaluator Closes #5. Every worker built its own evaluator, so it loaded the flake and forced everything the outputs share before it could answer anything, and a pool of N paid that N times. This forks the pool from one process that has already done it, so the workers inherit it copy-on-write. ## What it does A **substrate host** is forked before anything else. It fetches and loads the flake once, forces what the outputs share, and from then on evaluates nothing: it only forks workers when asked. The coordinator still creates each worker's pipes and speaks to each worker exactly as before, and passes the worker's own ends to the host over a seqpacket socket with `SCM_RIGHTS`; only the `fork()` happens elsewhere. Three things a fork must not carry across, and what happens about each: - **The store.** A daemon connection is a socket two processes cannot both speak on, and a local store holds an SQLite connection that SQLite documents as unusable in a forked child. Each worker opens its own store and points the evaluator it inherited at it: `EvalState`'s `store` and `buildStore`, and the accessor that validates store paths during pure evaluation. The two store members are declared `const` and nix offers no supported way to replace them, so that part writes through the `const`. If you would rather not carry that, the alternative I know of is a small upstream change to nix. - **A background thread.** Downloading starts nix's file transfer, whose thread does not survive a fork, so a worker that inherited one would hang the first time it used it. The host fetches the flake in a child that then exits, and refuses to fork workers at all if it ever finds itself with more than one thread. I checked this one both ways: before the prefetch, a cold `$HOME` made the host bow out and the run fall back; after it, the same run forks warm workers. - **The point of the exercise.** Forcing one attribute shares most of the derivation graph under it, but a flake's `checks` and its `packages` can come from different package sets, so the host also forces one attribute per output type, named from a derivation a worker has actually reported. Guessing an attribute up front is not enough: on nx-experiment the first derivation the walk reaches costs 0.15s and shares nothing with the work that follows. Nothing here can fail a run. A host that cannot start, cannot fork, is busy, or dies leaves the coordinator forking its own workers, each loading the flake for itself, which is what it did before this change. ## Measurements One machine (64 cores, nix daemon, warm store), each pair run back to back so they share conditions, with an unrelated load average of 90-165 throughout: absolute times are inflated, and the pairs are what to compare. | flake | before | after | jobs | |---|---|---|---| | nix-ci | 447.9s | 137.3s | 53 | | nx-experiment @4fe5dee | 9.8s | 5.8s | 176 | | sydtest | 60.9s | 49.0s | 12 | What moved is the per-worker startup, which is what the issue is about. On nx-experiment a worker's first derivation cost 1.8-2.5s and now costs 0.03-0.09s; the whole pool of 17 spent 3.1s starting, against 7-9s for a pool of 4 before. On nix-ci, 17 workers spent 82s starting where 6 workers used to spend 1170s. What is left is the flake load, which is now paid once and is most of what remains: 89s of nix-ci's 137s, and the substrate forcing another 10s. sydtest moves least because it has 12 attributes and each costs seconds: there is little for a pool to do, and the run is nearly all flake load. That matches the issue's note that a single-worker run wins nothing. Memory, mid-run on nix-ci: a worker holds about 1.0 GB of its own on top of 1.2 GB shared with the host, where before each of them held 2.0-4.4 GB alone. ## How it is tested - A new check, `nix-ci-eval-substrate-test`: a hermetic fixture whose outputs all share one expensive value that traces when it is forced, run with `--workers 4`. It asserts that value is forced once. On `development` it is forced four times, which is the bug. - Unit tests for passing descriptors between processes, and for reading a worker's private memory out of `/proc/self/smaps_rollup`. - The existing integration golden covers the store swap end to end without knowing about it: every reported `drvPath` is queried back out of the store to build the dependency edges, so a worker whose derivations went to the wrong store, or whose store handle was shared with another worker, fails it. That test runs against a local store, and my own runs above went through a daemon, so both store kinds are covered. - Stress: the fixture with `--max-memory-size 1`, which puts every worker over the ceiling after every attribute, still reports all 40 jobs and the completion sentinel; worker recycling goes through the host like any other fork. - `nix flake check` passes. ## Deliberately not in this PR - **The pool sizing policy.** With a worker costing 19-30s and the whole run's attribute work costing 3-10s, no pool size wins: measured on `development`, sydtest is 30.1s with `--workers 1` against 44.4s with the default pool. This change makes workers cheap enough for the ramp to open up on its own, but the policy is worth revisiting on its own terms. - **A memory-aware ceiling.** Cheap workers mean the pool reaches its ceiling quickly, so on a machine with many cores and a heavy flake the limit becomes memory rather than CPU. Each worker now holds about half of what it used to (on nix-ci, 1.0 GB of its own against 2.3 GB before), but there can be many more of them. - **The flake load itself**, which is what is left on the critical path: 59s on nix-ci, 18-27s on sydtest, and nothing about it is parallel. - **`EvalState::store` being `const`.** A small upstream change to nix would let the store swap drop its `const_cast`; I would rather ask for one than keep this indefinitely.