54b36279

By: Tom Sydney Kerckhove <syd@cs-syd.eu>

Fork every worker from one warm evaluator

A pool of N workers evaluated the flake's shared substrate N times: each one
built its own evaluator, so it loaded the flake and forced nixpkgs, the
overlays and the package set before it could answer anything. Measured with a
warm store, that is 195s per worker on nix-ci against 262s of per-attribute
work for the whole run, and 19s per worker on sydtest against 9s.

The run now forks a substrate host before anything else. It fetches and loads
the flake once, forces what the outputs share, and from then on evaluates
nothing: it only forks workers on request, and those inherit the forced
evaluator copy-on-write. The coordinator still creates each worker's pipes and
speaks to it exactly as before; only the fork happens elsewhere, which is what
passing descriptors is for.

Three things a fork must not carry across, and what is done about each:

- The store. A daemon connection is a socket two processes cannot both speak
  on, and a local store holds an SQLite connection that SQLite documents as
  unusable in a forked child. Each worker reopens the store and points the
  evaluator it inherited at the new one.
- A background thread. Downloading starts nix's file transfer, whose thread
  does not survive a fork, so a worker that inherited one would hang on the
  first thing it asked of it. The host fetches the flake in a child that then
  exits, and refuses to fork workers at all if it ever finds itself with more
  than one thread.
- The point of the exercise. Forcing one attribute shares most of the
  derivation graph underneath it, but a flake's checks and its packages can
  come from different package sets, so the host is also asked to force one
  attribute per output type, named from a derivation the run has reported.

Nothing here fails a run: a host that cannot start, cannot fork, or dies
leaves the coordinator forking its own workers, each loading the flake for
itself, which is what it did before.

The per-worker memory ceiling now counts what a worker does not share, since
what it inherited is held once however many workers are looking at it.
NIX_CI_EVAL_TRACE reports where a run's time went, which is how the numbers
above were measured.

Timeline