Load the flake once, in the coordinator
Closes #5, as an alternative to #9. Do not merge both.
Same goal: stop every worker from loading the flake and forcing everything its
outputs share. Different answer to "which process do the workers get forked
from". #9 adds one; this one uses the process that already forks them.
The coordinator builds an evaluator before it forks anything, forces the first
derivation the discovery root can reach, and hands that evaluator to each
worker it forks. A worker takes it over by opening its own store and pointing
the inherited evaluator at it; nothing else changes about how a worker is
spoken to.
**726 lines against #9's 1694, and 145 of them in the coordinator against 431.**
What is not here, because there is no second process: passing descriptors over
a socket, a protocol to ask for a fork, workers that are grandchildren and so
can be signalled but not waited for, a host that has to be replaced when it
cannot fork, and the four bugs the review of #9 turned up, three of which were
about that host's lifetime.
What it costs, honestly:
- **A flake that has to be fetched gets no benefit.** Fetching starts nix's
download thread, a forked child inherits no threads, so an evaluator that
fetched cannot be handed over. #9 recovers this by starting a second host
once the first has warmed the store; here the coordinator is the process
that got tainted, so there is nothing to replace. Workers then load the
flake themselves exactly as they do today, and the coordinator's own load
was wasted work.
- **Forcing blocks the run.** It happens before any worker exists, where there
is nothing else to do, and once per output type afterwards, which is rare
and usually finds its work already done. But the coordinator is also the
process servicing every worker, so while it forces, nothing moves.
- **No crash isolation.** If forcing the substrate takes the evaluator down,
it takes the run with it. In #9 that happens in a process the run can lose.
## Measurements
`bench/compare.sh`, which this PR adds, interleaves the two builds and checks
they discovered the same thing. Machine at load 24-60, which is far quieter
than when I measured #9; each line is one interleaved pair.
| flake | master | this | #9 |
|---|---|---|---|
| nix-ci | 236s, 255s | 102s, 142s | 116s, 150s |
| sydtest | 20.5s, 21.7s, 25.6s | 21.8s, 24.6s, 27.3s | 22.2s, 24.0s, 26.7s |
| nx-experiment | 3.4s, 3.5s, 3.5s | 4.6s, 4.6s, 5.0s | 2.1s, 2.2s, 3.8s |
Read that carefully, because it says something neither #9's description nor
mine assumed:
- On nix-ci, where the substrate is most of the run, both branches are about
twice master and within noise of each other.
- On sydtest and nx-experiment, **neither branch is reliably faster than
master, and both are sometimes slower**. Forcing the substrate once, before
any worker exists, is serial; master forces it N times but in parallel,
across workers that overlap. When the substrate is cheap relative to the
attributes (nx-experiment: ~2s against 176 attributes of ~40ms), serialising
it loses more than sharing it wins.
- The 3.3x I reported for #9 earlier was measured on a machine at load 90-350.
On a quiet machine the same comparison is 1.7-2.3x. Redundant work costs
wall-clock time in proportion to how contended the machine is.
Every run above discovered exactly the same outputs, derivations and
dependencies as master, with two exceptions described below.
## One thing I cannot explain yet
Twice out of twelve sydtest runs, this branch reported
```
error checks.x86_64-linux.shell
error packages.x86_64-linux.release-to-hackage
```
where master reported them as jobs, both times while the machine was also
running the other build. I have not reproduced it in eight further runs,
including four under deliberate contention, so I do not have the error text.
My hypothesis is that it is a worker dying twice on the same attribute, which
is reported as an error by design, and that the cause is the memory ceiling
now counting only what a worker does not share: a worker holding 3 GB of its
own plus 2 GB inherited reads as 3 GB and is left alone, where on master the
same worker would read 5 GB and be recycled. Under memory pressure that ends
in the OOM killer rather than an orderly restart. If that is right it applies
to #9 as well, and the fix is to stop pretending the inherited pages are free.
I would not merge either branch until that is understood.