Created Started Cancelled
Load the flake once, in the coordinator Closes #5, as an alternative to #9. Do not merge both. Same goal: stop every worker from loading the flake and forcing everything its outputs share. Different answer to "which process do the workers get forked from". #9 adds one; this one uses the process that already forks them. The coordinator builds an evaluator before it forks anything, forces the first derivation the discovery root can reach, and hands that evaluator to each worker it forks. A worker takes it over by opening its own store and pointing the inherited evaluator at it; nothing else changes about how a worker is spoken to. **726 lines against #9's 1694, and 145 of them in the coordinator against 431.** What is not here, because there is no second process: passing descriptors over a socket, a protocol to ask for a fork, workers that are grandchildren and so can be signalled but not waited for, a host that has to be replaced when it cannot fork, and the four bugs the review of #9 turned up, three of which were about that host's lifetime. What it costs, honestly: - **A flake that has to be fetched gets no benefit.** Fetching starts nix's download thread, a forked child inherits no threads, so an evaluator that fetched cannot be handed over. #9 recovers this by starting a second host once the first has warmed the store; here the coordinator is the process that got tainted, so there is nothing to replace. Workers then load the flake themselves exactly as they do today, and the coordinator's own load was wasted work. - **Forcing blocks the run.** It happens before any worker exists, where there is nothing else to do, and once per output type afterwards, which is rare and usually finds its work already done. But the coordinator is also the process servicing every worker, so while it forces, nothing moves. - **No crash isolation.** If forcing the substrate takes the evaluator down, it takes the run with it. In #9 that happens in a process the run can lose. ## Measurements `bench/compare.sh`, which this PR adds, interleaves the two builds and checks they discovered the same thing. Machine at load 24-60, which is far quieter than when I measured #9; each line is one interleaved pair. | flake | master | this | #9 | |---|---|---|---| | nix-ci | 236s, 255s | 102s, 142s | 116s, 150s | | sydtest | 20.5s, 21.7s, 25.6s | 21.8s, 24.6s, 27.3s | 22.2s, 24.0s, 26.7s | | nx-experiment | 3.4s, 3.5s, 3.5s | 4.6s, 4.6s, 5.0s | 2.1s, 2.2s, 3.8s | Read that carefully, because it says something neither #9's description nor mine assumed: - On nix-ci, where the substrate is most of the run, both branches are about twice master and within noise of each other. - On sydtest and nx-experiment, **neither branch is reliably faster than master, and both are sometimes slower**. Forcing the substrate once, before any worker exists, is serial; master forces it N times but in parallel, across workers that overlap. When the substrate is cheap relative to the attributes (nx-experiment: ~2s against 176 attributes of ~40ms), serialising it loses more than sharing it wins. - The 3.3x I reported for #9 earlier was measured on a machine at load 90-350. On a quiet machine the same comparison is 1.7-2.3x. Redundant work costs wall-clock time in proportion to how contended the machine is. Every run above discovered exactly the same outputs, derivations and dependencies as master, with two exceptions described below. ## One thing I cannot explain yet Twice out of twelve sydtest runs, this branch reported ``` error checks.x86_64-linux.shell error packages.x86_64-linux.release-to-hackage ``` where master reported them as jobs, both times while the machine was also running the other build. I have not reproduced it in eight further runs, including four under deliberate contention, so I do not have the error text. My hypothesis is that it is a worker dying twice on the same attribute, which is reported as an error by design, and that the cause is the memory ceiling now counting only what a worker does not share: a worker holding 3 GB of its own plus 2 GB inherited reads as 3 GB and is left alone, where on master the same worker would read 5 GB and be recycled. Under memory pressure that ends in the OOM killer rather than an orderly restart. If that is right it applies to #9 as well, and the fix is to stop pretending the inherited pages are free. I would not merge either branch until that is understood.