Investigate using fix as the evaluator
Investigates replacing the nix-linked evaluator with
[fix](https://github.com/psyclyx/fix), a from-scratch parallel Nix evaluator
in Zig.
**Recommendation: not yet.** The write-up is in `docs/fix-evaluator.md`; the
measurement is reproducible with the committed harness.
## What landed
- `docs/fix-evaluator.md`: the investigation, with every number measured here.
- `bench/discover.nix`: this tool's discovery written as one Nix expression,
so any evaluator can be asked to perform it. Tagged against the
coordinator's depth bound so the two cannot drift apart.
- `bench/fix-differential.sh`: drives that expression through fix and CppNix
on real flakes and reports drvPath disagreements.
```console
$ bench/fix-differential.sh path/to/fix github:NorfairKing/autodocodec
```
No production code changed. The only edit under `src/` is the `[tag:]` comment
that anchors the harness to `kMaxAttrPathDepth`.
## What the measurements say
fix is correct where it is comparable: byte-identical drvPaths over ~110,000
nixpkgs attributes, and over every output of the flakes tried.
It is 1.4x to 5.9x faster in bulk. It is **slower** on the shape of work this
tool actually has, consistently, across repeats:
| Flake | Attrs | `nix-instantiate` | `fix` | ratio |
|---|---|---|---|---|
| this repository, pinned rev | 7 | 0.72-0.74s | 0.82-0.89s | 0.81-0.91x |
| autodocodec | 17 | 3.92-3.98s | 4.18-4.50s | 0.88-0.94x |
Our flakes have a few dozen outputs over an enormous package set, so the run
is dominated by reaching the first derivation. A faster evaluator does not
help with that, and fix's startup makes it a little worse. There is no
performance argument here to weigh the risk against.
## The risk
fix computes **wrong derivations, silently**, in two situations:
1. By default, flake sources stay in `~/.cache/fix` rather than the store, so
`self.outPath` is not a store path and everything reached through
`src = ./.` gets a drvPath nothing else will produce. Six of this flake's
seven outputs. `--read-write-mode` fixes it.
2. Even with `--read-write-mode`, Nix names a flake's source store path
`source` while fix names it after the directory, so `path:` and bare-path
references disagree where `github:` and `git+` agree. That is a relief for
the deployed case and a problem for local use and for
`test/integration/run.sh`, which evaluates `path:$here`.
Neither is caught by fix's own 80,000-derivation nixpkgs differential, because
it passes nixpkgs as a plain `--arg` path, so flake source handling never
enters it.
## The blockers for a replacement
- **No library.** `build.zig` produces executables only; the only `export fn`s
are a SHA256 helper. There is a clean embedder API underneath
(`Engine.init`/`evaluate`/`stringValue`, exercised by
`src/integration/expr_api/*.zig`), so a C ABI shim is plausible, but it does
not exist.
- **No local store.** `local`/`auto`/chroot are "not implemented yet", and
there is no way to give fix a store dir other than `/nix/store`. Every check
in `flake.nix` runs daemon-free on a throwaway store in a sandbox where
`/nix/store` is read-only, so none of them can run under fix today. This is
what keeps the differential out of CI.
- **Inputs are refetched** through fix's own fetchers rather than read from
the lock. On `validity` that meant a network round trip and
`FetchGitFailed` after 2m14s, against 5.9s from the lock.
## Questions and advice
- You chose a fork under NorfairKing for the eventual patches. I have **not**
created it, since this PR carries no patches to fix. Worth doing when the
first one exists.
- The two upstream changes worth proposing, in order: a store-directory
setting (small, unblocks our checks) and a `libfix.a` with a C ABI.
- Worth revisiting when fix has a local store, at which point the differential
can move into CI instead of being a script someone remembers to run.
## Things to check by review
- `bench/discover.nix` really mirrors `selectExpr` and the coordinator's walk.
If it does not, every number here is measuring the wrong thing. The one
deliberate difference is that a throwing attribute becomes `null` where the
tool emits an `error` line.
- Whether `docs/fix-evaluator.md` draws the conclusion you would draw.
## Things to try locally
```console
$ nix-build -E 'with import <nixpkgs> {}; callPackage /path/to/fix/nix/fix.nix {}'
$ bench/fix-differential.sh ./result/bin/fix \
git+file://$PWD github:NorfairKing/autodocodec
$ bench/fix-differential.sh ./result/bin/fix path:$PWD # should disagree
```
The last one is the finding worth seeing yourself: it exits non-zero and names
the attribute whose derivation differs.