aaf32230

Created Started Finished

By: Tom Sydney Kerckhove <syd@cs-syd.eu>

Report where a mutation run's time goes

Mutation checks routinely take hours in nix-ci and there was no way to find
out why. The driver measured each child's wall time already but threw it away
unless the child timed out, and nothing was aggregated or reported, so the
only way to guess at the cost was to watch the progress lines go by.

One wall-clock number per child would not have been enough anyway. A child
pays three separate costs, and which one dominates decides what would actually
make the run faster: process startup (exec, RTS init, reading the augmented
manifest, building the spec forest, printing output), suite setup
(around/aroundAll resources, paid once per child rather than once per suite),
and the leaf tests themselves. Only the child can tell those apart, so it now
writes its own forest time and summed leaf-test time to a file the parent
names with a new hidden --mutation-timing-output; the parent subtracts to get
the split, and clamps the child's numbers to its own wall measurement so the
parts always sum to exactly the wall time.

Both phases emit the result. The mutation phase writes timing.html and
timing.json into its report directory, the coverage phase into its
augmented-manifest directory, and each prints an abridged copy to stdout so a
build log shows the headline without burying the survivors that print after
it. The page carries the whole per-child listing, sortable by any column
(numeric columns sort on nanoseconds via data-v, so 9.0ms does not sort after
1m02s) and filterable, which is what a run with thousands of mutations needs
and a terminal cannot give. It is self-contained, so it opens from a Nix store
path with no server.

On the in-repo example this immediately says that 96% of mutation-phase child
time and 99% of coverage-phase child time is process startup, which points at
batching mutations per child rather than at making any test faster.

.gitignore had a blanket *.html, which would have left the golden pages
untracked and the sydtest-mutation test failing on missing goldens in a fresh
clone; that rule is aimed at haddock and coverage output, so test_resources is
now excepted from it.