d0697beb

By: Tom Sydney Kerckhove <syd@cs-syd.eu>

Retry mutation overruns, and stop scoring them as kills

A mutation check failed with nothing wrong with the code, and could not be
retried without deleting things out of the nix store by hand. The run that
prompted this reported 1132 killed, 0 survived, and 2 "failed" controls, both
of them timeouts at 30s and 50s, on a 64-core machine at load average 42.

Three things were wrong.

One overrun decided a control. A control is a no-op, so a suite that kills
one is unsound however often it is repeated, and concluding that on the first
run is right. A suite that merely overruns its budget has shown nothing of
the sort: the budget is ten times a baseline measured on an idle machine.
Only an outcome that says nothing is retried now, three attempts, for
controls and ordinary mutations alike. A kill and a survival are both
observations and stand on the first run, because repeating either would bias
the score towards whatever we repeated into -- the trap sydtest's own 'flaky'
warns about, seen from the other side.

An overrun was scored as a kill. So a loaded machine raised the score: those
ten mutations counted as caught while the tests had caught nothing. It is its
own category now, named in the report and in the assert-score total, and
asserted on nowhere, because a mutation that makes the code hang runs out of
time however idle the machine and failing on that would leave no way to pass.

The verdict was then stored where it would be read forever. 'run' exited zero
whatever the controls did, so the expensive derivation succeeded while
recording a failing verdict; nix caches successes, so every later attempt
read the stored copy rather than running anything. Retrying meant deleting
the report path, which was itself pinned by a gc root. An unsound run exits
non-zero now, so nothing is stored and the next attempt runs the mutations
again.

That makes the two nix derivations mean something they only happened to mean
before: the report gates soundness and assert-score gates the score. A score
is reproducible, so running for it once and reading the answer as often as
asked is what a derivation is for. Soundness is not, because what makes a run
unsound is a suite nobody can rely on, so there is nothing worth keeping.
'runSoundness' is the name for the distinction.

Total

38m06s

Created 3 weeks ago Started 3 weeks ago Finished 3 weeks ago

Timeline