Created Started Finished
Retry mutation overruns, and stop scoring them as kills ## The bug A mutation check failed with nothing wrong with the code, and could not be retried without deleting things out of the nix store by hand. The run that prompted this was nix-ci's `mutation-leader-background`: ``` Killed: 1132 (of which timed out: 10) Survived: 0 Controls failed: 2 Failed controls (mutation testing may be unsound) ``` Nothing survived. Both "failed" controls were timeouts, at 30s and 50s, on a 64-core machine at load average 42. The ten timed-out mutations were scored as kills. Three things were wrong. **One overrun decided a control.** A control is a no-op, so a suite that *kills* one is unsound however often you repeat it — sound on the first run. A suite that merely *overruns its budget* has shown nothing of the sort: the budget is `max defaultTimeoutMicros (10 * baseline)` and the baseline is measured on an idle machine. **The verdict was then stored where it would be read forever.** `run` exits zero whatever the controls did, so the expensive mutation derivation *succeeded* while recording a failing verdict. Nix caches successes, so every later attempt re-read the stored report rather than running anything. Retrying meant `nix store delete` on the report path — which was itself pinned by a GC root, so it could not be deleted at all. **An overrun was scored as a kill.** `tallyGroups` incremented `tallyKilled` for `OutcomeTimedOut`. So a loaded machine raised the mutation score: those ten mutations counted as caught while the tests had caught nothing. ## The fix **Retry only an outcome that says nothing.** A kill and a survival are observations and stand on the first run; repeating either biases the score towards whatever you repeat into. That is the trap sydtest's own `flaky` warns about, seen from the other side — retrying a survivor until something fails scores the mutation as caught on the strength of an accidental failure, which is exactly the flakiness `flaky` assumes tests have. An overrun is not an observation, so re-running it replaces a non-observation with a real one and favours neither outcome. Three attempts, for controls and mutants alike. **An overrun that survives all three is its own category**, not a kill, with its own line in the report. **An unsound run exits non-zero** even without `--fail-fast`, so nothing is stored and the next attempt runs the mutations again. No nix change was needed: a failing derivation is not cached. Survivors keep the old behaviour deliberately — they are reproducible, so doing the expensive run once and letting the cheap `assert-score` gate re-read it is right. `runSoundness :: MutationRunReport -> RunSoundness` names that distinction. ## The tests Three regression tests, each falsified by undoing the corresponding half (6 passed / 2 failed with both undone): - `MutateSpec` "gives a mutation that only overran its budget another go": a stand-in suite overruns a 200ms budget once and is prompt afterwards, so it survives only if the overrun was retried. - `MutateSpec` "counts a mutation that kept overrunning as timed out, not as killed": a suite that always overruns must land in `timedOut`, not `killed`. - `AugmentedManifestSpec` covers `runSoundness` directly. ## On reusing sydtest's own combinators Worth recording, since it was the first thing to check. `flaky`/`flakyWith`/`withRetries` annotate `TestDefM` — spec definitions. The engine beneath, `runSingleTestWithRetries`, is welded to `TDef`, `TestRunResult`, `ProgressReporter` and `HList externalResources`, and its unit is a single leaf test in-process. The driver's unit is a child OS process whose exit is classified into a `MutationOutcome`, and the mutation child itself runs a *filtered* spec forest (the covering tests for one mutation), so the three levels do not line up. `Test.Syd.MutationMode.Common.retryingIO` is closer — it is the repo's existing retry idiom — but its `Left` carries only a `String`, and this needs the final attempt's outcomes on the failing path. Generalising it to `IO (Either e a)` would fit, at the cost of an `Either a a` at the call site, which is not clearly better than the six-line recursion. Note that `runSingleTestWithRetries` carries an explicit `-- Don't retry on timeout` branch, i.e. sydtest retries failures and never timeouts. This change retries timeouts and never failures. Those are consistent once you take the direction of bias into account — sydtest is avoiding accidental passes, this is avoiding a non-observation — but it is worth knowing they differ. ## Things to check manually - **Three attempts costs 3x on genuinely hanging mutations.** Mutating a loop bound produces a real hang that overruns all three times, so the slowest mutations get three times slower. The retry is load-bearing for controls (whose overrun cannot be mutation-caused, and which fail the run) and merely sharpens the score for mutants (whose overrun plausibly is mutation-caused, and which no longer fail the run). Fewer attempts for mutants than controls would be a reasonable change. - **`assertAllKilled = false` (report-only) mode** now fails when a control fails, where before it produced a report. - **`report.txt` changed shape**: `Killed: N` / `Timed out: M` as separate lines, rather than timed-out nested under killed. Anything parsing that output needs updating. - The mutation score now *drops* on a loaded machine rather than rising. That is the honest direction, but it does mean a score comparison across machines is not meaningful.