Created Started Cancelled
Retry mutation overruns, and stop scoring them as kills One commit. `nix flake check` green. ## The bug A mutation check failed with nothing wrong with the code, and could not be retried without deleting things out of the nix store by hand. ``` Killed: 1132 (of which timed out: 10) Survived: 0 Controls failed: 2 Failed controls (mutation testing may be unsound) ``` Nothing survived. Both "failed" controls were timeouts, at 30s and 50s, on a 64-core machine at load average 42. The ten timed-out mutations were scored as kills. Three things were wrong. **One overrun decided a control.** A control is a no-op, so a suite that *kills* one is unsound however often you repeat it. A suite that merely *overruns its budget* has shown nothing of the sort: the budget is `max defaultTimeoutMicros (10 * baseline)`, measured on an idle machine. **An overrun was scored as a kill.** `tallyGroups` incremented `tallyKilled` for `OutcomeTimedOut`, so a loaded machine raised the score. **The verdict was stored where it would be read forever.** `run` exited zero whatever the controls did, so the expensive derivation *succeeded* while recording a failing verdict. Nix caches successes, so every later attempt re-read the stored report rather than running anything. Retrying meant `nix store delete` on the report path — which was pinned by a GC root, so it could not be deleted at all. ## The fix **Retry only an outcome that says nothing.** Three attempts, controls and ordinary mutations alike. A kill and a survival are observations and stand on the first run; repeating either biases the score towards whatever you repeat into. That is the trap `flaky`'s own warning describes, seen from the other side: retrying a survivor until something fails scores the mutation as caught on the strength of an accidental failure. **An overrun that survives all three is its own category**, not a kill. Named in the report and counted in the assert-score total, asserted on nowhere — a mutation that makes the code hang runs out of time however idle the machine, so failing on it would leave no way to pass. **An unsound run exits non-zero**, so nothing is stored and the next attempt runs the mutations again. ## What that does to the nix It gives the two derivations a meaning they only had by accident. The report gates **soundness**; `assert-score` gates the **score**. A score is reproducible, so running for it once and reading the answer as often as asked is what a derivation is for. Soundness is not — what makes a run unsound is a suite nobody can rely on — so there is nothing worth keeping, and nix keeps nothing. Both nix files now say so; `assertMutationScore`'s control-failure branch is unreachable through that path. The split is deliberately kept. Collapsing it would make survivor failures uncached and retryable too, at the cost of the cheap re-read of a reproducible verdict and of `.report.<lib>` being buildable when the score is bad. Happy to do that instead if you would rather. ## Tests Four, each falsified by undoing the corresponding half: - `MutateSpec` "gives a mutation that only overran its budget another go" — a stand-in suite overruns a 200ms budget once, is prompt afterwards, and survives only if retried. - `MutateSpec` "counts a mutation that kept overrunning as timed out, not as killed". - `AssertScoreSpec` "counts a timed-out mutation in the total and says it went unmeasured" — `total` was `killed + survived + uncovered`, which was right only while timeouts lived inside `killed`; splitting them out made a pass claim to account for mutations it had left out. - `AugmentedManifestSpec` covers `runSoundness`. ## On reusing sydtest's own retry combinators `flaky`/`flakyWith`/`withRetries` annotate `TestDefM`, and the engine beneath them, `runSingleTestWithRetries`, is welded to `TDef`, `TestRunResult`, `ProgressReporter` and `HList externalResources` at single-leaf-test granularity. The driver's unit is a child OS process, and the mutation child itself runs a *filtered* spec forest, so the three levels do not line up. `MutationMode.Common.retryingIO` is closer, but its `Left` carries only a `String` and this needs the final attempt's outcomes on the failing path. Note `runSingleTestWithRetries` carries an explicit `-- Don't retry on timeout`: sydtest retries failures and never timeouts, this retries timeouts and never failures. Consistent once you take the direction of bias into account, but worth knowing they differ. ## Things to check manually - **Three attempts costs 3x on genuinely hanging mutations.** A mutated loop bound overruns all three times. Load-bearing for controls (whose overrun cannot be mutation-caused, and which fail the run); for ordinary mutations it only sharpens a score that no longer fails anything. Fewer attempts for those would be reasonable. - **`report.txt` changed shape**: `Killed: N` / `Timed out: M` as separate lines. Nothing in-tree parses it (it is only copied and existence-checked), but anything out of tree would need updating. - **`assertAllKilled = false` (report-only) mode** now fails when a control fails, where before it produced a report. - The score now *drops* on a loaded machine rather than rising, so comparing scores across machines is not meaningful.