Created Started Finished
Count a timed-out mutation as killed again Fixes a mistake in #148, which I merged. ## What was wrong A mutation easily turns a loop into one that never ends, and a suite that never finishes is a suite that **noticed** the mutation. That is what a kill means. #148 stopped counting those as kills and then failed the run over them, so caught mutations were marked uncaught and the check went red for them. nix-ci's `mutation-leader-background` shows it both ways. Before (with #148 as merged): ``` FAIL: 0 surviving, 0 uncovered, 21 timed out out of 1132 mutation(s). Killed: 1111 Timed out: 21 Survived: 0 Controls passed: 88 Controls failed: 0 ``` Those 21 are real hangs in the worker's retry and announce paths; several had sat out a full 134 or 161 second budget three times over, which is a hang and nothing else. After (this branch, via `--override-input`): ``` PASS: All 1132 mutation(s) accounted for. Killed: 1132 (of which timed out: 19) Survived: 0 Controls passed: 88 Controls failed: 0 ``` The count moved between runs, 21 then 19, which is reason enough on its own not to fail a build on it. ## Why #148's reasoning was wrong The argument for taking the kill away was that a loaded machine times out too, so crediting it flatters the score. The retry answers that, and is the part of #148 that was right: an overrun is retried three times, so one that survives that is the mutation's doing rather than the machine's. With the retry in place, counting it as a kill says something true. Removing the retry and keeping the kill would have been unsound; this keeps both. ## What changes - `tallyGroups` credits the kill again. - `isMutationFailure` leaves timeouts alone, so they do not stop a group or trip fail-fast. - `assert-score` does not fail on them, and `total` goes back to `killed + survived + uncovered` since timeouts are inside `killed` again. - The report reads `Killed: N` / ` (of which timed out: M)` as before. - `runFoundFailure` is removed. It existed only because a timeout tripping fail-fast could abort a run the driver then reported as finished; a timeout no longer trips it. ## What #148 keeps - **The retry.** An overrun gets three attempts before it is believed. - **`runSoundness` and the non-zero exit on a failed control**, so an unsound verdict is never written to a build artefact that later attempts read instead of re-running. That was the original bug, and the same nix-ci check confirms it fixed: 88 controls passed, none failed, where two had failed on timeouts before. ## Versions Bumped forward rather than rewriting #148's versions out of history, since they are already on `master`: - `sydtest` 0.32.0.0 → 0.33.0.0 - `sydtest-mutation-driver` 0.3.0.0 → 0.4.0.0 - `sydtest-mutation-runtime` 0.1.2.0 → 0.2.0.0 (major: `runFoundFailure` is removed) ## Tests Two now pin the correct behaviour, and both would have caught the mistake: - `MutateSpec` "counts a mutation that kept overrunning as killed, and as timed out" — asserts `killed == 1` and `timedOut == 1`. - `AssertScoreSpec` "passes a timed-out mutation, which is a kill" — asserts `PASS: All 5 mutation(s) accounted for.` ## Things to check - `sydtest` 0.32.0.0 and `sydtest-mutation-runtime` 0.1.2.0 are on `master` with behaviour this undoes. If either was published anywhere, 0.33.0.0 and 0.2.0.0 are the ones to use. - Whether `runFoundFailure` is worth keeping as API despite having no caller now. I removed it; it is a one-line revert if you would rather keep it.