cb4b4046

By: Tom Sydney Kerckhove <syd@cs-syd.eu>

Count a timed-out mutation as killed again

A mutation easily turns a loop into one that never ends, and a suite that
never finishes is a suite that noticed the mutation. That is what a kill
means, so the previous commit was wrong to take it away: it marked caught
mutations uncaught, and then failed the run over them. On nix-ci's
leader-background it turned 21 real kills in the worker's retry and announce
paths into a failing check, several of which had sat out a full 134 or 161
second budget three times over, which is a hang and nothing else.

The argument for taking it away was that a loaded machine times out too, so
crediting the kill flatters the score. The retry answers that, and is what
the previous commit got right: an overrun is retried three times, so one that
survives that is the mutation's doing rather than the machine's. With that in
place, counting it as a kill says something true.

So 'tallyGroups' credits the kill again, 'isMutationFailure' leaves timeouts
alone, and 'assert-score' does not fail on them. 'runFoundFailure' goes with
them: it existed only because a timeout tripping fail-fast could abort a run
the driver then reported as finished, and a timeout no longer trips it.

What stays is the part that was right: an overrun is retried, and a run with
a failed control exits non-zero so its verdict is never stored for a later
attempt to read back instead of re-running.

sydtest itself is unchanged by all this once more, so its version goes back
to where it was.

Total

2m06s

Created 3 weeks ago Started 3 weeks ago Finished 3 weeks ago

Timeline