1cfe16c0

By: Tom Sydney Kerckhove <syd@cs-syd.eu>

Make controls sparser and cheaper to run

A control is a no-op mutation that is expected to survive, so unlike a real
mutation its child cannot stop at the test that kills it: it runs every
covering test it is given, to the end. That makes the cheapest kind of
mutation to produce the most expensive kind to run, and on a large project
the controls of a run came to well over a third of its mutation phase while
being a fraction of its mutations.

What a control is for is noticing a suite that cannot be relied on, which the
number of controls answers rather than the number of tests each one runs. So
there is now one every hundred mutations rather than every twelve, and each is
checked against ten of its covering tests rather than all of them. Not the
ten cheapest: the fastest tests are the least likely to be the ones a suite
cannot be relied on for, so taking those every time would aim the canary away
from what it is watching for. Which ten follows from the mutation's own id, so
different controls take different tests while a rerun takes the same ones. The
cap lives in the coverage phase so that the record says what the child will
run and the budget is derived from it.

Every module still gets a control at its first mutation site, which it did not
need before: at one per twelve mutations every module of any size got one
anyway, and at one per hundred a project of small modules would get none. That
also keeps this repository's own example, whose modules are far smaller than
that, exercising controls at all.

Total

3m03s

Created 1 week ago Started 1 week ago Finished 1 week ago

Timeline