By: Tom Sydney Kerckhove <syd@cs-syd.eu>
Make controls sparser and cheaper to run A control is a no-op mutation that is expected to survive, so unlike a real mutation its child cannot stop at the test that kills it: it runs every covering test it is given, to the end. That makes the cheapest kind of mutation to produce the most expensive kind to run. On nix-ci's leader, controls were 7% of the mutations and 39% of the mutation phase, with single controls covered by 150 and 178 tests taking two and a half minutes each. What a control is for is noticing a suite that cannot be relied on, which the number of controls answers rather than the number of tests each one runs. So there is now one every hundred mutations rather than every twelve, and each is checked against its cheapest ten covering tests rather than all of them. The cap lives in the coverage phase so that the record says what the child will run and the budget is derived from it. Every module still gets a control at its first mutation site, which it did not need before: at one per twelve mutations every module of any size got one anyway, and at one per hundred a project of small modules would get none. That also keeps this repository's own example, whose modules are far smaller than that, exercising controls at all. On the leader manifest this is 213 controls rather than 15 by the interval alone, 46 with the per-module floor, each bounded at ten tests instead of up to 934.