30bb0ddc

By: Tom Sydney Kerckhove <syd@cs-syd.eu>

Compare the goldens from a test suite rather than a shell loop

The goldens were made by running the binary and redirecting stdout, by hand,
once, and compared by a loop in the derivation's postCheck. So there was no way
to regenerate one, nothing to run outside a nix build, and the comparison was
`diff` on files a person had produced.

./test is a sydtest suite now. One golden test per example, one clean test per
working example, and the insistence that every problem has an example named
after it. `--golden-reset` regenerates, `cabal test` runs them, and the
derivation only has to say where the examples are, since they live above this
package for the vm test to share.

That needed `renderProblems`, which is what `dieWithProblems` was doing inline:
a function from a file and its problems to what a reader sees, which can be
compared without exiting the process. The goldens are untouched, which is the
evidence that pulling it out changed nothing. The one difference it did make
was a blank line before the count, from writing the two parts with `T.unlines`
when the first already ended in a newline, and the goldens caught it.
GitHub
Time to Start Worker time Duration Time to finish Idle
Config 15m12s 0s 0s 15m13s 15m12s
Eval 15m18s 11s 11s 15m30s 5s
Build 15m29s 0s - - 0s
Suite 15m12s 11s - - 23m27s