Report a resource that fails around a group of tests
Stacked on #160, which is
stacked on #155. This is the last of the places an exception took the suite
down: a resource around a group of tests, rather than a test.
## What it does
`afterAll`, `aroundAll`, `aroundAll_` and friends run around a whole group. One
that threw took the suite down, which lost the summary and every result in it,
over a resource rather than a test. Every way a resource can misbehave is now
reported:
- **Fails after its tests** (`afterAll`, `aroundAll` on the way out): reported
where it happened, and the tests keep their results, because they ran and
said what they found.
- **Fails before its tests** (`beforeAll`, `beforeAll_`, `beforeAllWith`,
`aroundAll` on the way in): reported, and every test under it is reported as
having failed for that reason.
- **Never runs its tests**, or **runs them more than once**: reported.
`FootgunSpec` documented both of those as hangs, with its tests commented
out because running them hung the suite. Where a resource runs its tests
more than once, the first run is the one reported.
A suite with a `beforeAll_` and an `afterAll_` that both throw:
```
Tests:
✓ still runs after both
database
✗ reads a row
✗ writes a row
cache
✓ reads a key
Failures:
Res2.hs:9
✗ 1 database.reads a row
user error (could not start postgres)
Res2.hs:10
✗ 2 database.writes a row
user error (could not start postgres)
✗ database.beforeAll_
user error (could not start postgres)
✗ cache.afterAll
user error (could not flush)
Examples: 5
Passed: 3
Failed: 2
```
## How
`ResourceFailureNode` is a new `SpecTree` constructor carrying the name of the
combinator it came from and the exception. It counts as neither a success nor
a failure, because there is no test whose fault it is, so no count moves and
`Passed: 3, Failed: 2` above counts only tests. `shouldExitFail` is the only
place it tells, so a run that is otherwise green still exits non-zero.
The reporting wrapper applications in `Runner/Wrappers.hs` say which of four
things became of a resource — ran, threw after the tests, threw before them,
or never ran them — because only "threw after" has results to report next to,
and the other two need results for tests that never started.
Those invented results carry `ExpectPassing` whatever the tests around them
asked for, since a test that never ran did not fail in the way `expectFailing`
means, and `MayNotBeFlaky`, since there is nothing to retry when the resource
is gone rather than the test flaky. They are also what keeps the asynchronous
runner going: the thread collecting results waits on each test in turn, so a
subtree whose tests are never scheduled is a thread that never stops waiting.
In the asynchronous runner the thread that catches a resource failure is not
the thread that builds the result forest, so failures are collected with the
describe path above them and `insertResourceFailure` puts each one in the
forest afterwards. Two groups can share a path, in which case the first gets
it: a misplaced resource failure beats a dropped one.
## Tests
27 of them. The end-to-end shapes through both runners (after, before, never,
twice), `insertResourceFailure` on its own including the path collisions and
groups nested under a resource, `shouldExitFail` saying yes while the counts
say nothing, and a case in `sydtest-output-test`, which renders it through all
four runners and golden-tests both formats. `FootgunSpec` is tests now instead
of four commented-out cases, each against a nested runner under a timeout so a
wait shows up as a failure rather than a stuck suite.
`sydtest-output-test`'s own tidying now covers a resource failure's exception,
whose context carries a package-id hash that differs between builds and would
have broken the golden file on every rebuild.
## Version
0.36.0.0: `SpecTree` has a constructor more, so anything matching on one
exhaustively needs a case for it. `flattenSpecForest` leaves resource failures
out as it does pending tests, and `flattenResourceFailures` is what returns
them. `applySimpleWrapper`, `applySimpleWrapper'` and `applySimpleWrapper''`
are gone, replaced by their reporting versions. (`applySimpleWrapper2` was
already unused before this work, so it stays as it was found.)
## Gaps I left, deliberately
- **`--fail-fast` does not stop for a resource failure**, only for a failed
test. The run still fails. Whether a teardown that throws should stop the
rest of the suite is a decision for you: it means threading the setting into
`resourceSubtree` for the synchronous runners and putting `AbortFailFast` in
the abort var for the asynchronous one.
- **The streamed tree of `--threads 1` does not show tests that never ran**
because their resource failed: it prints as it goes, and those tests never
run. They are in the Failures section and the final report, and the
asynchronous runner does print them. Matching them means moving the
before-failure handling out of `resourceSubtree` and into the runner so it
can print.
- **An inner resource that throws still discards its test's own failure.**
`after_ (throwIO ...)` around a test that fails reports only the teardown's
exception; `applyWrapper2` has always let the wrapper's exception override
the inner result. This PR makes that inconsistent with outer resources,
which now keep their tests' results. Fixing it needs a decision about where
an inner resource failure lives, since one test has one result.
- **One commit has no regression test**: a worker could reach a result slot
that a failed resource had filled on its behalf and block on it. The window
is between a job being queued and a worker reaching it, and I could not
close on it deterministically without adding hooks to the runner. The commit
message says so.