0563caa4

Let a postgresql suite have a real read replica

## Why

An application that offloads reads onto a read replica is an application whose reads can be older than its writes, and nothing here could say so. The server this library starts has no replica, so a read that must not be answered by one looks exactly like a read that may be, and the difference only shows up in production.

That is where this came from. A binary cache pointed its read-only pool at an Aurora reader endpoint and started answering 404 for content it had just accepted, because a read that decided a write was being answered by a replica that had not caught up. Every test passed, because in the tests that pool was the primary.

Wired into that project's suites, this fails **144 of the cache's existing 573 tests** and **440 of the leader's**, and turned up several bugs nobody was looking for.

## What it looks like

Writing a test against a replica should cost what writing one without costs. What a suite pays for a replica is finding out what its application does when its reads are older than its writes, not a harder test to write.

```haskell
-- without a replica                        -- with a replica
spec = persistPostgresqlSpec migration $    spec = persistPostgresqlReplicatedSpec migration $
  it "..." $ \pool -> do                      it "..." $ \pools -> do
    i <- runPostgresqlTest pool $ insert p      i <- asOfLongAgo pools $ onPrimary pools $ insert p
    mp <- runPostgresqlTest pool $ get i        mp <- onReplica pools $ get i
    mp `shouldBe` Just p                        mp `shouldBe` Just p
```

Same shape, no outer resources, no config to pass. The one thing the replica version says that the other does not is the one thing that is actually different about it: which part of the test happened long ago.

## What it is

A real standby, rather than a stand-in for one. `pg_basebackup` into a directory of its own, a replication slot, hot standby, torn down with the suite. It replays the same write-ahead log a production replica does, in the same order, and refuses writes because postgres refuses them, not because a test says so.

`recovery_min_apply_delay` is what makes being behind a fact rather than a race: without it a local standby follows fast enough that whether a read sees a just-committed write depends on the machine. `defaultReplicaConfig` holds it twenty milliseconds back, which is what a replica keeping up looks like. The tempting setting is a large one, on the grounds that it is the harsher test, and it is the milder one: a replica far enough behind to have replayed nothing answers every read with a clean miss, and a clean miss is the case an application handles. The partial-lag case is what found the interesting bugs.

## `asOfLongAgo`

Most of a test is arranging what, in production, was arranged long before the request under test: an account registered last week, a build that finished this morning. A replica has all of that. Only what happens after the arranging is recent enough for lag to be a question.

Without somewhere to draw that line, a suite pointed at a replica fails on its own setup, for a reason its production counterpart would never meet, and says nothing about the request it was written for.

It waits rather than lifting the standby's apply delay while it catches up. Lifting would be quicker, and `ALTER SYSTEM SET recovery_min_apply_delay` does work on a standby, but how far behind the standby is belongs to the cluster: every test running at that moment would get a replica that had caught up, and a test whose subject is what happens when it has not would quietly stop testing anything. Waiting costs the configured delay at worst, because the delay is measured against each record's commit timestamp rather than from when replay reaches it.

## What it needed from the rest of the library

`tmp-postgres`' `defaultConfig` asks for `wal_level = minimal`, which writes too little to the log for anything to replay, and at which postgres refuses to start with a nonzero `max_wal_senders`. So the server configuration becomes the caller's to choose, through `postgresqlServerSetupFuncWith` and `persistPostgresqlAdminSpecWith`, with `replicationPrimaryConfig` for the case this is all for. The existing entry points are unchanged and defined in terms of the new ones. `testDatabaseSetupFunc` exposes the connection options alongside the pool, because a second connection to the same database needs to know which database that is.

## Tests

`StandbySpec` asserts the five things that make this a replica rather than a copy: a write is not visible while it is behind, it is visible once caught up, `asOfLongAgo` settles the arranging, a write *through* the standby is refused by postgres (SQLSTATE 25006), and the same test body works on `unreplicatedPools`, which is what an application configured without a replica has.

The second of those caught a bug in the first draft of the wait: it waited on `pg_current_wal_lsn()`, the write position. Under `synchronous_commit = off`, which is how a server meant for tests is usually run, a commit is acknowledged before its log record has been written out, so that position can be behind a transaction that has already committed, and the wait was waiting for something that had already happened. `pg_current_wal_insert_lsn()` counts the records still in the buffers. It passed at half a second of lag, where the gap is too small to land in, and failed at twenty.

Total

32m36s

Created 1 month ago Started 1 month ago Finished 1 month ago

Timeline