03cd2202

By: Tom Sydney Kerckhove <syd@cs-syd.eu>

Let a postgresql suite have a real read replica

An application that offloads reads onto a read replica is an application
whose reads can be older than its writes, and nothing here could say so.
The server this library starts has no replica, so a read that must not be
answered by one looks exactly like a read that may be, and the difference
only shows up in production.

That is where this came from. A binary cache pointed its read-only pool
at an Aurora reader endpoint and started answering 404 for content it had
just accepted, because a read that decided a write was being answered by
a replica that had not caught up. Every test passed, because in the tests
that pool was the primary.

So: a real standby rather than a stand-in for one.
postgresqlStandbySetupFunc runs pg_basebackup into a directory of its
own, creates a replication slot, and starts the copy in hot standby, torn
down with the suite. It replays the same write-ahead log a production
replica does, in the same order, and refuses writes because postgres
refuses them, not because a test says so.

recovery_min_apply_delay is what makes being behind a fact rather than a
race: without it a local standby follows fast enough that whether a read
sees a just-committed write depends on the machine. Twenty milliseconds
by default, which is what a replica that is keeping up looks like. A
larger delay tests less despite sounding harsher: a replica far enough
behind to have replayed nothing answers every read with a clean miss, and
a clean miss is the case an application handles.

persistPostgresqlReplicatedSpec is persistPostgresqlSpec with a replica,
and is meant to cost about as much to write: a test takes ReplicatedPools
instead of a ConnectionPool and has onPrimary, onReplica and awaitReplica
to reach either end. unreplicatedPools is the same database through both,
for the application configured without one.

Two things this needed from the rest of the library. tmp-postgres'
defaultConfig asks for wal_level = minimal, which writes too little to
the log for anything to replay and at which postgres refuses to start
with a nonzero max_wal_senders, so the server configuration becomes the
caller's to choose through postgresqlServerSetupFuncWith and
persistPostgresqlAdminSpecWith. And testDatabaseSetupFunc now exposes the
connection options alongside the pool, because a second connection to the
same database, which is what the standby pool is, needs to know which
database that is.

One thing deliberately missing: how far behind the standby is belongs to
the cluster, not to a database. Tests sharing a server share their
standby's lag, so a test cannot pause replay or change the delay for
itself alone without stalling every other test's setup. Waiting is the
one control that is safe to hand a single test, so awaitReplica is the
only one there is.

StandbySpec asserts what makes it a replica rather than a copy: a write
is not visible while it is behind, it is visible once caught up, and a
write through the standby is refused by postgres with SQLSTATE 25006.
That second test earned its keep immediately. The first draft of the
catch-up wait waited on pg_current_wal_lsn(), the write position; under
synchronous_commit = off, which is how a server meant for tests is run, a
commit is acknowledged before its log record has been written out, so
that position can be behind a transaction that has already committed, and
the wait was waiting for something that had already happened.
pg_current_wal_insert_lsn() counts the records still in the buffers. It
passed at half a second of lag, where the gap is too small to land in,
and failed at twenty milliseconds.