Download the existing history in batches before syncing ## The bug Syncing on a new machine fails with a 502 when the server already holds a large history (819k commands, in the reported case). ## Root cause `Appendful.serverProcessSyncQuery` goes through appendful-persistent's `serverSyncProcessorRead`, which is `selectList (idFilter ++ filters) []`: no `LimitTo`. A fresh client sends `max-synced: null`, so the server materialises every command the user has into a `Map` and JSON-encodes the whole thing before writing a byte. That takes longer than warp's default 30s handler timeout, the connection is reaped, and nginx (`nix/nixos-module.nix`, plain `proxyPass`) reports the upstream close as a 502 rather than a 504. ## Why appendful's read is not just truncated The client's download cursor is `max(server_id)` in its own database, and the client's *own* uploaded rows are assigned the highest server ids. So a truncated read combined with an upload in the same request moves the cursor past rows the client never downloaded, and it never asks for them again. Server ids are also global across users (`ServerCommand` is one table filtered by `server_user`), so a single user's ids have gaps and the client cannot infer a safe cursor from its own data either. ## The fix A separate endpoint for the bulk case, leaving appendful to do what it is good at: - `POST /download` takes the greatest server id the client has and returns the next batch above it, in ascending id order, capped at the server's `download-batch-size` (new setting, default 1024). An empty batch means the client is caught up, so the client never needs to know the server's batch size. - `bevel sync` drains `/download` first, then runs the existing appendful sync, which now only has the recent delta to move. Appendful's read stays untruncated, so its semantics are unchanged and still correct. - The client asserts the cursor strictly advances on a non-empty batch, so a server bug cannot turn into an infinite loop. - Added a `command (server_user, id)` index, which is what the download query wants. ## The tests - `DownloadSpec` "sends no more than the batch size of commands at a time, and sends every command exactly once": drains the endpoint from an empty cursor against a server holding `3 * batchSize + 1` commands, asserting each response is within the cap, that each batch strictly advances the cursor, and that the drained result is exactly the commands that went in. - `DownloadSpec` "leaves nothing for the appendful sync to download afterwards": the composition property, that after draining `/download` a `postSync` at that cursor returns an empty `server-added`. - `Bevel.CLI.Commands.SyncSpec` "downloads a history that spans more than one batch onto a fresh client": runs the client's `download` loop against a seeded server and asserts the fresh client database ends up with everything. - `genValidSpec` and `jsonSpec` for both new types. `envDownloadBatchSize` lives in `Env` (following the `envHashDifficulty` precedent) so tests can run with a batch size of 5 instead of needing thousands of commands. ### On falsification The 502 itself is a scale and timeout symptom that I cannot reproduce in-process, so I checked the tests by breaking the fix instead: - removing `LimitTo` from the handler makes the boundedness test fail with 16 commands in one response instead of 5; - removing the client's loop recursion makes the `SyncSpec` test fail. Worth knowing: I first wrote the client test end-to-end through `bevel sync`, and it passed *with the download loop broken*, because the appendful sync silently picked up the 11 commands the download had left behind. That is why the test drives `download` directly. It is also a reminder that the two paths overlap: a regression in the download loop degrades to the old behaviour rather than failing outright. ## Things to check manually - Sync `tv` against your real server and confirm it works. At 819k commands and the default batch size that is about 800 round trips over a reused connection; if it feels slow, `download-batch-size` is the knob, and I would be interested to know what value feels right before we change the default. - The upload direction is still unbounded: `clientMakeSyncRequestQuery` selects every row with `server_id IS NULL` and sends them in one request, and nginx's default `client_max_body_size` is 1M. A machine that gathered a large backlog while the server was unreachable would 413 rather than 502. I left it out of scope here since a new machine has nothing to upload, but it is the same class of bug and the fix would be to batch the client's `added` map and loop. - `bevel sync` now makes at least two requests where it used to make one, so a server that is up for the download and down for the sync leaves the client fully downloaded but with its local commands unsent. The next sync fixes it.