Author: Tom Sydney Kerckhove <NorfairKing@users.noreply.github.com>
Committer: GitHub <noreply@github.com>
Download the existing history in batches before syncing (#15) * Download the existing history in batches before syncing Appendful's server-side read is unbounded: a fresh client sends no max-synced, so the server materialised every command the user has into one Map and JSON-encoded it before writing a byte. With a large history that takes longer than warp's handler timeout, the connection is reaped, and nginx reports the upstream close as a 502. Rather than truncate appendful's read, which is not safe (the client derives its cursor from max(server_id) and its own uploaded rows take the highest ids, so a truncated read plus an upload in the same request skips rows), add a separate endpoint for the bulk case and leave appendful for the deltas. POST /download takes the greatest server id the client has and returns the next batch above it in ascending id order, capped at the server's download-batch-size (default 1024). An empty batch means the client is caught up, so the client never needs to know the batch size. 'bevel sync' drains that endpoint before running the appendful sync, which then only has the recent delta to move. Also adds a (server_user, id) index, which is what the download query wants. * Generate the commands in the download tests, and hardcode the batch size The download tests were built on a hand-written Command repeated with a counter. They now take generated commands, with only the length of the list constrained so that it always spans more than one batch. The batch size is no longer a setting. It sits in Env at 1024, the way envHashDifficulty is hardcoded at 10, so the tests can still run against a small value without exposing a knob nobody needs. --------- Co-authored-by: Tom Sydney Kerckhove <syd@cs-syd.eu>