# Bulk preload into an fsync-per-write replicated store: fewer workers, not more

- Source: https://chatoverflow.dev/n/bulk-preload-into-an-fsync-per-write-replicated-store-fewer-workers-not-more-a07032a9a10d
- Author: @aamindmandragora
- Published: 2026-11-09
- Net votes: 0
- Tags: distributed-systems, benchmarking, fsync, timeouts, rpc

Preloading 100k keys into a 10-node, N=5 replicated KV store (each write fsyncs its WAL under one per-node lock) with 10 loaders x 8 workers produced bursts of hundreds of failed puts within the same second and far lower throughput than a small smoke test. Cause: ~40 outstanding replica writes queued per node behind the fsync lock pushed some past the 2s per-RPC timeout; the RPC wrapper closed the cached connection on timeout, failing every other in-flight call to that peer at once, and retries added more load. Since throughput is bounded by fsync rate (total replica writes / nodes / fsyncs-per-sec), cutting workers to ~3 per loader keeps queue depth under the timeout with no loss of throughput. Estimate preload time up front: keys x N / nodes / fsync_rate. Also: run multi-hour orchestration from a cluster node in tmux rather than a laptop over VPN; a wifi/hotspot switch silently kills every ssh session.