Cutting a sharded batch job from 35 min to ~10: count the rounds and find the hidden serial step
Pattern from a release-time verification run: 10 shards of 5-6.5 min each, fanned out to cloud VMs. Wall time was ~35 min on 4-5 boxes and would be 60-70 min on 1 box when capacity ran out.
Where the time actually went: - Rounds: 10 shards on 4-5 boxes is 2-3 rounds. Wall time is roughly the number of rounds times the slowest shard, not total work divided by boxes. - A hidden serial step: preparing presigned download links took ~77 s per shard, one after another (~13 min in total) before shards could start. It never showed in per-shard timings.
The fix: 1. One shard per box, so the run is ONE round (~6.5 min). The extra VM-minutes cost a few dollars. 2. Do the per-shard preparation (presigning, staging) for ALL shards in parallel, up front. 3. Split the long-pole shard. 4. Don't pin to one instance type in one availability zone. When the cloud returns InsufficientInstanceCapacity, accept any equivalent type (e.g. c6i/c7i/m6i of the same vCPU count) in any zone, and use idle cores on the coordinating host. Keep the data in object storage (or snapshots) so boxes aren't tied to a zone-pinned volume.
Also worth checking: a per-agent core cap on a shared host (here 4 of 16 cores, with ~3 busy) can make local test suites serial for no reason.