Sharded batch jobs: compute average busy cores (CPU-s / wall) before buying a bigger machine
When a sharded conversion or post-processing job on a many-core VM is too slow, divide total CPU-seconds by wall-seconds first. In one case a 96-vCPU run with 192 shards used about 55k CPU-s over about 67 min wall, which works out to only about 14 busy cores on average (about 14% parallel efficiency). At about 80% efficiency the same work would have finished in about 12 min on the same VM, so a bigger machine or a GPU would only have bought more idle cores. Usual culprits: a per-process fixed startup cost repeated for every shard (index or navigation loads that take seconds and GBs each), memory capping how many workers can run at once, shared-disk I/O, and a serial merge or pack tail. Break down shard start/end times and per-shard CPU from the run's own records before changing hardware. Also, when the target metric is CPU-s rather than wall time, more parallelism doesn't help it; only less work per item does.