Median-of-samples hides GC cost when benchmarking big json.loads
When timing json.loads (or orjson/msgspec loads) on multi-MB payloads with an auto-calibrated loop count, the loop count often drops to 1, so each sample is one call. Full cyclic-GC collections then land on some calls and not others: I saw orjson on citm_catalog.json alternate between 3.3 ms and 10.3 ms per call on CPython 3.9. The median reports the fast mode and drops the amortized GC cost a real server pays. Calibrate only after one warm call (the first call pays page faults and inflates dt, pushing loops to 1), and report the mean per call within a process, then take a median/bootstrap across processes.
Reproduced with stdlib json (5 MB synthetic doc, 30 single-call samples, results kept alive the way a server's caches would be): 3.10.22 median 52.7 ms vs mean 148.7 ms (max 687); 3.12.3 median 56.9 vs mean 71.7; 3.13.18 56.5 vs 73.2; gc.disable() gives about 35-39 ms. So the median really does hide the GC cost. Version caveat: what kind of GC you see depends on the version. On 3.10, 20 loads ran 4687 gen0 / 427 gen1 / 14 full collections inside the C call. On 3.12+, collections are only scheduled and run at the next eval-breaker check, so a single loads() gets at most about one collection, at the end: 19/1/0 on 3.12, 18/2/0 on 3.13. The spikes there come from an occasional gen1 pass (~230 ms traversing everything the previous loads promoted), not full collections. Count collections with gc.callbacks next to the timings, so you know which one you are measuring.