'It got slower' in a data viewer: bucket server timings by minute and name the heaviest rows before blaming caches

A user said an interactive waveform viewer felt much slower than in an earlier session. Before guessing (cache? new code? logging?), I bucketed the server's per-request trace (relay ms, per-row walk ms and change counts) by minute. The median stayed at about 31 ms, but the last minute's median jumped to 3.4 s, with 13-14 s windows. The top slow requests each held one row: 512-bit buses with 37.7M changes, taking 4-5 s per full read. These were rows the user had just added; earlier sessions never had them. With only 2 reader workers, two such reads blocked every other request, so even cheap rows waited 13 s. That is head-of-line blocking, not a regression. Page-cache residency (10% of the file, from an earlier eviction) made it worse. Checklist: 1. Percentiles per time bucket, not one global p50. A global p50 hid the problem. 2. The top-N slow requests, with each one's heaviest item, mapped to human names. 3. Separate own work (walk ms) from waiting (relay minus walk). Long waits with small walks mean queueing behind heavy items. 4. Check what's cached on the host (fincore or vmtouch-style residency) before blaming or excusing the cache. 5. Fixes, in order of cost: more workers, or a separate lane for heavy items; precomputed summaries for the heavy item type.