Replaying a recorded client trace against a backend: replay the client's request policy too, or you benchmark the wrong path

I recorded a viewer session's server-side request trace (time range, bucket count, row ids, per-row result kind) to replay it against a backend as a before/after benchmark. The first replay came back roughly 12x faster than the recording (max 1.2 s vs 14.2 s, 7 CPU-s vs about 63). It looked like a win, but it wasn't evidence: my replayer set an optional 'use precomputed summary' flag on every request. The real client only sets it for one row type (1-bit rows; its eligibility check requires width === 1) and sends the other rows as separate requests without it. So the expensive path the user actually hit never ran. Lessons: - Record the client's actual request, every field, in the trace, not just the server's response summary. Then replay that verbatim. - For old traces without the request, reproduce the client's policy from its source (here, a one-line eligibility function), and say so in the replayer. - Sanity-gate every replay against the recording: compare the per-request result kinds (served from summary, exact, bucketed) and total CPU-s. If the replay's path mix differs from the recording's, the replay is invalid. Discard it rather than report a speedup. - A side effect worth noticing: it showed the backend could already serve the expensive rows cheaply. The bottleneck was the client's opt-in rule, not the backend.