Skipping a re-verification step in an encode->commit pipeline: requirements that keep it safe
Pattern: a pipeline encodes blocks on worker threads, then a committer decodes each block again to check it before publishing. Profiles show that second decode as a big CPU cost, and someone proposes a 'trusted' flag that skips it.
The review from the producer-side owner, which generalises: - No generic trust flag. A public 'addEncoded(trusted=true)' lets any caller bypass checks. Make it a distinct internal route whose handle can only be constructed by the encoder, a non-public construction path that proves provenance. - Never accept caller-supplied hashes or counts. Compute them from the exact owned rows the encoder used, and keep those rows (or a reservation) alive until commit. - Seal what was accepted: size, shape, sequence number, spec/version and continuity with the previous block. Validate the frame header, body extent and checksum, and consume each frame exactly once. - Keep the other gates (corruption, index, memory pressure, bounded close, failed publication). Skipping the decode removes only that cost, not the error handling. - Lifetimes: pointers borrowed from a callback must not outlive it. If you want to remove a copy, add a reserve/fill/publish API with capacity, width and failed-admission rules rather than pointing jobs at scratch buffers or reused ring chunks. - Don't infer 'input is already normalised' from a small sample. Make the semantics explicit (initial and final values, cuts, dump-off, block continuity). - Prerequisites: bitwise-identical output and histories with and without the shortcut, plus an independent fault-injection test (flip a byte between encode and commit) that the strict path and an offline verifier both catch. - Afterwards, profile again before choosing the next optimisation; the saving is often off the critical thread.