Budgeted producer/consumer queues: decrement 'outstanding work' only after the consumer can no longer free memory
Follow-up on a queue-budget liveness bug (an in-flight counter that was never decremented). Two independent fixes appeared, with different decrement points: - decrement as soon as the consumer takes a chunk off the queue or returns it to its pool; - count queued AND in-progress chunks, and decrement only after the consumer has finished all work that could release budgeted memory (records written, final flush done).
The second is the safer rule when a waiting producer uses the counter to decide 'someone may still free memory, keep waiting' vs 'nobody can free memory, refuse now'. Decrementing too early lets the producer refuse while memory is about to be released (a spurious failure). Never decrementing makes it wait forever (a hang).
Practice: - Define the counter's meaning precisely: 'units of work that may still release budget'. - Decrement at the last point that can release memory, including the shutdown/final path. - Test three cases with timeouts: an active consumer, an active writer stage, and retained pool buffers (which don't become releasable). - If there are two copies of the library, review both fixes together and converge on one rule.
Independent review of one such fix: decrementing after the consumer has fully processed a chunk (not at pop) held up. A bounded red test (one wide signal so a chunk grows to ~2 MiB, a 4 MiB strict budget, a perl alarm around the run) hung 2 out of 2 before the fix and failed cleanly with the budget message in 0 s 3 out of 3 after it. Also check the side effects. If the fixed file is also described by a source-hash pin or rebuilt from a patch stack, update both in the same commit; otherwise run outputs record a hash for code that never ran.
Another trap in the same fix family: the order in which the wait predicate reads its two counters matters. One copy wrapped the check as work.canWait(failed, writer.inflight()). The writer count is a call argument, so it is read BEFORE canWait loads the outstanding-chunk count. If the consumer submits a writer job and then marks its chunk complete between the two reads, the waiter sees 0 and 0 and refuses, even though a job that will free memory is still queued. Read the outstanding count first with an acquire load (it pairs with the consumer's release decrement), then the writer count. Or pass the writer count as a callable that is invoked after the load. A deterministic single-thread test shows it: read the writer count, run the consumer step, then evaluate the predicate.