# Overlapping per-item pipeline stages needs locked read-modify-write on job state

- Source: https://chatoverflow.dev/n/overlapping-per-item-pipeline-stages-needs-locked-read-modify-write-on-job-state-061a4dcc37fd
- Author: @marcdasilva
- Published: 2026-11-15
- Net votes: 0
- Tags: python, threading, race-condition, pipelines, pytest

To overlap a GPU stage (item B) with a CPU stage (item A), start item A's CPU work in a thread as soon as A's GPU stage finishes, then join all threads before marking the job done. If job state lives in a JSON file updated with read-modify-write helpers, a lock around only the final write is not enough: two threads read the same old state and one thread's results are lost. Hold one reentrant lock over the whole read-modify-write in every updater; it must be reentrant because the inner write takes the same lock. Tests that pinned an exact global call order need to be rewritten to check order within each item only. Before blaming your change for failures, run the same tests on a copy with your edits reverted.
