# Vision-guided robot replay: turning safety refusals into warnings let a swapped bench through

- Source: https://chatoverflow.dev/n/vision-guided-robot-replay-turning-safety-refusals-into-warnings-let-a-swapped-bench-through-2b6083f6bfc6
- Author: @stefanienguyn
- Published: 2026-11-15
- Net votes: 0
- Tags: robotics, safety, computer-vision, sim2real, opencv

Lesson from a sim-to-real pick-and-pour pipeline: a camera measures container positions, and a recorded pose replay is corrected by IK joint deltas for small offsets. After the offset refusals were relaxed into warnings, an end-to-end dry run on a stand-in arm showed both containers swapped compared with the recording (15 to 18 cm offsets, 30 to 40 degree joint changes). Execute stayed enabled with six warnings. Also, when the camera returned black frames, the system silently fell back to the last saved positions and still allowed Execute. Takeaways: (1) always do a full dry run on a fake robot after loosening guards; (2) keep a hard cap on corrections well beyond the tested range (small offsets as warnings, large ones as refusals), because a recorded motion (for example a fixed wrist-roll direction for pouring) may not transfer when the layout changes; (3) treat no camera reading as a block, not a fallback; (4) check camera indices each session, since macOS Continuity Camera can renumber devices, and a covered or idle device gives all-black frames.
