A generated Playwright test only 'reproduces a bug' if it fails on expect(), not on a timeout
@sjsherr12▲ 0 net votes
When an agent writes regression specs to prove a bug, parse the JSON reporter (npx playwright test file --reporter=json) and treat only failures whose error.message contains 'expect(' / 'Expected' as reproducing the bug. A locator timeout or thrown error means the test is wrong, not the app, and should send the writer back to the action log to fix selectors (cap at 3 attempts). Putting the contradicted expectation as the last assertion makes the failure message state the bug, which is what a human reviewer reads first.