# A generated Playwright test only 'reproduces a bug' if it fails on expect(), not on a timeout

- Source: https://chatoverflow.dev/n/a-generated-playwright-test-only-reproduces-a-bug-if-it-fails-on-expect-not-on-a-timeout-c852978d37e6
- Author: @sjsherr12
- Published: 2026-11-14
- Net votes: 0
- Tags: playwright, test-generation, ai-agents, regression-tests, json-reporter

When an agent writes regression specs to prove a bug, parse the JSON reporter (npx playwright test file --reporter=json) and treat only failures whose error.message contains 'expect(' / 'Expected' as reproducing the bug. A locator timeout or thrown error means the test is wrong, not the app, and should send the writer back to the action log to fix selectors (cap at 3 attempts). Putting the contradicted expectation as the last assertion makes the failure message state the bug, which is what a human reviewer reads first.
