In plain words
When a coding agent changes a website, the requested feature can work while another page or interaction breaks. ProductWebBench is a test suite for frontend coding agents that checks both the requested work and website behavior already established during the task.
In staged tests, the suite keeps checking earlier passing requirements as the agent makes later changes. Among 674 changes that passed all requested-work checks, 28.2% still failed a separate conservative check.
That shows a gap in what completion checks catch, though it does not prove each failure was newly caused by the change.
The research
When an agent changes one part of a website, how can we tell whether the requested feature works while established routes and interactions still hold? Passing checks for the new request may not answer that question, because shared pages and workflows can be affected by the same change.
ProductWebBench is a benchmark that tests frontend coding agents by checking requested changes alongside website behavior already established in the same repository.
ProductWebBench evaluates frontend coding agents by checking a requested change together with website behavior established during development. In staged tests, the harness rechecked earlier passing requirements after each later edit, then compared requested-work completion with separate checks on the final workspace.
Among 674 Change runs passing every requested-work check, 190 (28.2%) still failed a conservative check. This shows that passing the requested task did not ensure those additional checks passed, though a failed check alone does not show the change caused it.
These rates describe the frozen checks and evaluated runs. Some verifier judgments were wrong, and the untouched-repository comparison relied on initial captures from a different environment; the authors say evidence of agreement across environments is supportive, not conclusive.
Analysis
Deep
- Original paper authors
- Jiajun Wu, Jian Yang, Yaxin Du, Shukai Liu, Linzheng Chai
- Submitted to arXiv
- License
- CC BY 4.0