Interaction traces help judges assess software as people use it
In plain words
A program can look correct in its code or a picture and still fail when someone uses it. This paper introduces Interaction-as-a-Judge: one computer-use agent clicks, types, and observes an application, then a judge uses that record to assess the application or suggest fixes.
In comparisons of website preferences, Gemini 3 Flash matched users’ choices more often when it received interaction records than when it received code and a screenshot. The result is specific to that judge and the tested websites.
The research
When a website or game compiles, can an automated judge tell whether it behaves correctly or matches what people prefer? Static code and a screenshot can miss problems that appear only during use, so the question is whether evidence from interacting with the application improves those judgments.
Interaction-as-a-Judge has a computer-use agent operate an application, then gives its trace to a judge for evaluation or feedback that can guide repairs.
On ArenaPreferences, Gemini 3 Flash matched users’ pairwise website preferences 69.1% of the time with iterative interaction records, versus 67.6% with code and a screenshot. The judge assessed both candidate orders, and the reported result averages agreement across them.
On game requirements labeled broken, GPT 5.6 Sol identified them with 71.1% accuracy using interaction, compared with 63.0% using code alone. Together, these comparisons show gains on the tested preference and correctness tasks.
These findings come from filtered pairwise website comparisons and labeled requirements for games; they do not establish performance across all applications or judges. For preference judgments, GPT 5.6 Luna showed no statistically significant difference between approaches.
Analysis
Deep
- Original paper authors
- Wayne Chi, I-Hung Hsu, Li Chen, Wei-Lin Chiang, Ion Stoica
- Submitted to arXiv
- License
- CC BY 4.0