In plain words
To test whether a computer chose the right place on a screen, researchers often check whether its click falls inside a marked box. That can mark a useful click wrong if the same action works outside the box, or a useless click right if it lands on an inactive part inside.
This paper introduces a way to judge clicks by what happens when they are made. On a tested code-forge site, about one in five clicks rejected by box scoring still reached the requested result across three tested systems.
The research
When a system is asked to click an interface element, researchers often treat a click outside its labeled box as wrong. But that score cannot tell whether the click still did the requested action.
It remains unclear how much this grading rule distorts comparisons between systems, especially when scores are close.
The paper introduces an evaluation method that judges a screen click by its result and checks whether model predictions achieve the target outcome.
On a code forge and an online shop, the probe checked three grounders’ predictions against mapped target outcomes. Of clicks rejected by box scoring, 17.6%–20.5% still reached the outcome on the code forge and 29.7%–38.1% on the shop, using template instructions.
A blind audit also found equivalent clicks in static screenshots, especially web images. Together, these results support correcting box scores for missed successes; the screenshot estimate covers rejected clicks only.
Live execution covered a self-hosted code forge and shop, while screenshots could only be judged visually; that audit cannot detect accepted clicks with a different outcome. The separate reward study found no stable advantage only for the tested 3B model and training schedule.
What the evidence shows
- Why boxes miss clicks
Outcome-based graphical user interface grounding evaluation is a method for checking whether systems that predict screen locations actually produce the requested action. The familiar box rule counts a point as correct only if it falls inside one marked target.
Yet another point may trigger the same action, and some points inside the box may do nothing. The paper asks how much this rule misreads the errors that remain in interface-clicking systems.
- Build the outcome judge
The method executes clicks on live pages and records their results, building a map of points that produce the target action.
It then executes a model’s predicted point and compares its result with the target’s. For public screenshots, which cannot be clicked, the authors use a blind human audit based on the same outcome definition.
- Find mismatches on live pages
On the tested Gitea code forge, 17.7% of 4,970 live targets disagreed with box boundaries. On the Django Oscar shop, 31.2% of 1,224 targets on 56 qualifying pages disagreed.
Those pages met a threshold where at least 95% of rechecked points repeated their outcome. A target counted once if it had an equivalent click, an inert region covering at least 10% of its box, or a working click outside it.
- Check actual model clicks
Mapping rule differences is not enough to show that model scores change, so researchers executed predictions from three strong 7–8B grounders on the same live items. Among clicks the box rule rejected, 17.6%–20.5% still achieved the target outcome on Gitea, and 29.7%–38.1% did so on Oscar. These are separate model shares under template instructions, not pooled results.
- Extend the audit to screenshots
Static ScreenSpot-v2 tests whether a system can locate an interface target from an instruction and screenshot; its usual accuracy is the share of predicted points inside annotated boxes. In a blind audit of rejected clicks from 11 models, researchers found that 6.4% of 360 sampled cases with clear judgments were equivalent overall, with 14.4% on web screenshots versus about 1% on mobile and desktop. Crediting estimated equivalents raised model scores by 0.2–1.8 points.
- Locate the supported correction
The correction is in the range of close reported differences: across 58 literature-coded comparisons of proposed models with the strongest prior model of the same size in the same table, the median absolute gap was 1.15 points.
A separate three-seed reward study found no stable gain over box reward when a 3B model was trained with equivalent-click credit. The evidence therefore supports a scoring correction more clearly than a training benefit.
Deep
- Original paper authors
- Xiaokang Yuan, Binhui Yao, Luwei Yang, Ruimin Chen, Zeng Wang
- Submitted to arXiv
- License
- arXiv non-exclusive