A world model checks whether GUI actions lead as expected
In plain words
A graphical-interface agent can reach the same login screen after a sensible action or after a hostile one. Looking at the screen alone misses the difference.
This paper introduces LGWM, a model that predicts what screen should follow an action and compares that prediction with what appeared. On a test built from identical screens reached through legitimate and hijacked transitions, its score separated the two kinds of transition. The finding is limited to that constructed test.
The research
When an agent sees a credential screen, the pixels may not reveal whether it followed a legitimate action or a hijacked one. Can a monitor check what happened by considering the action and the screen that came before, rather than judging the final screen alone?
LGWM predicts the next screen in the observation’s representation space, letting an agent check a transition by comparing prediction and outcome.
LGWM predicts the next screen’s representation from the current screen and action, then compares it directly with the observed screen’s representation.
On RSWT-Bench, which pairs identical credential screens reached through hijacked and legitimate transitions, its training-free score reached 0.987 AUC across 222 pairs.
That score ranks the hijacked case above its legitimate counterpart, showing that transition context can distinguish cases that screen appearance alone cannot. Generative GUI world models scored lower on the same test, using a separate judge to assess rendered predictions.
RSWT-Bench has 222 donor pairs, selected because each credential screen appears as both a hijacked outcome and the pixel-identical legitimate outcome of its own transition. The result does not establish performance on other GUI transitions or settings.
Analysis
Deep
- Authors
- Jiaming Zhang, Xuan Wang, Fuyao Zhang, Yang Cao, Lingjuan Lyu
- License
- arXiv non-exclusive