A model can see a failed step yet trust the agent’s record

When an agent’s record says a step worked but the current observation disagrees, which should its decision model trust? A terminal choice can turn a mistaken record into a consequential action.

The paper tests whether models use each source of information and whether internal evidence can help catch the mismatch.

EviGate reads an intermediate model state and vetoes a terminal choice when that state does not support the agent’s previous step having worked.

Researchers first varied the record and observation separately in matched phone, ALFWorld and WebShop cases. Across four models in all three domains, record cues had the larger mean effect in every model–domain configuration.

False success claims also led to consequential choices in many constructed cases, showing why the mismatch matters. A linear readout could find outcome evidence in intermediate states, so EviGate uses that signal to veto unsupported terminal choices.

With calibration examples from the matching setting, it raised live-agent task success over no gate in all three evaluations. Matched-rate random vetoes suggest selective vetoing helped in the phone simulator and WebShop; the ALFWorld comparison remained uncertain.

The live evaluations deliberately injected silent failures, and the phone agent ran in a text simulator rather than on a live Android device. The study does not establish how often mismatches occur naturally or whether these gains carry over to live phone use; the fine-tuning results were offline.