How simulator report delivery changes agents’ answers

When an agent gets a wrong answer from a tool, the answer alone does not reveal the failure: the tool may not have produced the needed value, the agent may not have seen it, or it may have used it incorrectly. Where does report retrieval affect this chain?

The authors compare inline simulator reports with section-by-section retrieval, then trace whether agents produce, see, and correctly use the needed value.

The key question is where an agent loses accuracy when a long simulator report is retrieved in pieces: in producing the right value, seeing it, or using it correctly. The authors checked 158 PHREEQC questions against verified answers and traced these stages in agent logs.

Across pooled runs, retrieval lowered GPT-5.1 and GPT-5.2 accuracy relative to inline reports. A replay with GPT-5.2’s simulation and history fixed also found lower accuracy with retrieval.

Instructions to use the simulator’s printed value improved evidence use, supporting the authors’ view that what agents read contributed to the loss.

The study covers one simulator, 158 verified questions, and the tested model and API settings. The full-agent conditions ran at different times, and the authors do not establish which model property explains the differences or whether they persist with explicit reasoning settings.