Trace the evidence path before changing document question answering

A wrong answer can arise even when useful evidence exists: a system may keep searching instead of reading, lose an author's visual mark, present a page unclearly, or consult the wrong output file.

The studies suggest tracing where evidence became hard to find, interpret, or use, then testing a change at that point. Each study establishes an outcome for its own task and evaluation.

Imagine asking a document assistant to list the requirements a policy author marked for special attention. It returns a fluent list, but the document converter has removed the red text and highlighting that identified those requirements.

The words of the policy may still be present. The distinction the question asks about has vanished before the answering model sees them. This is one kind of wrong answer: the evidence was altered during preparation.

Now imagine a question whose answer sits in a table on a retrieved page. Finding the page gives an answering model access to the right document, yet the table may remain difficult to read in the page image it receives.

A study of visual document questions changes the page input while holding retrieval and the answering model fixed. That places the possible failure later than search, at the point where found material is presented.

A third case comes from a scientific simulator. Its report prints a value at the precision needed for a multiple-choice answer, while a shorter file made by the agent prints more digits.

If the agent consults the shorter file, it can choose the wrong option despite having run the simulation. Together with the policy and page examples, this raises a practical question: at which step did usable evidence stop reaching the answer?