In plain words
When a tool-using agent does something risky, its final answer may not reveal whether the action was tried, completed, or stopped, or what events support that account. Different agent systems also record events in different ways.
AgentTriage turns those records into a repeatable account that cites the events behind each claim. If required evidence is missing, it says it cannot make the diagnosis.
In tests across five runtimes, it captured all applicable acquisition runs, and its diagnoses were fully correct on 0.858 of 120 adjudicated cases under a strict joint scoring rule.
The research
When a tool-using agent takes a security-sensitive action, can someone independently check what happened from the runtime record? A final answer may not show whether the action completed or which evidence supports the account, and runtimes record events differently. The paper asks how to make those diagnoses replayable across systems.
AgentTriage turns recorded agent events into deterministic, evidence-cited security diagnoses and abstains when required evidence is missing.
AgentTriage turns runtime records into a diagnosis that can be recomputed and checked against cited events. Researchers tested whether it could both cover different runtime formats and correctly account for what happened.
Across five pinned runtimes, it captured all 159 applicable acquisition runs; on a separate set of 120 adjudicated cases, its grounded joint diagnosis score was 0.858.
A case earned that score only if its action lifecycle, core security fields, policy outcome, applicable dangerous arguments, and required claim evidence were all correct. The diagnosis evaluator also found high required-evidence coverage and a low unsupported-claim rate.
Together, these measures show the system can produce evidence-accountable diagnoses in the tested setup.
The evaluation covers five pinned runtimes, tested scenarios, a fixed policy oracle, and their recorded evidence. A repeatable diagnosis is not necessarily correct if relevant events were never recorded; missing required evidence leads to abstention.
Analysis
Deep
- Original paper authors
- Jie Ying, Xutao Zhang, Ningyuan Chen, Mengce Zheng, Jun Li
- Submitted to arXiv
- License
- arXiv non-exclusive