A benchmark checks the reasoning behind clinical timeline verdicts
Testimony from Timelines: Evidence-Grounded Significance Testing of Temporal Clinical Hypotheses · arXiv:2610.40619
In plain words
A claim such as “one symptom usually comes before another” needs more than a few examples. A system must decide which reports count, what would happen by chance, and which statistical test to use.
The paper introduces Strata, a test set that checks those decisions, and Art, a method that makes them explicit before testing. On 1,500 test hypotheses, Art’s Qwen2.5 setup got more verdicts right than a single-query system given the same codebook. The test covers patterns in reports, not causes.
The research
When a system says one event tends to come before another across clinical reports, how can we tell whether the verdict rests on the right records and statistical choices?
A plausible answer can conceal a mistake in either, so checking the conclusion alone leaves an important part of verification unmeasured.
Art turns each timeline hypothesis into a checked plan, retrieved records and a statistical test, while Strata scores each decision as well as the verdict.
Can a system verify a pattern across clinical reports for the right reasons, rather than merely return the right answer? The authors introduce Strata, a benchmark that scores each statistical decision, and Art, a staged method that makes those decisions explicit.
On Strata’s 1,500-hypothesis test set, Art’s Qwen2.5 setup produced more correct verdicts than a single-query system given the codebook. Its evidence and decision scores also show why verdict accuracy alone can miss errors in the reports counted and the test choices.
These results concern one test set of hypotheses generated from VAERS timelines under a released codebook. A supported pattern describes what was reported, not a causal effect; transfer to clinician-written hypotheses or templates not used to build the test set remains untested.
What the evidence shows
- Check the steps behind claims
Strata is a benchmark for checking whether systems correctly verify claims about event order or timing across clinical timelines. Its test set grades the plan, selected evidence, test statistics and final verdict, because the answer alone may look right even when the system counted the wrong reports. Art is the paper’s staged method for making those decisions explicit and checkable.
- Why a verdict is not enough
A temporal hypothesis describes a possible pattern across a population of reports, so the verifier must decide what would count as chance and which reports can witness the pattern. Strata tests whether systems make those choices as well as whether they return the right three-way verdict: supported, refuted or insufficient. Its clinical timelines come from vaccine adverse-event reports.
- Make each decision inspectable
Art separates verification into a plan, evidence retrieval and a statistical test. The model states the null, test and rule for selecting reports; the harness resolves that rule, and the model retrieves the evidence.
A deterministic function then runs the test and returns the verdict. This makes intermediate decisions available for checking instead of leaving them implicit in one answer.
- Compare full verification procedures
On the 1,500-hypothesis test set in the Original register, Art with the Qwen2.5 model pair reached 0.806 verdict accuracy, versus 0.577 for EHRSQL-C, which uses one query and receives the codebook. The score is the fraction of hypotheses with the correct three-way verdict.
Across the tested configurations, the authors also found that verdict accuracy could conceal mistakes in evidence selection and other decisions.
- Look beyond the final answer
For the Qwen2.5 pair, Art’s correct verdicts more often used the gold denominator than EHRSQL-C’s, and its full chain of evidence and test decisions was more often correct.
Removing contract resolution from Art lowered accuracy on this test setup, indicating that translating the plan’s named report-selection rule into an explicit retrieval condition contributed to its result.
Deep
- Original paper authors
- Chengyang He, Wenlong Zhang, Koduvayur Subbalakshmi, Ping Wang
- Submitted to arXiv
- License
- CC BY 4.0