In plain words
Medical guidelines change, and a model may notice that a recommendation changed without carrying every important consequence into its answer. ChronoEvo is a test built from 1,709 verified changes to guideline recommendations.
It gives models earlier recommendations and, in the main test, evidence from updated guidelines, then checks what their answers reflect.
For Qwen3-4B, a GPT-5 evaluator judged 90.8% of answers to recognize the change, but only 58.5% to include all its important consequences. The test shows a gap in these responses; it does not explain the model's internal reasoning.
The research
When medical guidance changes, is it enough for a model to recognize the revision, or must it also carry every clinically important consequence into its recommendation?
ChronoEvo tests these as separate response outcomes, addressing a gap that a general judgment of medical correctness may not reveal.
ChronoEvo tests whether models carry updated guideline evidence into recommendations, separating recognition of a change from fully incorporating it.
In the full-evidence test, Qwen3-4B received earlier recommendations and updated guideline evidence. A GPT-5 semantic evaluator judged whether responses recognized the change, were clinically correct, and included all clinically important consequences. Recognition was 90.8%, clinical correctness 80.2%, and full incorporation 58.5%.
Because recognition and correctness were judged more often than complete incorporation on these same responses, the results show that noticing a revision does not ensure that its consequences reach the recommendation. The gap was also found across the study's 14 evaluated models.
These results concern model responses to guideline revisions when updated evidence is supplied. The study does not test whether models can retrieve the right guideline, choose the version for a particular patient or time, or maintain updated knowledge during prolonged real-world use.
Analysis
Deep
- Original paper authors
- Min Zeng, Zaifu Zhan, Kai Yu, Rui Zhang
- Submitted to arXiv
- License
- CC BY 4.0