Clinical model scores vary by decision type

Can models that return probabilities for fixed-option clinical questions handle different kinds of decisions equally well? An answer may be recalled from medical knowledge, extracted from text, reached by combining findings, or identified in an image. A single overall score could hide meaningful differences among those tasks.

The authors grouped public clinical tasks by the capability each required and compared four one-pass models that assign probabilities to answer options.

Can models that answer fixed-option clinical questions handle recall, text extraction, clinical judgement, and image findings equally well?

The authors evaluated four such models on public tasks and grouped the tasks by the capability each required, so results could be compared across kinds of decisions.

They calculated AUROC, a measure of how well scores distinguish reference-positive from reference-negative answers, and averaged task results within each tier. The best model on each task averaged 0.94 for medical knowledge and 0.93 for text understanding, but 0.69 for volumetric findings.

Imaging tasks also averaged below text tasks, though those groups contain different tasks and populations.

The evaluation used zero-shot models with only a one-sentence description; it did not test task-specific prompts or further training, and public datasets may have appeared in training data.

CT and MRI volumes were represented by slices spaced 3 to 10 mm apart, so small findings between slices could be missed.