Medical captions can give segmentation models lesion size
Counterfactual Analysis of Extent Leakage from Annotation-Derived Prompts in Vision-Language Medical Image Segmentation · arXiv:2611.13970
In plain words
Some medical image segmentation models receive a short caption along with an image. In several public datasets, that caption includes clues about lesion size that come from the mask the model is later scored against.
The researchers changed caption details and retrained one model with edited captions. In colonoscopy tests, the size word accounts for most of the caption-related score gain.
An image-based model that estimates size without seeing the test mask came close to, but scored below, the published model using its mask-related caption.
The research
A high segmentation score can suggest that a model finds lesions from images, even when its caption supplies clues about the answer.
In the audited colonoscopy sets, much of the caption-related gain comes from a size word computed from the scoring mask, so the protocol offers a way to distinguish image-based performance from help supplied by benchmark captions.
The authors propose an evaluation protocol that edits captions, measures their label-related information, and tests how training captions shape medical image segmentation scores.
The protocol changes one caption detail at a time for published models, comparing each image’s released caption with empty and edited versions.
In five colonoscopy test sets, the size word alone recovers nearly all of the matched-caption Dice gain over an empty caption, while removing it loses much of that gain.
Because the caption’s size word was computed from the reference mask, these comparisons support the authors’ interpretation that the reported gain partly reflects access to label-related extent information, not only image-based segmentation.
The results cover two published model architectures, with retraining limited to MedCLIPSeg on three datasets and ExtentSeg tested only on colonoscopy. The study did not test clinician-written descriptions, and the estimated-extent advantage over ExtentSeg without extent input was uncertain.
What the evidence shows
- What the protocol tests
The authors propose an evaluation protocol for detecting extent leakage: cases where a caption gives a segmentation model lesion-extent information taken from, or written for, benchmark labels. They audit published MedCLIPSeg models across colonoscopy, breast ultrasound and chest X-ray, and test RecLMIS on chest X-ray.
The question is whether caption-related score gains reflect what the image shows or extent clues supplied with the caption.
- The size word carries the gain
In colonoscopy, a size-only caption recovers 84.1% to 98.3% of the matched-caption gain over an empty caption, measured as a share of the Dice improvement. That comparison changes caption content while keeping the published model and test images fixed.
It shows that the size word accounts for most of the measured gain on these five test sets, even though the score itself measures mask overlap.
- Other images add other cues
The extent cue also matters on the QaTa-COV19 chest X-ray test set: the lung-zone phrase alone recovers most of the matched-caption gain for both tested models. Breast ultrasound is a qualified case.
There, captions can name diagnosis as well as size, and the size-only edit removes diagnosis too. The comparison therefore shows that the measured caption benefit can include more than extent.
- Training captions shape reliance
Retraining shows that caption wording during training shapes how much the models rely on captions later: removing the size word weakens the text gain, while keeping only that word preserves most of it.
The control suggests this dependence is linked to pairing images with their own extent captions, rather than simply to evaluating without captions.
- An image-based reference
ExtentSeg estimates lesion extent from the image and uses it to guide segmentation without test-mask information. It reaches 74.59 mean Dice across five colonoscopy test sets, below 77.23 for the published model using its matched caption.
That comparison mixes different models and training histories, and the published caption contains test-mask extent. The image-based method offers a reference for evaluation without that test-time clue, not a direct replacement result.
Deep
- Original paper authors
- Niran Nataraj, Ankita Das, C. Krishna Mohan, Kenji Kawashima
- Submitted to arXiv
- License
- arXiv non-exclusive