Visual agents can learn to use evidence returned by tools
Reframe: Observation-Aligned Multi-Teacher On-Policy Distillation for Agentic Visual Reasoning · arXiv:2611.12886
In plain words
Reframe is a way to train visual models that answer questions about images and videos using tools. It rewards useful crops or video frames, and sometimes hides the original visual input after a tool call so the model must use what the tool returned. Later, it fixes weak tool actions during training before teacher feedback continues.
Across the paper's image and video tests, Reframe's average scores were above a standard teacher-guided version, though it did not lead on every test.
The research
A visual model can call a tool and still ignore what it returns; later training then builds on weak evidence. That makes it hard to tell whether a tool call improved reasoning or merely added steps. Reframe tests whether a unified image-and-video model benefits from the evidence its tools retrieve.
Reframe trains image and video specialists to use retrieved evidence, then repairs weak tool actions so a shared student can learn from informative observations.
The first stage trains image and video specialists with rewards for grounded crops or frame selections, and sometimes hides the original input after a successful call.
The second checks a student's tool actions during training, repairs weak ones from annotations, and resumes teacher feedback from the returned observation.
Across seven image tests and seven video tests, the resulting student's averages exceed vanilla MOPD's, which consolidates the same specialists. This supports the framework as a whole; it does not show that every individual test improves.
The controlled observation probe covered 191 multiple-choice questions, with one sample per question in each condition and the original image visible throughout. It measures responses to supplied image content, not whether a model can independently acquire the right crop.
What the evidence shows
- Make tool observations matter
Reframe is a two-stage post-training framework that trains image and video specialists to acquire useful visual evidence and use the observations returned by tools. It then repairs deficient tool actions during a shared student's training, so teacher feedback can continue from a more informative observation.
This targets two linked problems: a model may ignore what a tool returns, and later training cannot recover evidence its earlier action failed to obtain.
- Higher averages after repair
The higher averages indicate a modest advantage for Reframe in image and video understanding when each modality is averaged separately. Since MOPD scores higher on some individual tests, this comparison supports an overall gain across the tested sets, not uniform improvement.
- A stronger tool-related gain
The increase in image-test scores when tools were enabled was larger with grounding rewards and original-input dropout than with correctness-only training.
Tool-free accuracy changed little with the former setup, suggesting the advantage is related to tool use; because both rewards and dropout changed, the comparison cannot identify which produced the pattern.
- Answers respond to supplied image crops
A controlled probe measures whether answers change when the image returned by a fixed crop action changes. Reframe's observation-dependence score was 75.53, versus no more than 41.93 for the other compared visual agents.
The score estimates how many errors under random or blank crops are resolved by a ground-truth crop, with an adjustment for baseline accuracy. It indicates sensitivity to supplied observations on this probe, not autonomous tool skill in general.
Deep
- Original paper authors
- Haozhi Yuan, Handong Li, Yifan Zhang, Qianshan Wei
- Submitted to arXiv
- License
- arXiv non-exclusive