A tracker helps video models keep object identities through a shuffle
In plain words
Video models can recognize an object in a frame but lose track of which one it is after objects move or look alike. This paper introduces Track, Then Bind, a training-free method that pairs a frozen video-and-text model with a tracker.
The model identifies the target when it is visible, the tracker follows it, and the model answers from a written description of its tracked state.
On real cup-shuffling videos, the method answered more object-identity questions correctly than the same model answering directly. The result is limited to the tested videos and setup.
The research
When an object is hidden among look-alikes, can a video model still answer where that same object ends up? The difficulty is that recognizing an object in one frame does not ensure the model will preserve its identity after the evidence disappears. The paper asks whether identifying it earlier and tracking it separately can help.
The method binds a target to a numbered track when it can be identified, then answers from the tracked state rather than re-reading the video.
The authors tested whether separating recognition from tracking could help a model answer questions about one object after it is hidden or moved. Track, Then Bind asks the model to identify the target when it is recognizable, lets a tracker carry that identity, then answers from the tracked state.
On real cup-game videos and questions about object consistency in real videos, the method scored higher than direct answers from the same model. The comparison supports the approach on those test sets, although the methods used different execution engines.
These results come from the paper’s evaluated video question sets and one frozen configuration. The direct-answer comparisons use a different execution engine for Track, Then Bind, so they do not isolate the inference procedure from that difference; tracker errors also remain.
Analysis
Deep
- Original paper authors
- Di Yang, Xiaochang Li, Muyang Liu, Yufei Wang, Yanhai Xiong
- Submitted to arXiv
- License
- arXiv non-exclusive