A video agent tracks changing goals to time its help
StreamVibe: Proactive Video Agents for Evolving User Intent · arXiv:2610.34662
In plain words
A video assistant may need to figure out what someone wants without being told directly, then keep track as the task changes. The paper introduces a test set, a trained video model, and a system that tracks active goals and progress.
In tests, adding that tracking system to the same model reduced unnecessary replies and improved answer quality in most settings, especially when goals changed. The tests cover constructed video tasks, so they do not show how the system would handle every real interaction.
The research
When a task changes, an earlier request may no longer call for the same help, while a new goal may need attention. Can a video assistant keep its replies aligned with both the user’s current intent and the task’s progress?
The paper pairs a video model that decides when to help with a state-tracking system, then tests how tracking affects sustained assistance.
The authors introduce StreamVibe-VL, a model that decides when to respond and generates help, alongside StreamVibe-Harness, which tracks active goals, task progress, and earlier replies. They test the same model during continuous video interaction with and without the Harness.
Adding the Harness reduced unnecessary responses and improved answer content in most settings, especially for changing goals. This supports the view that tracking task state can help maintain assistance over a video stream.
These results come from the paper’s video-task setting and tested evaluation suites.
The pointwise and trajectory averages cover different groups of models, and the authors note that their setting does not capture all the ways user needs and environmental conditions may change together over longer interactions.
What the evidence shows
- Test help as tasks change
The paper introduces StreamVibe-Bench, a test set for video agents that must decide when and how to help as tasks unfold. It covers explicit requests, unstated goals that must be inferred, and goals that change during interaction.
Its evaluation set contains 4,100 samples and 20,546 annotated help points. These annotations let the authors assess both whether a response comes at an appropriate time and whether its content fits the task.
- Separate local from sustained help
The authors test decisions in two ways. Pointwise evaluation checks individual timestamps using supplied interaction history, while trajectory evaluation runs through a full video using history generated by the model itself.
Across the models available in each setting, mean Timing F1 was lower in trajectory evaluation than in pointwise evaluation. The authors read this as evidence that continuous interaction brings added difficulty, though the two averages cover different groups of models.
- Track goals across a video
To address changing task state, the paper introduces StreamVibe-Harness, which tracks active goals, progress, and delivered replies, then supplies relevant context to StreamVibe-VL. The model still decides whether to respond and writes the answer.
In trajectory tests with the policy fixed, adding the Harness reduced unnecessary responses and improved content quality in most settings, with the clearest gains on changing goals.
- Check results on other tests
The authors also report results on EgoProactive, a test of proactive interaction scored with its own aggregate measure, and IPIBench, which assesses interaction behavior such as timing and understanding under its official protocol.
StreamVibe-VL is competitive on EgoProactive and gains consistently over other evaluated models on IPIBench. These results extend the evidence to two external suites, but do not establish performance across all forms of ongoing interaction.
Deep
- Original paper authors
- Chengzhi Liu, Yuxiang Lai, Yuzhe Yang, Yanqing Liu, Jiheng Hou
- Submitted to arXiv
- License
- arXiv non-exclusive