VideoMemBench tests memory across video conversations
VideoMemBench: How Well Do MLLMs Remember in Video Conversations? · arXiv:2610.40131
In plain words
VideoMemBench is a collection of video conversations designed to test whether a model remembers what was discussed earlier. Some later questions point back to a scene; others build on the model’s earlier answer.
The researchers tested 25 models by giving them sampled frames and keeping their previous replies in the conversation. Even the top-scoring model had trouble overall, especially when a question depended on an earlier answer.
The research
Can a model discussing one video remember a scene it described earlier and reuse an answer it gave in the same conversation? Existing video tests ask questions independently, so they do not measure whether dialogue history helps answer a later question.
VideoMemBench links video questions to earlier scenes or answers to test whether models can retrieve and reuse what a dialogue has established.
The authors built VideoMemBench with planned, typed links between dialogue turns, automated checks for necessary and visually grounded dependencies, and expert review.
They then evaluated 25 models with the same 128 sampled frames per video, keeping each model’s earlier answers in its dialogue history.
Claude Opus 5 scored highest overall at 45.7%, the arithmetic mean of accuracy on all turns and two overlapping dependent-turn groups. For the answer-dependent measure, a turn counted as correct only when its response and all its answer-dependent ancestors were correct.
Accuracy also tended to fall as a dependent turn’s earliest parent was farther back, while replacing earlier model answers with reference answers helped three models but did not close the gap to independent turns.
The main results apply to models given 128 sampled frames per video, not the full videos. The distance pattern is an average across models and does not isolate distance from other differences among turns.
What the evidence shows
- A missing memory test
VideoMemBench is a dataset and benchmark for evaluating whether multimodal language models can retrieve and reuse information from earlier turns in a dialogue about one video. Existing video tests generally ask questions independently, so they can be answered from the video and current question without using earlier dialogue. That leaves a specific ability, cross-turn video memory, unmeasured.
- Two kinds of recall
The dataset links later questions to earlier turns in two ways. An episodic dependency requires recalling a previously discussed scene or entity; an answer dependency requires using the model’s earlier response and staying consistent with it.
This distinction lets the benchmark ask whether a model remembers what it saw in the conversation and what it said.
- Making history matter
The authors planned dependencies before generating dialogue, then used automated checks for necessity, information leakage, and visual grounding. Eight experts reviewed retained dialogues against their videos.
The result contains 326 video dialogues and 2,476 question-and-answer turns, of which 1,387 depend on earlier turns. This construction makes dialogue history relevant to the questions being evaluated.
- Models still struggle
The authors tested 25 models using the same 128 sampled frames per video and retained each model’s earlier replies as it answered in order. GPT-5.4 mini graded open-ended answers, while multiple-choice answers were scored by extracting the chosen option.
Claude Opus 5 had the highest reported overall score, 45.7%, the arithmetic mean of three accuracy measures on overlapping turn groups. The answer-dependent measure gives credit only when the current answer and all its answer-dependent ancestors are correct.
- Where difficulty appears
Across evaluated models, mean accuracy was lower for dependent turns whose earliest parent was at least five turns back than for those one turn back. In a separate control for three models, replacing earlier model replies with reference answers improved answer-dependent accuracy, but it remained below independent-turn accuracy.
These results suggest that longer gaps and earlier answer errors contribute to difficulty, without showing that either explains every error.
Deep
- Original paper authors
- Zhengdong Hu, Chao Wang, Kaituo Feng, Huanjin Yao, Mingfei Lu
- Submitted to arXiv
- License
- CC BY 4.0