ChronoReplay trains video models to answer spatial questions
ChronoReplay: One Answer Interface Is Not Enough for Streaming Spatial Intelligence · arXiv:2610.34283
In plain words
Video questions can ask for different things: which side an object is on, how far away it is, or when it appeared. ChronoReplay is a way to train one vision-language model on all three kinds of answers while keeping each answer with the video history available at that point.
In tests of categorical questions, multi-turn spatial state, and timestamped events, the trained model scored higher than the same official starting model under fixed observation access. The results cover the paper's tested indoor scenes.
The research
If a video model learns to answer one kind of spatial question, will it also handle questions that ask for a distance or a time?
The paper's single-family diagnostic shows uneven transfer across these readouts, raising the question of whether a shared model can learn them together from the history available at each question.
ChronoReplay trains one vision-language model on categorical, distance, and time answers tied to the video history available for each question.
OVO-S tests categorical spatial reasoning, OST tests multi-round state and relations, and UCS tests timestamp-causal questions.
With observation access fixed, added training raised the same Qwen3-VL-8B model's scores from 48.98 to 53.89 on OVO-S, 50.52 to 63.93 on scene-disjoint OST, and 45.33 to 52.77 on all 8,114 UCS questions.
Because each comparison keeps its observation and scoring protocol fixed, the results support gains across these three tested forms of spatial reasoning. OVO-S's Overall score is the unweighted mean of accuracy across 12 task families; the OST score is assigned by its official evaluator.
The evaluation covers indoor scenes, and the researchers did not audit high-stakes uses. The single-family diagnostic also compares training groups with different task coverage, so it cannot show that answer format alone caused the observed transfer pattern.
What the evidence shows
- One history, different questions
ChronoReplay is a post-training method for vision-language models that combines categorical, distance, and time supervision for questions about video history. The researchers ask whether training on one spatial answer form transfers to other forms.
OVO-S-Bench tests categorical reasoning such as object permanence and mapping; its Overall score averages accuracy across 12 task families. OST-Bench tests state and relations during multi-round exploration, while UCS-Bench tests timestamp-causal questions about motion and position.
- Single-family training transfers unevenly
The single-family diagnostic shows why the question matters. Categorical-only training scored 58.03 on OVO-S but 47.48 on OST, below the official model's 50.52 on OST.
Trajectory-only training scored 51.03 on OST, while full ChronoReplay scored 63.93. These results suggest uneven transfer across readouts, but the single-family variants differ in source data and task coverage, so the comparison cannot attribute the pattern to answer format alone.
- Scores rise across three tests
The researchers compared trained ChronoReplay with the official Qwen3-VL-8B model from the same starting checkpoint, keeping observation access fixed in each comparison. OVO-S accuracy averaged across its 12 task families rose from 48.98 to 53.89.
The official evaluator's score on the scene-disjoint, multi-round OST test rose from 50.52 to 63.93. On all 8,114 timestamp-causal UCS questions, the score rose from 45.33 to 52.77.
- Controls test two design choices
Controls probe whether the recipe's parts matter in the tested settings. With joint data and other listed conditions held fixed, ChronoReplay's mean score across the three tests was 56.86, above 55.57 for natural turn weighting and 54.10 for ordinary token training.
In paired OST runs, chronological replay scored 2.41 points higher on average than cyclically shifted replay. That shift also changes context before each target, so it tests chronological construction as a whole.
Deep
- Original paper authors
- Yihao Hu, Yitong Lin, Pan Wang, Zhihao Wen, Hang Wang
- Submitted to arXiv
- License
- arXiv non-exclusive