A game commentator learns when to ask for unfamiliar names
StreamDuo: A Live Commentator and a Research Agent for Unseen Games · arXiv:2611.01120
In plain words
A live game commentator may know when to speak but not the names of characters in a game it has never seen. StreamDuo pairs the commentator with a research agent that looks at the video and searches that game's wiki when asked.
The commentator learns when a lookup could help and how to use an imperfect answer. In tests on ten games it had not trained on, StreamDuo named more of the entities mentioned by human casters than a comparable commentator trained on the same source game.
The research
A live commentator that gets the timing right can still make a game hard to follow if it cannot name what is on screen. StreamDuo's results suggest that looking up game-specific names during the stream can improve naming on games absent from training, while reducing carryover from the training game.
StreamDuo pairs a live commentator with a wiki-search agent and trains the commentator to ask when it needs game-specific knowledge and use the answers.
In the cross-game test, both commentators were trained on Baldur's Gate 3 clips and evaluated on 2,470 clips from ten other games.
StreamDuo's entity recall was 0.116, versus 0.082 for Proact-VL retrained on the same clips; recall counts human-mentioned entities the model also names in each clip's first 126 words.
The higher score supports the claim that StreamDuo can recover more game-specific names in this test. Its training procedure and access to the research agent both differ from Proact-VL's, so the comparison does not attribute the gain to one change.
The comparison covers a commentator trained on Baldur’s Gate 3 and tested on ten unseen games. It does not show how StreamDuo performs with other training games or on other unseen-game sets.
What the evidence shows
- Timing travels, knowledge does not
StreamDuo is a two-agent method for live commentary on games unseen during training. The question is whether a commentator can preserve its timing while naming the unfamiliar characters and items that make commentary specific.
Existing systems can carry over when to speak across games, but their game-specific knowledge does not transfer reliably; they may fall back on vague descriptions or names from the game they trained on.
- A lookup during the stream
The commentator watches and speaks, while a frozen research agent can search the current game's wiki when asked. It receives video frames rather than a text description, then returns up to three possible entries after a delay.
The commentator must learn when a lookup would help and how to use an answer that may contain wrong or distracting entries. This divides live perception and speaking from game-specific lookup.
- More names from the new game
The comparison suggests that StreamDuo's combined design helps the commentator name more entities from unfamiliar games. Since the comparison changes both training and access to wiki answers during the stream, it cannot show which change produced the improvement.
- Fewer names carried over
A separate measure asks how often models use names from the training game that belong to neither the test game nor its wiki. Across all test clips without a word cap, the paper reports source-game leakage of 1.0% for StreamDuo and 9.9% for Proact-VL.
This suggests StreamDuo's higher recall does not simply come from saying more names from its training game. The paper's prose and table disagree on the total mention counts, so those counts are not used here.
- Learning to use answers
A component comparison adds the second training stage to the first-stage condition, with the same inference settings. Entity recall rises from 0.092 to 0.116 on the unseen-game clips.
This is consistent with the proposed role of the second stage: helping the commentator use answers, not only request them. The comparison does not specify all training conditions held constant, so it does not isolate that stage for every possible setup.
Deep
- Original paper authors
- Zixian Gao, Penghao Luo, Xiangcheng Cao, Lixin Xiu, Ziqi Cai
- Submitted to arXiv
- License
- Not specified by source