EN
Executable code can preserve details in video captions
CodeCap-Bench: Can Code Serve as a Good Video Caption? · arXiv:2610.37341
In plain words
A video caption usually describes what happens in words. This paper asks whether code can capture details such as where objects are, how they move, and when they change.
The authors created CodeCap-Bench, a set of structured 2D videos for testing this idea. Models write code that is turned back into video and compared with the original.
In one test on 100 selected videos, a question-answering model answered more accurately from videos made from code captions than from natural-language captions.
The research
Can a caption preserve where objects are, how they move, and how they change together without relying on lengthy or potentially ambiguous prose? The paper examines whether executable code can express those details in a form that can be rendered and compared with the original video.
The authors introduce CodeCap-Bench, a test collection that compares videos rendered from code captions with structured 2D reference videos.
The authors compare code captions with natural-language captions to ask whether code can preserve video information for later use. On 100 selected videos with annotated questions, Gemini-3.8-flash answered from videos rendered from Opus-5 code captions, from Opus-5 text captions, and from the original videos. Accuracy is the share of answers matching the provided answers.
The code-rendered videos yielded 83.02% accuracy, versus 72.64% from text captions and 83.96% from the originals. In a separate reconstruction comparison, code rendering also scored higher on appearance, movement, and keyframe structure.
Together, these results support the authors' view that code can preserve useful visual information in the tested settings.
These findings come from selected structured 2D videos and the specific evaluation setups described. The benchmark excludes videos dominated by complex 3D viewpoints, detailed textures, or dense pixel-level motion, so the results do not establish how code captions perform on those kinds of video.
What the evidence shows
- Why test code captions
CodeCap-Bench is a test collection for evaluating whether multimodal language models can describe structured 2D videos with executable code. The paper addresses a specific difficulty: words can require lengthy, potentially ambiguous descriptions of layouts, motion, and coordinated changes.
In the test, a model writes code that specifies visual elements and how they change over time; a renderer turns the code into a video for comparison with the reference.
- What fidelity means
The benchmark evaluates more than whether code runs. It compares the rendered video with its reference for overall visual similarity, object locations and timing, and retention of annotated details.
This distinction matters because a successfully rendered video can still omit or misrepresent what the reference shows. Human-reviewed annotations mark focal points and objects so the comparisons can assess static content as well as changes over time.
- Motion remains harder
The measurements reveal a specific challenge: dynamic details scored below static ones across the evaluated configurations. For GPT-6-astra with Codex at medium thinking effort, average retention was 68.61 out of 100 for dynamic details and 80.72 for static details.
This comparison keeps the model and harness fixed while comparing the two focal-point types. It illustrates the temporal-dynamics gap without implying that every model has the same scores.
- Questions test retained detail
The authors tested whether code captions preserve information useful for questions. On 100 selected videos with annotated questions, Gemini-3.8-flash answered with 83.02% accuracy from videos rendered from Opus-5 code captions, compared with 72.64% from Opus-5 natural-language captions.
Accuracy is the share of answers matching the provided answers. Answers from the original videos reached 83.96%, giving a reference point for the same question-answering model.
- Reconstruction adds evidence
A separate reconstruction comparison measured appearance, movement patterns, and keyframe structure. For Opus-5, direct rendering of code captions scored higher on all three measures than videos generated from its natural-language captions with Seedance-2.5.
The comparison supports the authors' view that executable code can preserve useful visual information in these tested settings. Since the two caption types used different video-production methods, it does not establish a universal advantage for code captions.
Deep
- Original paper authors
- Jiafu Tang, Yifan Xu, Eric Tang, Junjie Zhao, Shipeng Luo
- Submitted to arXiv
- License
- CC BY 4.0