In plain words
Audio descriptions tell blind and low-vision viewers about important visual events. For them to work in a video, they need to fit between speech and add useful details.
The authors created a collection of full videos with descriptions written by sighted and blind professionals, plus a way to assess both what a description says and how it sounds alongside the video.
In tests with one model, planning how many words fit into each pause reduced overlap with speech on validation videos. A small listener study found a slight overall preference for the professional descriptions.
The research
How can audio descriptions help blind and low-vision viewers follow user-generated videos if they leave long silences or speak over dialogue? Judging isolated clips or scripts can miss these problems because the descriptions need to fit the sound of the whole video.
MTG-AD-Eval scores descriptions as spoken in full videos, measuring their content and fit around speech and silences.
The authors test whether inline audio descriptions can be judged for both what they say and how they fit into a video's sound. They create MTG-AD, full-length videos with professionally co-authored descriptions, and MTG-AD-Eval, which assesses descriptions as rendered speech alongside the video.
In validation videos, Gemini-2.5 Flash had less speech overlap when its prompt explicitly planned each pause's duration and word budget, given expert guidelines and timestamped speech. Across test videos, model strengths differed by measure, and a small listener study found a slight overall preference for professional descriptions.
These results support measuring timing and content together, but do not establish a universal best model or listener preference.
The listener comparison involved seven blind and low-vision participants and 28 video evaluations. The framework measures six of ten expert-guideline dimensions, and its timing results use speech synthesized at one fixed rate.
What the evidence shows
- The full-video challenge
Audio descriptions give blind and low-vision viewers information about visual events in videos. Inline descriptions must fit into natural pauses without repeating what can already be heard or running into speech.
Judging a short clip or script by itself can miss how a sequence sounds, including whether descriptions leave long silences or overlap dialogue. The paper asks how to assess both the content and timing of descriptions across full user-generated videos.
- A dataset and evaluation framework
The authors introduce MTG-AD, a dataset of 433 English user-generated videos with descriptions co-authored by sighted and blind professionals. They also introduce MTG-AD-Eval, a framework that renders timestamped descriptions as speech and measures their timing alongside content and writing quality.
By evaluating the rendered audio in the full video, the framework can measure speech overlap and remaining quiet gaps as well as what the descriptions convey.
- Content needs timing
In a prompt comparison on validation videos, Gemini-2.5 Flash produced more speech overlap when expert description guidelines were added without timestamped speech. With the timestamped transcript and guidelines already provided, explicitly planning the duration and word budget for each silent gap reduced mean overlap from 21.34 to 12.77 seconds across videos. This shows that content guidance alone does not ensure descriptions fit the available pauses.
- Trade-offs and listener evidence
On the test videos, nine multimodal models used the same gap-planning prompt, but no model led on every measure: content scores, use of available silence, and lower speech overlap did not all coincide. In a separate small study, seven blind and low-vision participants slightly preferred professional descriptions overall.
A scoring approach that judged scripts against video and speech transcript also aligned with participant descriptive ratings, although this evidence does not establish broad listener preferences or validate every quality dimension.
Deep
- Original paper authors
- Subhashini Venugopalan, Jenny Tan, Taylor Roper, Jimmy Tobin, Anton Kast
- Submitted to arXiv
- License
- CC BY 4.0