A method measures and improves audio models’ use of sound
Beyond the Transcript: Measuring and Improving the Use of Acoustic Information in Audio Language Models · arXiv:2611.11573
In plain words
Audio models can answer questions about a recording by using its words, even when the question is really about sound, such as a speaker’s emotion or which word was stressed.
This paper introduces a way to estimate what the recording adds beyond a transcript, then uses it to inspect and train two models.
The training improved the two models’ average results on questions about sound and structure, while their average results on word-based questions stayed close to where they started. The test covered selected tasks, and the repair was tried in only those two models.
The research
A correct answer about emotion, speaker identity or emphasis does not necessarily show that a model heard those properties; the words may have supplied the answer.
Measuring the recording’s contribution changes how accuracy on these tasks can be interpreted, and the two-model repair results show that targeted training can improve average performance on acoustic questions while keeping average word-based performance close to its starting level.
Beyond the Transcript estimates what sound adds beyond words, then trains adapters in two models’ layers to improve answers about acoustic properties and structure.
The method compares answers to the same questions using the original recording, a synthetic reading of its transcript, and transcript-based inputs.
Across attribute and structure tasks, recording-gain differences account for about half or more of the accuracy gap between the open-model mean and Gemini 3.1 Pro.
Probes then find relevant cues readable in early layers of two open models, and adapters trained in layers where readout declines improve their average attribute and structure accuracy.
This supports the view that training can improve use of acoustic information, while the probe results do not measure how much of the improvement has that cause.
The recording-gain measures depend on a Whisper transcript: S also reflects differences between natural and synthetic speech, while H also reflects how a model combines text and audio.
The repair was tested in two models, and the supervised probes cannot measure how much improvement came from using information already present rather than making it more readable.
What the evidence shows
- Scores can hide sound use
Beyond the Transcript is a method for estimating how much an audio language model benefits from the recording beyond its words. It addresses a problem with accuracy scores: questions about emotion, speaker identity or emphasis may be answerable from wording alone.
The method compares answers to the same question under different audio and transcript inputs, so it can separate word-based performance from measured recording gain.
- The recording explains part
The recording-gain comparison shows that about half or more of Gemini’s accuracy advantage on attribute and structure tasks coincides with a larger measured benefit from the recording.
This suggests the models differ not only in overall accuracy, but also in how much they benefit from sound beyond the transcript.
- Cues appear inside models
The probe comparison shows that relevant acoustic cues can be read from early layers even when direct answers are weak. Because probes receive task labels that the models do not, this gap alone cannot show that the models fail to use information available to them.
- Targeted training improves answers
The adapters are added in layers where probe readout declines, and a content anchor is intended to keep answers on content questions close to the base model. After training, average accuracy improves on attribute and structure tasks while remaining close to baseline on lexical tasks in both models.
- The anchor limits a tradeoff
Removing the content anchor in a separate training comparison lowers lexical accuracy in both models, while the acoustic-task gains remain similar or larger. The anchored version retains 84–92% of the no-anchor version’s increase in recording gain.
Together with the probe results, this supports the interpretation that the repair helps the models use acoustic information, while leaving open how much comes from better use versus making that information easier to read.
Deep
- Original paper authors
- Linkai Peng, Baorian Nuchged, Yuyang Yao
- Submitted to arXiv
- License
- CC BY 4.0