Voice isolation improves overlap, but still harms lone callers
The Voice on the Line · arXiv:2610.35272
In plain words
A call transcriber needs to hear the caller while ignoring other people nearby. This paper presents live voice-isolation models that use the call audio itself to pick out the main speaker and suppress the rest.
In tests, the released model made fewer word errors than VoxLock on long meetings and calls with another voice. It was level on Krisp recordings, but made more errors on clean calls with one speaker, so keeping other voices out can still come at a cost to the caller.
The research
Can a live call transcriber suppress another person’s speech after a long interruption and still keep the caller’s words when that caller is alone? Short evaluation clips can miss both failures, so the question is whether a model can handle longer calls without trading away the lone caller.
The authors build causal voice-isolation models that select a caller from live audio, then test how well they suppress other speakers while preserving that caller.
The authors built causal voice-isolation models that identify a caller from the call audio and suppress other voices as they arrive. The release adds a causal WavLM branch trained inside the model.
On dev-4, which combines three meeting development sets and clean calls, it scored 10.55% word error rate before a later training change, compared with 11.44% without WavLM.
Against VoxLock running live on test recordings, the release made fewer word errors on long meetings and calls with another voice, but more on clean calls with one speaker.
The split supports a bounded answer: the model handles several overlapping-speech tests better, yet caller preservation remains a weakness.
These results come from the paper’s specified test recordings and speech recognizers.
The development comparison of models with and without WavLM does not fully specify whether all other training conditions and starting states were identical; the authors also report that training-run variation can exceed small development-set differences.
What the evidence shows
- Two ways isolation can fail
The paper presents causal voice-isolation models for transcription: they use a call’s audio to identify its primary speaker, preserve that voice, and suppress others. The challenge is two-sided.
Other speakers can leak into the transcript after a long stretch of speech, while a model that suppresses too aggressively can remove a caller who is alone.
- Select the caller from the call
The models work on incoming audio without waiting for future sound. They guide speaker selection with a vector built from the call itself: the live system starts without a sample, then chooses the loudest three seconds it has heard, fixing that choice once the call reaches 15 seconds.
The release also trains a causal WavLM-Large branch inside the model, using its pre-trained weights.
- A development-set improvement
The authors compared the medium model with and without WavLM on dev-4, a development measure combining three meeting sets and clean real calls. Word error rate is the percentage of reference words recognized incorrectly.
Before a later training-window change, the WavLM version scored 10.55%, compared with 11.44% without WavLM. This is a result on that development aggregate, not a guarantee across test sets.
- Test outcomes depend on the call
On test recordings, the release made fewer word errors than VoxLock running live on long meeting windows and calls with another voice. It was level on Krisp recordings and worse on clean calls with one speaker.
On VoxLock’s own long meeting windows, the interval over callers still crossed zero. The results therefore show improvement in some overlapping-speech conditions alongside a remaining cost for lone callers.
- Training cuts partly help starts
The authors traced much of the damage at the start of lone-caller recordings to a training gap: simulated calls did not begin with someone already speaking. In 55 of 120 development calls that opened mid-speech, the model before the training change removed 38% of the caller’s first three seconds.
Cutting some training windows from longer calls exposed the model to that situation. On clean test calls, this change lowered word error rate from 7.27% to 5.58%; VoxLock live scored 2.92%.
On those test calls, the release removed 7% of the caller’s opening speech, down from 23% before the change. The results support the authors’ cold-start explanation and show a partial remedy, while the clean-call error rate remained above VoxLock live.
Deep
- Original paper authors
- Husein Zolkepli
- Submitted to arXiv
- License
- CC BY 4.0