EN
A broader speech test and a single-pass extractor
Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes · arXiv:2610.32749
In plain words
The paper introduces RealSSA, a way to test systems that use a person’s mouth video and audio to isolate that person’s voice as a conversation unfolds. It includes varied simulated scenes and a separate set of real recordings for evaluation.
It also introduces Falcon, which separates speech at several scales in one pass and learns to distinguish the target from nearby speakers.
On RealSSA-Sim, Falcon’s average separation score was slightly higher than a causal comparison system’s; the result covers varied scenes, not every recording or system.
The research
Can a system isolate one person’s voice when the number and distance of other speakers, room acoustics and noise change, and can it be tested on real recordings as well as simulations? The paper addresses this gap in both evaluation and extraction.
The authors introduce RealSSA to test varied acoustic scenes and Falcon, a causal extractor that separates at multiple scales in one pass and trains against nearby-speaker leakage.
The authors evaluated Falcon on RealSSA-Sim, where scale-invariant signal-to-noise ratio improvement (SI-SNRi) measures separation improvement over the unprocessed mixture. Falcon scores 7.7 dB, compared with 7.6 dB for causal AV-TF-GridNet.
The scores average across varied scenes and interfering-speaker counts, so the comparison gives an overall result for this test set.
A separate control reports gains when the speaker contrastive loss is added to three architectures. On held-out RealSSA-Real recordings, training on RealSSA-Sim improved DNSMOS, an audio-based estimate of perceived speech quality, for all five tested architectures and NISQA, another such estimate, for four; AV-TF-GridNet was the NISQA exception.
For Falcon, both scores were higher after RealSSA-Sim training than after VoxCeleb2 training. Together, these findings support the paper’s approach under its tested conditions.
These findings come from the paper’s tested systems and conditions. RealSSA-Real covers about 84 minutes from 10 participants in eight environments; Falcon processes single-channel audio, and its contrastive loss uses clean near-field interferer recordings available in simulation.
What the evidence shows
- Two gaps in extraction
Audio-visual target speaker extraction (AVTSE) uses a speaker’s mouth video along with audio to isolate that person’s voice. In the paper’s causal setup, audio output advances every 8 ms without an additional future-chunk buffer, while the visual stream buffers three future frames, or 120 ms of lookahead.
The paper identifies two gaps: many existing tests fix the number of speakers and omit real recordings, while some causal systems are computationally-heavy or repeatedly run a small separator. The authors present RealSSA and Falcon to address the testing and extraction sides of this problem.
- A broader test
RealSSA pairs a controllable simulated suite with a separate set of real recordings reserved for evaluation. Its simulations vary speaker counts, near- and far-field interference, reverberation, environmental noise and sudden noise.
This gives researchers ways to examine performance across changing acoustic conditions and to test on recordings outside the simulated suite. Falcon is the paper’s causal extraction method, designed to process multiple time and frequency scales in a single separator pass.
- Difficulty and quality
The scene breakdown shows why the range of conditions matters. On RealSSA-Sim, small-meeting scenes were hardest and outdoor scenes easiest across the evaluated architectures.
The authors interpret this as evidence that near-field crowding contributes substantially to difficulty in this benchmark, rather than reverberation alone. On the aggregate test, Falcon’s improvement in scale-invariant signal-to-noise ratio over the unprocessed mixture was 7.7 dB, compared with 7.6 dB for causal AV-TF-GridNet; these scores average across the varied scenes and speaker counts.
- Training and real recordings
A separate control added Falcon’s speaker contrastive loss to three architectures. This training signal pulls estimated speech toward the clean target’s audio features and pushes it away from near-field interferers’ features.
On RealSSA-Sim, the reported separation score rose for Falcon, SwiftNet-12 and Dolphin. On held-out RealSSA-Real recordings, training on RealSSA-Sim improved DNSMOS estimates of perceived speech quality for all five tested architectures, and NISQA estimates for four; AV-TF-GridNet was the NISQA exception.
For Falcon, both measures were higher than after VoxCeleb2 training. These results support the method and dataset in the tested settings, with limits on the size of the real-recording set and the loss’s training data.
Deep
- Original paper authors
- Wendi Sang, Kai Li, Yifan Li, Jianqiang Huang, Xiaolin Hu
- Submitted to arXiv
- License
- arXiv non-exclusive