In plain words
Researchers created a way to make a speech model react to who is speaking. They first train the model to use speaker identity, then connect one person’s voice to a chosen output while including clean speech from other people.
In tests with two speech models, ordinary recordings of one target speaker often triggered a fixed replacement phrase, while recordings from other speakers rarely did. The results apply to the tested models and speakers.
The research
If a speech model is secretly changed during training, can a person’s unaltered voice trigger behavior that is absent from the words in the transcript? Many audio backdoors need an attacker to add or change a sound when the model is used; speaker identity offers a different activation route.
The paper’s two-phase Speaker-Activated Backdoor method makes speaker identity easier for speech models to use, then ties a target speaker to an attacker-chosen output.
The method first trains a model to use speaker identity, then trains it to produce a chosen output for the target while including clean examples from other speakers. In the main setting, the target was Judi Dench, with two minutes of real target audio within a 20-minute budget.
Across three runs, fixed-phrase replacement affected 90.2% of held-out target clips for Whisper-medium and 97.0% for Parakeet; non-target activation averaged 1.7% and 0%.
These results show that the tested models could use a target’s ordinary voice to trigger a fixed output, with little activation on non-target speech in this setting.
These results cover particular models and targets under the tested fine-tuning setup. The study does not establish that the same activation rates hold for other speakers, models, or adaptation settings.
Analysis
Deep
- Original paper authors
- Nataša Krčo, Euodia Dodd, Peter Romov, Yves-Alexandre de Montjoye
- Submitted to arXiv
- License
- arXiv non-exclusive