Choosing a voice prompt by the speech it helps a model produce
What Makes a Good Audio Prompt for Zero-Shot TTS? Learning Prompt Selection from Synthesis Outcomes · arXiv:2610.40372
In plain words
A voice prompt is a short clip that helps a text-to-speech model speak in someone’s voice. When a longer recording offers many possible clips, the cleanest-sounding one may not help that model say the words clearly.
The paper introduces a tool that learns from speech a model has already generated, then ranks new clips using their audio alone. On held-out English recordings, it lowered word errors for one tested model compared with choosing a clip at random, without generating speech during selection.
The research
When someone has a long recording of a speaker, which few seconds should they choose before knowing what text will be spoken? The clip that sounds cleanest may not help a particular text-to-speech model produce intelligible speech. The study tests whether the model’s own generated outputs can reveal which prompt works better.
An audio-only selector learns from a TTS model’s past outputs to rank voice clips by predicted intelligibility, without generating speech when choosing.
The authors trained an audio-only selector using outcomes from speech generated by a particular TTS model. It ranks a recording’s candidate clips by predicted word error rate (WER), which compares a speech-recognition transcript of generated speech with the target text, then chooses one without generating speech during selection.
On 400 held-out English recordings, the selector lowered Qwen3-TTS 0.6B’s WER by 1.36 percentage points, or 32% relative to a random clip.
That gain matched selection based on 12 syntheses per candidate and exceeded the UTMOS rule, which chooses the clip with the highest predicted naturalness score, by 0.54 percentage points in the pre-specified comparison.
The results support using a model’s synthesis outcomes to rank prompts for intelligibility, which a naturalness score alone may miss. Across four models, the selector beat the strongest simple rule for three and tied it for Chatterbox.
The evidence covers four tested speech-generation models and English YouTube recordings, plus one separate added-noise study. Most measures were automatic, the listening test was too small to resolve a clarity advantage, and the selector’s margin over one quality rule varied across training runs.
The results do not establish performance for other languages, recording conditions, or longer prompts.
What the evidence shows
- Clean does not mean clear
The paper asks which few seconds from one speaker’s longer recording will help a particular text-to-speech model produce intelligible speech. The common shortcut is to choose the clip that sounds cleanest, but a clean clip may not be the one that leads that model to make fewer word errors.
The authors frame prompt choice as a question about the generated speech, rather than audio quality alone.
- Test the resulting speech
To measure whether the choice matters, the authors generated speech from candidate clips for five target sentences, using three random seeds per sentence. They averaged each clip’s outcome across its 15 outputs.
As a reference, they also chose clips using actual synthesis outcomes from four target sentences and checked each choice on the fifth. This reference reduced word error rate (WER), the rate of words transcribed incorrectly compared with the target sentence, versus random choice for Qwen3-TTS and F5-TTS on training recordings, showing that prompt choice can affect intelligibility.
- Predict without generating
The audio selector learns to rank clips from their audio, using outcomes from speech generated by a fixed TTS model during training. At selection time, it does not generate speech.
On 400 held-out English recordings, it reduced Qwen3-TTS 0.6B’s WER by 1.36 percentage points, or 32% relative to random choice. Its gain matched the reference that used 12 syntheses per candidate and exceeded the UTMOS rule, which selects the clip with the highest predicted naturalness score, by 0.54 percentage points in the pre-specified comparison.
- Different models, different prompts
The result extended across the four tested models with a qualification: model-specific selectors beat the strongest simple selection rule for three and tied it for Chatterbox, which had little room for selection. A separate added-noise study found that UTMOS more often picked the clean clip and improved naturalness, while the selector reduced WER more.
The contrast supports the view that clean sound and intelligibility can favor different prompts, within the settings tested.
Deep
- Original paper authors
- Arnon Turetzky, Yossi Adi
- Submitted to arXiv
- License
- CC BY 4.0