Image-generation models can help a speech system learn words
ImageTTS: Repurposing Pretrained Image Diffusion Transformers for Speech Synthesis · arXiv:2610.34807
In plain words
ImageTTS is a speech-making system built by adapting a model trained to generate images. It turns speech into a picture-like sound map, compresses that map with a frozen image model, then trains an image-generation model to produce speech from text and, in one mode, a voice reference.
In a matched test, the image-trained model made fewer word-recognition errors than the same kind of model started from random weights. That result applies to the tested speech and training setup.
The research
Speech synthesis has to produce the requested words while preserving a speaker’s identity from a short reference. It remains unclear whether a model trained on images can transfer useful representations and generation knowledge to that task.
ImageTTS adapts an image-generation model for speech, and matched tests find an image-trained start improves learning and word accuracy over random initialization.
The authors ask whether an image-generation model can help speech synthesis preserve words and speaker identity. ImageTTS represents speech as picture-like sound maps, then adapts an image-trained generator to produce speech from text and, in zero-shot synthesis, a voice reference.
In a matched LibriTTS training comparison, the image-pretrained model learned faster than a randomly initialized one. On 200 English test utterances at the same 100,000 updates, a speech recognizer found 2.7% word error for the pretrained model and 4.2% for the random model. This supports a benefit from image initialization in the tested setup.
The evidence comes from particular speech datasets and evaluation settings. The authors also report lower speaker similarity on the Chinese Seed-TTS test set and attribute that gap to identity information not fully preserved by the frozen image VAE; they did not test that explanation separately.
What the evidence shows
- Can image models help speech?
Speech generation must get the words right and, when given a reference voice, preserve speaker identity. ImageTTS is a framework for testing whether a model trained to generate images can help with those demands.
It converts speech into picture-like spectrograms, uses a frozen image model to represent them, and adapts an image-generation model to produce speech.
- Check the speech representation
Before testing the generator, the authors checked whether the frozen image model could represent speech. They encoded and decoded 100 sampled LibriTTS utterances, then compared the reconstructed waveforms with the originals.
At frequency resolution F=512, evaluators measured high intelligibility and speaker similarity. This supports the representation on those samples, while leaving the broader coverage of speech untested.
- Compare the starting weights
The authors then compared image-pretrained and randomly initialized speech generators. The models shared their architecture, LibriTTS training data, objective, and procedure, while their starting weights differed.
The image-pretrained model reached a training loss comparable to the random model’s after 150,000 updates in 100,000 updates. This indicates faster learning under the tested training setup.
- Measure generated words
To check generated words, a speech-recognition model evaluated 200 utterances from an official English test split. Each system heard the first three seconds and generated the remainder from the transcript.
At 100,000 training updates, the recognizer’s word error rate was 2.7% for image pretraining and 4.2% for random initialization. This matched comparison supports an image-initialization advantage in the tested continuation task.
- Extend the tests
The framework was also tested at larger scale and with style instructions. On an English test of speech generated from text and a short voice reference, ImageTTS-base had lower word error rate than two primary systems trained on comparable amounts of speech.
On a 200-example style-instruction test, Wav2Vec 2.0 predicted age and gender, while PENN extracted pitch and pitch variation for categorization. The authors averaged attribute-level accuracies, obtaining 0.66.
These results show transfer across the tested tasks, with speaker similarity remaining a limitation, especially on the Chinese test set.
Deep
- Original paper authors
- Hanchen Pei, Shujie Liu, Yayue Deng, Fangyun Wei, Yanqing Liu
- Submitted to arXiv
- License
- arXiv non-exclusive