A speech model confines feature matching to protect English

When one speech model serves languages with very different amounts of training data, a training signal that helps one group can weaken another. It remains unclear whether matching a teacher’s internal representations can add useful guidance without making the shared model worse for English.

CAFE sends teacher-feature matching through adapters used by non-English speech, keeping that update out of the shared encoder.

Researchers first compared a baseline trained with transcripts and teacher output scores against a run adding Centered Kernel Alignment (CKA), a loss that matches patterns in teacher and student features.

On the Open ASR Leaderboard, whose test sets measure speech transcription, the average percentage of words transcribed incorrectly fell for the multilingual group but rose for English.

With each system’s last seven checkpoints averaged, CAFE scored 9.03% on seven English sets and 7.00% on nine multilingual sets, versus 9.67% and 7.34% for Capacity, which has the same adapters but no feature loss. CAFE also scored lower than the baseline in both groups. The comparison supports a benefit from confined feature matching in this setup.

Each configuration was trained once, so the study does not show how results vary across repeated runs. The main evaluation also uses the test set’s language label to route adapters; results with an automatically predicted language are not the main comparison.