DiffuPlex lets spoken-dialog models predict ahead and revise drafts
In plain words
A spoken assistant that listens while it talks can react to interruptions, but its model usually has to compute a fresh answer every fraction of a second. DiffuPlex is a method that predicts a short stretch of what both speakers might say, then plays only a confident part of the assistant's draft.
When the user's actual speech differs from the prediction, DiffuPlex replaces only the assistant's unplayed words. In tests, its two settings advanced about two conversation frames per model computation on average, while staying close to the original model on conversational measures.
The research
Can a spoken assistant react to interruptions without computing a fresh model output for every brief moment of conversation? Predicting ahead might cut repeated work, but the assistant's draft depends on user speech that has not arrived yet, so it needs a way to adjust when the conversation changes.
DiffuPlex predicts masked future speech for both speakers, plays a confident assistant prefix, and revises only its unplayed part.
The authors trained DiffuPlex on a spoken-dialog model and tested whether it could reduce repeated computation without losing conversational behavior.
On continuous interaction tests, Listen advanced an average of 1.91 frames per model computation and Speak 2.23, versus 1.00 for the frame-by-frame reference.
On a fixed 60-clip subset, both settings also reduced measured deployment-path runtime against that reference. Human ratings were close for Listen, while Speak had lower speech naturalness despite similar dialog-flow ratings.
The results support the method's potential to reduce computation, with a quality trade-off for Speak.
The runtime result comes from a fixed 60-clip subset on one GPU, and applying DiffuPlex to a different tested backbone reduced model calls without preserving pause and interruption behavior. The results do not establish performance across other architectures or runtime environments.
Analysis
Deep
- Authors
- Heeseung Kim
- License
- CC BY 4.0