A speech encoder keeps streaming memory bounded
Gabor Gabor: E-Gaborformer, a Strictly Causal Speech Recognizer with Gabor-Dictionary Convolutions and Delta-Rule Memory · arXiv:2611.08092
In plain words
A voice agent needs to understand speech while someone is still talking. E-Gaborformer is a speech-recognition encoder that uses a small, fixed memory of earlier sound plus attention to a recent window, so it can work without waiting for future audio and its memory does not grow as a call continues.
In a test on LibriSpeech read speech, it matched a causal E-Branchformer after ten training epochs: both had the same average error rate on one test portion, and E-Gaborformer was slightly better on another. Production tests also showed mixed results across speech sets.
The research
For a voice agent, recognition must keep pace with a caller without storing an ever longer history of sound. This design offers a way to bound that per-call memory while preserving recognition performance in the reported causal comparison, so the memory cost need not rise with call length.
E-Gaborformer is a streaming speech-recognition encoder that combines fixed-size recurrence, recent-window attention and causal filters to bound memory without waiting for future audio.
On LibriSpeech-100, a read-speech collection scored by the percentage of reference words transcribed incorrectly, researchers trained E-Gaborformer and a causal E-Branchformer with the same front end and recipe.
After ten epochs, two-seed mean WER was 20.5% versus 20.8% on dev-clean and 39.2% for both on dev-other.
These results show that the recurrent design can match this attention-based baseline under the tested recipe, while using fixed-size streaming state. The small dev-clean difference and equal dev-other result support a claim of comparable recognition, rather than a broad accuracy advantage.
The LibriSpeech comparison covers two-seed means after ten epochs, and the production comparison with Nemotron uses one English set on which E-Gaborformer was trained. The authors identify longer controlled training, emission-latency measurements and multi-minute-stream evaluation as next steps.
What the evidence shows
- Bound memory as speech arrives
E-Gaborformer is a streaming speech-recognition encoder built to produce transcripts without waiting for future audio while keeping per-call memory bounded. That addresses two demands of real-time recognition: outputs must use only audio already received, and stored state must not keep growing as a call continues. Its design combines a fixed-size recurrent memory with attention to a recent audio window.
- Match a causal baseline
On LibriSpeech-100, a read-speech collection, researchers compared the two causal encoders using word error rate, the percentage of reference words transcribed incorrectly.
Because they shared a front end and training recipe, the results indicate that E-Gaborformer matched the baseline on dev-other and had slightly lower error on dev-clean after ten epochs, supporting comparable recognition under these conditions.
- Address stalled training
The causal architecture screen also exposed a training issue: some runs with recurrent layers stayed on an initial plateau, emitting only blank symbols. After researchers changed both the recurrent output gate and a normalization setting, none of 36 later runs stayed on that plateau for an epoch, compared with 4 of 18 runs under the standard settings.
This is evidence about early training behavior, not final accuracy, and the two groups’ other settings and seeds are not established as matched.
- Test components and transfer
In a separate three-epoch screen, a causal hybrid with Gabor-based temporal convolutions had lower WER than one with unconstrained depthwise convolutions: the latter was 1.2 percentage points higher on dev-clean and 1.1 higher on dev-other.
At production scale, the current model improved on three of four Hebrew test sets versus the previous transducer, but did worse on addresses, URLs and email addresses; English results were roughly unchanged. The selected design therefore showed promise across these evaluations, alongside a clear exception.
Deep
- Original paper authors
- Nir Krakowski, Tamir Cohen, Ohad Zohar, Chen Zarfati
- Submitted to arXiv
- License
- arXiv non-exclusive