An eight-speaker model lowers diarization errors across delays
Nemotron 3 Diarization: Advancing the Accuracy-Efficiency Frontier in Streaming Speaker Diarization · arXiv:2611.12246
In plain words
Speaker diarization marks who spoke when, including when people talk over each other. This paper introduces an open-weight model that tracks up to eight speakers and can work with different amounts of delay.
It uses a Transformer to understand audio, then adds finer timing to its output and training that discourages it from inventing speakers.
Across the tested meeting recordings and delays, the model had fewer diarization errors than the named earlier systems. The formal top-performance claim is limited to two datasets.
The research
When a recording has overlapping voices, more speakers, or little time to wait for future audio, a diarization system must still track who is speaking.
This work changes the expectation for one open-weight model: it can support eight speakers across offline and streaming settings while making fewer labeling errors than the tested earlier systems.
Nemotron 3 Diarization extends an open-weight streaming speaker-diarization model to eight speakers and finer output timing, with lower error than tested baselines.
The authors compare Nemotron with Streaming Sortformer-v2.1 at three input-buffer latencies across five meeting datasets and microphone conditions, scoring speaker-labeling errors with DER.
Nemotron has lower DER in every reported comparison; at 0.32 seconds on DIHARD III, the values are 13.55% and 19.85%.
That supports an accuracy advantage in the tested settings. A separate downstream test also finds lower word error with soft Nemotron conditioning than with hard conditioning in four AMI and NOTSOFAR-1 conditions.
The formal state-of-the-art claim is limited to DIHARD III and CALLHOME. For AMI, AliMeeting, and NOTSOFAR-1, this paper's DER uses forced-alignment reference labels, so some published comparisons using original annotations are not directly comparable; each training-component test also used one run.
What the evidence shows
- More speakers, same challenge
Nemotron 3 Diarization is an open-weight model for assigning speech to speakers in recordings. The central challenge is to preserve speaker identities across streaming chunks while handling overlap and newly arriving voices, with different amounts of future audio available.
The model extends Streaming Sortformer from four to eight speakers and is designed for both offline use and streaming delays.
- The deciding accuracy comparison
Diarization error rate (DER) measures speaker-labeling errors; lower is better under the paper's scoring rules. Across the evaluated meeting datasets and microphone conditions, Nemotron has lower DER than Streaming Sortformer-v2.1 at all three tested input-buffer latencies.
At 0.32 seconds, for example, DIHARD III DER is 13.55% for Nemotron and 19.85% for Sortformer-v2.1. The comparison supports an accuracy advantage in these tested settings, though the systems differ in architecture and training.
- Where the claim applies
The paper's state-of-the-art claim has a narrower scope than its within-paper comparisons. It formally claims the result on DIHARD III and CALLHOME.
On AMI, AliMeeting, and NOTSOFAR-1, the authors use forced-alignment reference labels, while some external results use original annotations; those numbers are not directly comparable.
- Efficiency under stated conditions
The training comparison suggests the Transformer can reduce computing demands: under bf16 mixed-precision training on 64 NVIDIA A100 GPUs, it used 2.8 times less memory and trained 2.2 times faster than an otherwise-identical model with the legacy FastConformer encoder.
In separate inference tests on one GPU, Nemotron had higher audio-duration-to-processing-time speed than Sortformer-v2.1 across matched conditions.
- What the training checks show
The authors compare training variants on recordings from all evaluation sets pooled together. Removing simulated data raised zero-collar diarization error rate (DER) at both tested delays: from 10.42% to 13.40% offline and from 12.06% to 15.21% at 0.32 seconds.
Removing NEST initialization raised DER by 1.63 and 2.22 percentage points; removing the activity loss raised it by 1.00 and 0.97 points. Removing high-resolution upsampling raised DER more than downsampling the 10-millisecond predictions to 80 milliseconds before scoring, suggesting that learning finer outputs contributes beyond finer scoring alone.
Removing the phantom loss barely changed DER but raised speaker-count error. Each check used one training run, so these are observed effects, not statistical significance results.
- A check beyond diarization
The work also tests whether diarization helps a separate task: producing transcripts with speaker labels. With offline DiCoW speech recognition, soft conditioning on Nemotron's speaker posteriors lowered time-constrained word error rate in all four tested AMI and NOTSOFAR-1 microphone conditions compared with hard segment labels. This supports transfer to that speech-recognition setup, not a universal result across back ends.
Deep
- Original paper authors
- Ivan Medennikov, Taejin Park, Tatiana Timofeeva, He Huang, Alexander Polok
- Submitted to arXiv
- License
- CC BY 4.0