Speaker Diarization

Speaker diarization is the task of working out "who spoke when" in a recording: dividing the audio into stretches of speech and grouping them by speaker, without knowing in advance who the speakers are. The output is a set of anonymous labels such as Speaker 1 and Speaker 2, each with start and end times. Combined with speech recognition, it turns a wall of transcript into a conversation with turns, which is what podcast editing, meeting notes, interview cutting and named captions all depend on. Diarization is distinct from speaker identification, which attaches real names to voices and links them to a person's identity. That second step is where most of the privacy and consent questions arise.

How Diarization Works

The classic pipeline has three stages:

  1. Segmentation. A model finds where speech occurs and where the speaker changes. It increasingly also detects where two people talk at once.
  2. Embedding. Each segment of clean speech is turned into a speaker embedding, a vector that places similar-sounding voices close together. NVIDIA's TitaNet family is a widely used example.
  3. Clustering. The embeddings are grouped, either into a known number of speakers or by a similarity threshold, and each cluster becomes a label.

End-to-end neural models fold these stages into one network. NVIDIA's Streaming Sortformer v2.1 handles up to four speakers in real time and outputs them in order of first appearance. NVIDIA reports 6.65% diarization error on two-speaker telephone calls (CALLHOME) and 15.09% on the harder DIHARD III benchmark at about one second of latency. It also notes that quality degrades with five or more speakers. Words from the recognizer are then assigned to whichever speaker turn they overlap most.

Measuring It: Diarization Error Rate

The standard metric is diarization error rate (DER). The pyannote.metrics library defines it as false alarm plus missed detection plus speaker confusion, divided by the total duration of reference speech. A false alarm is speech reported where there was none, a missed detection is speech not found, and confusion is speech given to the wrong speaker. Confusion is the error users notice most, because it puts one person's words in another's mouth. Evaluations differ in whether they score overlapping speech and how much forgiveness they allow around boundaries, so DER figures from different papers rarely compare directly.

Tools in 2026

  • pyannote. The pyannote.audio library, an open-source project, (v4.0.7, June 2026) is a widely used toolkit. Its open "community-1" pipeline (September 2025, CC-BY-4.0) reports 17.0% DER on the AMI meeting benchmark, against 18.8% for the earlier 3.1 pipeline. The company behind it sells a hosted Precision-3 model (September 2026), which it says cuts average DER by 10.4% relative to Precision-2.
  • NVIDIA NeMo models. Sortformer for end-to-end and streaming diarization, and TitaNet embeddings for clustering pipelines and speaker verification. See NVIDIA.
  • sherpa-onnx. Runs pyannote segmentation with TitaNet or other embeddings on-device through ONNX Runtime, next to speech recognition (see edge AI).
  • Speech APIs. Many transcription APIs now return speaker labels directly. AssemblyAI's Universal-3.5 Pro (July 2026) produces speaker-annotated transcripts, and ElevenLabs' Scribe v2 diarizes up to 32 speakers.

On-device diarization is fast. In MetaRush's local transcription bake-off (October 2026, Apple M2 Max), sherpa-onnx with pyannote segmentation and a small TitaNet model processed 7.9 minutes of audio in 29 seconds using under half a gigabyte of memory. On a clean two-voice test file it reached 3.2% DER with no speaker confusion. The test notes that synthetic voices like these are far easier than real conversations.

Diarization Is Not Identification

Diarization says "these segments are the same voice." Identification says "this voice is a particular person," usually by comparing an embedding with enrolled voice profiles. The difference matters for data, errors and consent. A diarization mistake mislabels a turn within one recording. An identification mistake attributes words to the wrong named person, potentially across an entire library. The two tasks are compared in detail in speaker diarization vs speaker identification.

A stored voice embedding used to recognize someone is biometric data in many jurisdictions. Illinois's Biometric Information Privacy Act lists a "voiceprint" among its biometric identifiers. The EU's GDPR treats "biometric data for the purpose of uniquely identifying a natural person" as a special category whose processing is prohibited unless an exception applies, such as explicit consent. Voice data is also the raw material of voice cloning and deepfakes, which raises the stakes of storing it carelessly. See data privacy for the wider context.

Responsible systems follow a few practices:

  • Names come from people, not guesses.
  • Automatic matches are shown as unconfirmed until someone checks them.
  • Only confirmed voices are used to build profiles.
  • A correction ("that's not them") is remembered.
  • Voice data stays on-device or under the owner's control where possible.

MetaRush, an AI video editing harness, follows this model. It asks who a new voice is instead of guessing: voices stay as Speaker 1 or Speaker 2 until a person names them, recognized voices are marked as unconfirmed matches, and only voices a person has confirmed are enrolled.

Where It Fails

Diarization remains hardest exactly where real recordings are messiest:

  • Overlapping speech and crosstalk.
  • Short interjections ("right," "mm-hm") inside someone else's turn.
  • Similar-sounding voices, such as siblings.
  • Phone or far-field audio.
  • Music under speech.
  • Children's voices.

Thresholds that work on one kind of material over-split or merge speakers on another, so production tools let users correct a speaker count and re-run the analysis. In video work, separate microphones per person, as in remote podcast recordings, sidestep much of the problem and turn diarization into alignment (see multicam and audio sync).

Why It Matters for Editing

Speaker turns are what let an editor, or an agent, act on instructions like "use only the guest's answers" or "cut to whoever is talking." Avid's Media Composer 2026.8 now selects multicam angles by speaker and places cuts on speech boundaries. Speaker-aware captions, highlight clips and transcript-based editing all rely on the same labels. In agentic video editing, accurate turns are the difference between a cut that follows the conversation and one that puts words in the wrong person's mouth.

Further Reading