Quick answer
Speaker diarization is the process of dividing a recording by who is talking and labelling each segment with a speaker. It works by extracting a voice fingerprint (an embedding) for short windows of audio and clustering windows that sound alike. Deepgram, AssemblyAI, and Speechmatics offer it in their APIs, the open-source pyannote library is the common self-hosted option, and meeting tools such as Otter, Fireflies, and Fathom depend on it.
A transcript without speakers is a wall of text. Diarization is what turns it into a conversation, and it is also where meeting tools most often go wrong.
How it works
- Detect speech: find the parts of the audio where anyone is talking
- Embed: for each short window, compute a vector that captures the voice's characteristics
- Cluster: group windows with similar vectors; each cluster is a speaker
- Align: match the speaker segments to the transcript so each line gets a label
- Name: if the platform knows participants or has voice profiles, replace "Speaker 2" with a person
Where it fails
Overlapping speech is the hardest case: two voices in one window produce a confused embedding. Similar-sounding speakers get merged; one speaker with a cold or on a bad connection gets split into two. Shared microphones on a conference call, where everyone in a room comes through one channel, are worse than individual laptops. Short interjections are often mislabelled. That is why every meeting app lets you rename and merge speakers after the fact.
Why it matters
Action items depend on it ("who agreed to do that?"), clinical notes depend on it (what the patient said versus the clinician), and call analytics depend on it (agent talk time versus customer). If you are evaluating a transcription API or a meeting tool, test diarization on your own recordings with your real speakers, not on the vendor's demo audio.
Related reading
Bottom line
Diarization is a solved problem for clean two-person audio and an unsolved one for a noisy room with six people. Know which one you have before you trust the labels.
