Speaker diarization and why two channels help

Speaker diarization is the step that works out who spoke when in a recording, without necessarily knowing anyone's name. It takes a stream of audio and cuts it into turns, then groups those turns by voice, so the output reads "Speaker 1 said this, Speaker 2 said that." It's a separate job from transcription, which only cares about the words, and it turns out to be much easier when the recording already keeps each person on a channel of their own.

What does speaker diarization actually do?

Diarization answers "who spoke when." Transcription answers "what was said," and speaker identification answers "which known person is this." Diarization sits in the middle: it decides how many distinct voices are present and marks which stretches of audio belong to each, but it doesn't attach a name. The result is anonymous labels, Speaker 1 and Speaker 2, that a human or a meeting platform can later map to real people.

That distinction matters because the three jobs fail in different ways. A transcript can be word-perfect and still lump two people into one block of text. Diarization can split the voices correctly and still label them Speaker 1 and Speaker 2 forever, because nothing in the audio says "this is Anna." Getting a clean, readable transcript of a conversation needs both to go right.

Why is diarization hard from a single recording?

The classic setup is one microphone in a room recording everyone into a single mixed stream, and from that the software has to reconstruct the turns. It first finds where speech is at all, then cuts the speech into short pieces, then turns each piece into a numeric fingerprint of the voice, a so-called embedding. Then it clusters those fingerprints, hoping that all of one person's pieces land together and separate from everyone else's.

Several things make that genuinely hard. The number of speakers usually isn't known in advance, so the clustering has to guess it. Voices drift as people get tired or excited, and a phone or a bad connection can make the same person sound like two. The worst case is overlapping speech, two people talking at once, because a single stream can't be neatly split into "this half is her, this half is him." Open toolkits like pyannote have gotten good at this, but it's still an estimation problem, and it's doing all of it after the fact from a mixed recording.

Why do two audio channels make it easier?

Because a call recording doesn't have to be a single mixed stream. On a Mac, your own voice comes in through the microphone and the other people come in as system audio, the sound the call app plays through your speakers. If you record those as two separate channels, the biggest boundary in the whole conversation, you versus everyone else, is drawn for you by physics. No clustering, no guessing, no embeddings for that split. The microphone channel is you, the system channel is the far end, and that's already the correct answer for a one-on-one call.

This is the part of diarization that two channels hand you for free. Real diarization only has to run when there's more than one person on the far side, and even then it runs on just the remote channel instead of the whole mixed room. A smaller, cleaner problem, with half of it solved before the diarizer starts. Transcriber works this way: the mic side is labelled with your name from Settings, and pyannote runs over the system channel after the call to split it into Speaker 1, Speaker 2 and so on when it hears several remote voices.

What still needs real diarization on a call?

The far-end channel, whenever more than one person is on it. If three colleagues dial in from separate machines, their audio still arrives mixed together on your system channel, because your Mac only sees one output stream from the call app. So the hard clustering problem comes back, just confined to that one channel rather than the entire recording. The diarizer segments the remote channel, fingerprints the voices and groups them, and you get Speaker 1 through Speaker 3 to rename.

There's also a live option. By default the separation happens after the call, when the diarizer can hear the whole recording before deciding, which is the more accurate way. An experimental setting instead labels remote speakers during the call, matching each new turn against the voices heard so far. That's less accurate, because it has to commit to a speaker without hearing the rest of the conversation, and it's off by default for that reason.

Where does this approach fall short?

The honest limit is names. Diarization gives you groupings, not identities, so you still get Speaker 1 and Speaker 2 and have to type in who they are, because nothing in the raw audio knows that Speaker 2 is your client. Overlapping voices on the far-end channel are still the weak spot, the same as in any single-stream diarization. And the whole two-channel trick depends on the recording keeping the channels separate in the first place; feed a diarizer one already-mixed file of a group call and it's back to the hard problem with none of the head start.

For a full walk-through of how the two channels are captured and written to disk, the post on how Transcriber works covers the recording side from the audio tap to the transcript file.