Voice activity detection

A VAD operates on frames of roughly 10 to 30 milliseconds and emits a speech or non-speech decision for each one. Classical implementations used energy thresholds and zero-crossing rates; modern ones are small neural classifiers that hold up much better against non-stationary noise such as traffic, a television, or another conversation in the room.

The two failure modes pull in opposite directions and cannot both be minimised. A VAD tuned to be sensitive fires on background noise, so the agent believes the caller is speaking when they are not and stalls waiting for an utterance that never finishes. A VAD tuned to be conservative clips quiet speech onsets, so the first syllable of the caller’s reply is lost and the transcript begins mid-word.

It is worth being precise about the boundary with related concepts. VAD says speech is present. Endpointing decides that an utterance has finished. Turn detection decides that it is now the agent’s turn to talk. Each layer adds context the one below it does not have, and conflating them is a common source of agents that either interrupt constantly or leave awkward pauses.