The voice pipeline

From a caller's breath to Elle's first word back in about a second: how audio flows through the realtime path, where every millisecond goes, and the three-layer defense that makes her interruptible everywhere except her opening line.

The path of one spoken turn

Hearing: STT with intent-gated barge-in

Deepgram's stream does voice-activity detection and endpointing server-side. But its raw SpeechStarted event fires on any sound (a breath, a cough, line noise, an echo of Elle's own voice), and using it directly meant callers' background noise cut her off mid-sentence. Interruption is instead gated on recognized words:

// Note: we deliberately do NOT barge in on the raw `SpeechStarted` VAD event
// — it fires on any sound (breath, cough, line noise, echo of Elle's own
// voice) and was cutting her off. Barge-in is gated on recognized words.
if (!this.barged && text.length >= config.deepgram.bargeInMinChars) {
  this.barged = true;
  console.log(`[voice] barge-in trigger: "${text}"`);
  this.h.onSpeechStarted?.();
}

Utterance boundaries come from two paths: speech_final (the fast endpointing path, tuned by DEEPGRAM_ENDPOINTING_MS) and UtteranceEnd (a slower ≥1s fallback). Both are logged, because which path fires is itself a diagnostic: turns that keep flushing via the fallback mean the noise floor is holding the VAD open.

Thinking out loud: sentence streaming

The single biggest latency lever in a cascade: don't wait for the whole reply. The LLM streams tokens; the moment a complete sentence exists, it goes to TTS while the rest is still being generated:

// Emit a chunk once a sentence has clearly ended (punctuation followed by
// whitespace) and it's long enough to sound natural on its own.
const SENTENCE_RE = /^([\s\S]{15,}?[.!?…]["')\]]?)\s+([\s\S]*)$/;
const resp = await this.llm.chatStream(req, (delta) => {
  if (seq !== this.turnSeq) return;
  buffer += delta;
  let m: RegExpExecArray | null;
  while ((m = SENTENCE_RE.exec(buffer))) {
    speakChunk(m[1]);   // → TTS, while generation continues
    buffer = m[2];
  }
});

Meanwhile the TTS socket is pre-warmed during LLM thinking, so the connection handshake overlaps generation instead of delaying the first audible word. One TTS WebSocket stays open per voice for the whole call; no sentence pays a fresh TLS handshake.

Speaking: exact frames, paced delivery

Aura streams synthesized audio in chunks that aren't frame-aligned; the cascade re-frames them into exact 160-byte (20 ms) μ-law frames with a carry buffer, and the bridge releases them to Twilio on a drift-corrected 20 ms schedule. Pacing matters: dumping audio as fast as it synthesizes makes Twilio's jitter buffer choppy, and it makes barge-in worse, audio already shipped to Twilio cannot be un-played, so the pacer keeps the un-sent remainder on our side where a barge-in can discard it instantly.

The latency budget

Every turn logs its own timing; these lines are the pipeline's vital signs ([timing] in the logs):

Barge-in: three coordinated cancellations

When a caller interrupts, three things must stop at once: the LLM stops generating, the TTS stops synthesizing, and queued audio stops playing. One integer coordinates all three:

/** Bumped whenever a turn starts or is cancelled; stale turns bail on mismatch. */
private turnSeq = 0;

cancelResponse(): void {
  // Invalidate the running turn. Its captured seq no longer matches turnSeq,
  // so the LLM loop and TTS pump stop at their next checkpoint. Twilio's own
  // playback buffer is flushed by the bridge (the `clear` message).
  this.turnSeq++;
}

Every async stage captures seq at turn start and checks it at each checkpoint: before speaking a sentence, before emitting a frame, after every await. A barged-in turn doesn't error; it just quietly stops being real. The bridge simultaneously sends Twilio a clear to flush its playback buffer and truncates the model's context to what the caller actually heard.

Echo suppression: don't argue with your own voice

On a phone line, Elle's voice can leak back through the caller's handset and transcribe as "caller speech", which would both trip barge-in (cutting her off) and produce a bogus utterance she then answers. Everything she says is noted in a 30-second rolling window; a transcript that is a verbatim chunk of her own recent words is echo:

const norm = DeepgramCascade.normalizeForEcho(candidate);
// Too short to attribute — leave it alone (it can't barge in anyway, and
// dropping a real "yes"/"okay" would be worse than a stray echo word).
if (norm.length < 8 || !norm.includes(" ")) return false;
...
if (haystack.includes(norm)) return true;
// While the greeting could still be playing, also match loosely: line echo
// arrives garbled (dropped words, misheard names), so an exact-substring
// test misses it. A transcript made mostly of Elle's own recent words is
// echo. Greeting-only on purpose — mid-call, callers legitimately repeat
// her words back ("Tuesday at 2"), and loose matching would eat real turns.

The loose ≥75% word-overlap match runs only during the greeting window. That asymmetry is the tuned-on-real-calls part: greeting echo arrives garbled and needs fuzzy matching, but mid-call fuzzy matching would swallow legitimate caller turns that quote Elle's own words back at her. Normalization uses Unicode letter classes so Spanish accented words survive intact (see bilingual mode).

The uninterruptible greeting

The opening line is deterministic (no LLM round-trip, so no dead air) and it always plays to the end. Three independent layers guarantee that, because any one alone proved insufficient on real calls:

1. Echo filtering (above) stops the greeting's own line-echo from registering as caller speech at all.
2. The play-to-end gate. The bridge counts milliseconds of audio enqueued vs. sent; barge-in stays disabled until every frame of the greeting has been sent, not merely synthesized:
   src/realtime/twilioBridge.ts · onSpeechStarted

   if (this.greetingGate) {
     if (this.greetingEndMs === null || this.totalSentMs < this.greetingEndMs) return;
     this.greetingGate = false;
   }

3. The synthesis lock. A caller talking over the greeting queues a turn , and starting a turn bumps turnSeq, which would kill the greeting's own TTS mid-stream. So the turn drain awaits the greeting's synthesis promise before any turn may begin; the caller's words still land, as the first turn after the greeting finishes.

Turn batching and error recovery

A caller talking in bursts (or over the greeting) queues several utterances. They merge into one LLM call and one coherent answer, instead of an API call and a spoken reply apiece , fewer tokens and no stacked back-to-back replies. And a failed turn (a provider hiccup mid-call) never drops the call: one fast retry absorbs transient errors, and if the turn still fails, Elle says so in the caller's language instead of leaving dead air.

// Merge everything queued into ONE turn.
const parts = this.queue.splice(0);
if (parts.length > 1) {
  console.log(`[voice] merged ${parts.length} queued utterances into one turn`);
}
await this.runTurn(parts.join(" "));

Verified offline

The whole orchestration runs against fakes, no network, no Deepgram, no LLM, in deepgramCascade.verify.ts and bridge.verify.ts: streaming, tool round-trips, frame re-framing byte counts, barge-in mid-utterance, and bilingual voice selection all assert offline before anything deploys.