Architecture

Elle is an advanced cascade: separate best-of-breed components for hearing, thinking, and speaking, joined by orchestration code that makes them feel like one mind. This page explains that shape, the theory behind it, and the seam that lets whole architectures be swapped at runtime.

The four layers of a voice agent

Any phone-capable AI decomposes into four jobs, whether or not a product exposes them:

The design space is which of these get fused. Fuse everything and you have a pure speech-to-speech model (natural, but you lose text-level control). Keep everything separate and you have a classic cascade (controllable, but turn-taking runs on silence timers). Elle today is the second, with the timer problem engineered down to imperceptibility; the roadmap (below) moves her toward fusing layers 1+2 while deliberately keeping 3 and 4 separate.

Layer two is the contested one, and it is not optional. Thinking Machines calls the turn-based stack a stopgap and argues interactivity belongs inside the architecture rather than in a harness bolted around it¹; OpenAI removed the turn detector from GPT-Live's audio path entirely and escalates reasoning asynchronously so it never blocks the voice loop²; ElevenLabs frames it as the reason voice agents break in production, unable to survive interruption, silence, or context carried across turns, and describes reaching for it through an advanced cascade with in-house speech components³; an independent engineering read of the same release lands in the same place⁴. The disagreement across those four is about where layer two lives, fused into a model or written as orchestration, never about whether it has to exist. Elle writes it: the barge-in gate, echo suppression, the pacer, and turn-sequence cancellation in the voice pipeline are layer two in code. Full citations are in the introduction.

Why a cascade, concretely

Every word Elle speaks passes through a text bottleneck: the reply exists as text she controls before any audio is made. That single property yields most of what matters in a professional deployment:

• Verbatim delivery. Compliance wording (the SMS consent flow, for example) is spoken exactly as written. A fused voice model paraphrases; a cascade cannot.
• A house voice. The voice is a swappable component, not a property of the brain. Elle's English voice, and her Spanish voice, are one config line each.
• Inspectability. Every turn is text in the logs and the database: transcripts, tool calls, token counts. Debugging a voice bug never requires listening.
• Model freedom. The brain is any text LLM behind a provider interface; swapping it is an environment variable, not a migration.

The provider seam

The Twilio bridge does not know which voice architecture it is driving. It depends on a provider-neutral contract, and everything, barge-in handling, tool calls, web pairing , works against that interface:

export interface RealtimeModel {
  readonly currentItemId: string | null;
  appendAudio(base64ulaw: string): void;
  sendText?(text: string): void;      // typed input from a paired browser
  cancelResponse(): void;             // barge-in
  truncate(itemId: string, audioEndMs: number): void;
  sendFunctionResult(callId: string, output: string): void;
  greet?(text: string): void;         // speak before the caller says anything
  close(): void;
}

export type RealtimeFactory = (
  config: RealtimeSessionConfig,
  handlers: RealtimeHandlers
) => RealtimeModel;

Three implementations exist today: deepgramCascade.ts (the production architecture: Deepgram STT → Claude → Deepgram Aura TTS), geminiLive.ts (a fused speech-to-speech model on Vertex AI), and openaiRealtime.ts (the same class of model on OpenAI). The fused pair are live demonstrations of the other end of the spectrum, switchable per call.

The two fused options are not interchangeable in the way that matters operationally. Gemini Live authenticates with Application Default Credentials, so on Cloud Run it is the service account that is already attached: there is no key to mint, store, rotate, or leak, and the calls fall under the same GCP BAA as the compute and Secret Manager. The OpenAI path needs its own credential and its own agreement. That difference, not model quality, is why the Vertex one is the default way to show this architecture off.

It costs something. Twilio, Deepgram, and the OpenAI Realtime API all speak G.711 μ-law at 8 kHz, so those paths are a passthrough. Gemini Live speaks linear PCM, 16 kHz in and 24 kHz out, so caller audio is decoded and upsampled on the way in and the model's audio decimated 3:1 on the way out. Both ratios are integers, which keeps it exact; the conversion is stateful because chunk boundaries are arbitrary, and carrying interpolation state and decimation remainders across chunks is the whole game. Get it wrong and you get a tick at exactly the frame rate, twenty times a second, which is a miserable thing to diagnose from a phone call. npm run verify:audio asserts the chunk-boundary behaviour offline, and npm run verify:gemini drives the adapter through a fake session to cover audio both ways, barge-in, transcripts, tools, the greeting gate, and the close race.

The runtime architecture switch

The active implementation is chosen per call, not at boot. A small persisted setting overrides the environment default, and the media WebSocket resolves the factory when each call connects:

// factory.ts — owner-set override, persisted with an optional deadline
export function setVoiceProvider(p: VoiceProvider | null, ttlMinutes?: number): void { ... }
export function currentRealtimeFactory(): RealtimeFactory { ... }

// index.ts — resolved per connection, so a switch applies to the NEXT call
const mediaWss = createMediaWss(currentRealtimeFactory);

The owner flips it conversationally from the private intake channel ("demo mode on", "back to cascade"); the switch refuses the fused mode when its credentials are absent, so a toggle can never strand a caller. The admin status page always names the active architecture.

Two properties of that override are worth stating, because both were bugs waiting to happen:

• The demo switch has a deadline. The fused architecture costs roughly five to ten times as much per minute, and the realistic failure is not a bad switch, it is a switch nobody remembers to undo: the demo ends, the phone keeps answering on the expensive brain, and the bill arrives a month later. A demo switch therefore defaults to reverting on its own after thirty minutes. The deadline is enforced when the provider is read rather than by a timer, so it outlives the process that set it.
• The override is never cached in memory. Cloud Run can be running more than one instance, and a revision rollover overlaps them deliberately. An in-memory copy would mean the instance that did not handle the switch keeps answering on the old architecture until it recycles, so the settings store is read once per call instead. It is a local SQLite read, measured in microseconds, against a decision made once per call rather than per audio frame.

npm run verify:factory asserts both offline, along with the deadline surviving a recycle and the revert being persisted rather than merely forgotten.

The escalation roadmap: fuse 1+2, keep 3+4

The cascade's one native weakness is that turn-taking decisions are made from silence timers rather than from the sound of the caller's voice. The engineered mitigations (documented here) get within a few hundred milliseconds of natural; a fused listener closes the rest. The target architecture, designed for call-center-scale deployments:

Why this boundary and not more fusion: pulling TTS into the fused model loses the house voice and verbatim delivery; pulling inference in loses tool reliability and the cost tiering (a cheap talker for most turns, the full brain only when needed). Fusing exactly the seam that timers handle badly, and keeping the seams that give control, is the optimum for a customer-service experience. The economics of when this pencils, by call volume, are on the cost page.

Channels share one brain

Everything above concerns the phone. The other channels reuse the same foundations through thinner pipelines:

All five sit on the same persona (persona.ts), knowledge base (knowledge.ts), booking tools (booking/), and memory (db.ts), which is what lets a relationship span channels: a call books a meeting, a text reschedules it, the owner sees it all.