Bilingual mode

Elle speaks English and Spanish, and the caller never has to ask. There is no menu and no announcement: speak Spanish at any point, even mid-sentence, and Elle follows , new language, new voice, same brain. LANGUAGES=en,es (the default) turns it on; LANGUAGES=en restores the original English-only stack, model for model.

Where each layer changes

Detection: per-word tags, majority vote

Deepgram's standalone language-detection feature exists only for prerecorded audio. For live streams, the primitive is Nova-3's code-switching mode: one stream transcribes both languages and tags every word with the language it was spoken in. Each utterance's language is a majority vote over those tags, so one borrowed word ("okay", a project name) can't flip the call:

const multilingual = config.languages.supported.length > 1;
const params = new URLSearchParams({
  ...
  model: multilingual ? config.deepgram.sttMultiModel : config.deepgram.sttModel,
});
if (multilingual) params.set("language", "multi");

// at utterance flush: dominant language across the utterance's words
const tally = new Map<string, number>();
for (const f of this.finals) {
  for (const [lang, n] of f.langWords) tally.set(lang, (tally.get(lang) ?? 0) + n);
}

The cascade follows the caller: an utterance tagged with a different supported language switches the working language, visible in the logs as [voice] caller language: en -> es.

Two voices, one warm socket each

A TTS voice is fixed per WebSocket, so bilingual calls hold one session per voice, English (aura-2-thalia-en) and Spanish (aura-2-celeste-es, configurable via DEEPGRAM_TTS_MODEL_ES). The Spanish socket connects lazily on first use, so English-only calls pay nothing:

private sessionFor(lang: string): TtsSession | null {
  if (!this.tts.openSession) return null;
  let session = this.ttsSessions.get(lang);
  if (!session) {
    session = this.tts.openSession(this.ttsModelFor(lang));
    this.ttsSessions.set(lang, session);
  }
  return session;
}

Which voice speaks a reply

The caller's detected language is the default, but the reply can legitimately be in the other language ("do you speak Spanish?" asked in English deserves a Spanish answer). So each reply chunk gets a lightweight text scan: inverted punctuation, accented vowels, and the most common Spanish function words, none of which appear in normal English prose. Clear signal overrides; ambiguity sticks with the caller's language:

private replyLang(text: string): string {
  if (!spanishEnabled()) return "en";
  const words = text.split(/\s+/).filter(Boolean).length;
  const hits = (text.match(ES_HINT) ?? []).length;
  if (hits >= 2 && hits >= words * 0.2) return "es";
  if (hits === 0 && words >= 4) return "en";
  return this.lang;  // ambiguous: stick with the caller's language
}

The greeting's Spanish beat

The greeting stays English, but in bilingual mode it closes with one Spanish sentence , "Y si prefieres hablar en español, solo dímelo.", spoken by the Spanish voice (the English voice would mangle the pronunciation). Spanish-only callers hear, in their own language, that they can switch. The sentence is part of the uninterruptible greeting, and it's quoted in the greeting note fed to the model, so Elle never awkwardly re-offers Spanish on her first turn.

The persona's mirror rule

The LLM needs no per-language configuration, just an instruction, shared by phone, web, and SMS: reply entirely in the language the person is using; switch when they switch; never mix languages in one reply unless they do. Keeping each reply in exactly one language is also what makes the per-chunk voice choice coherent.

Details that only matter because they'd break

• Echo suppression survives accents. The echo normalizer uses Unicode letter classes (\p{L}) instead of [a-z0-9], the ASCII version shreds "dímelo" into "d melo" and quietly breaks echo matching on Spanish calls.
• Errors apologize in the caller's language. The turn-failure fallback line has an English and a Spanish variant, chosen by the current call language.
• Compliance is per-language. The persona requires the SMS consent flow to cover every element in the caller's language. For regulated deployments, approved scripts should be stored pre-translated and delivered verbatim, never machine-translated mid-call. The cascade's text bottleneck is exactly what makes that guarantee enforceable.

Extending beyond Spanish

Nova-3's multi mode already hears about ten languages; the gate is the voice map, since Aura's catalog covers English and Spanish. Adding French or German means one new entry in LANGUAGES, plus a voice from a second TTS provider wired behind the same Tts interface, a seam the code was shaped for from the start. An utterance in an unsupported language currently continues in the call's working language; a graceful "I speak English and Spanish" fallback is a planned refinement.