Diagrams

Every diagram in the documentation, collected in one place: the sequence diagrams for each live flow, the entity relationship diagram of the data layer, and the roadmap architecture. Each one also lives in context on its home page, linked below. All are generated from the actual code and schema, and follow your light or dark theme.

Call setup and the greeting

From dial tone to Elle's first word: two security gates, then a greeting engineered to be uninterruptible. Full context on the voice pipeline.

CallerTwilio/voice-realtimeBridgeCascadedials the numberPOST webhook (signed)verify Twilio signature · assess caller tier (normal / limited / blocked)TwiML <Connect><Stream> + HMAC media tokenWS /media · start frameverifyMediaToken: recompute HMAC, 5-min age, constant-time comparecreate model (static persona + per-call context)greet(): deterministic line, no LLM round-tripgreeting audio, streamed as it synthesizespaced 20 ms frames · first audio ≈300 msbarge-in stays locked until every greeting frame has been SENT
Call setup: signature gate, media-token gate, then a greeting that always plays to the end.

One spoken turn

The heart of the system: streamed hearing, streamed thinking, streamed speaking, all overlapped. Full context on the voice pipeline.

CallerDeepgram STTCascadeClaudeAura TTSBridgeμ-law audio, streamedinterim transcripts (echo-filtered)speech_final → utterance + language tagmerge queued utterances into ONE turnwarm(): connect overlaps thinkingchatStream (cached prefix + conversation)tokens… first full sentenceSpeak(sentence 1) while generation continuesaudio chunks → exact 160-byte framesonAudioDelta per 20 ms framedrift-corrected playback · ≈1 s to first word
Sentence streaming is the latency lever: Elle starts speaking while the rest of the reply is still being generated.

Barge-in

Three subsystems cancel from one incremented integer, and the model's memory is truncated to what the caller actually heard. Full context on the voice pipeline.

CallerDeepgram STTCascadeBridgeTwiliotalks over Elleraw VAD ignored: needs ≥6 chars of recognized, non-echo speechbarge-in triggeronSpeechStartedgreeting gate: ignored until the opening line has fully playedcancelResponse() → turnSeq++LLM loop + TTS pump bail at next checkpointclear → flush queued playbacktruncate(itemId, playedMs)context now matches exactly what the caller actually heard
The interrupted turn doesn't error; it quietly stops being real. The caller's words become the next turn.

Booking and consent

Tentative holds, human confirmation, and the carrier-compliant consent script. Full context on booking and compliance.

CallerElle (model)booking.tsGoogle Calendarsms.ts"Tuesday at 2 works" + name + emailreads date, time, timezone back → clear yesbook_intro_call(name, email, start ISO+offset)validate email shape + timestampfreeBusy(start, +30 min)slot freeinsert event "pending confirmation"bookings row · status = pendingtool result: TENTATIVE, say owner confirms by emailfull consent script → plain yes-or-nosave_user_fact("sms_consent") → confirmation SMSconsent fact + confirmation text = the audit trail; STOP/HELP enforced by Twilio upstream
The AI never finalizes: every hold is pending until the owner confirms, and texts require the spoken consent script.

Web pairing

A spoken code joins a browser to a live call, in both directions. Full context on web, SMS and intake.

Caller (phone)CascadeCallSessionBrowser"can I see that on screen?"speaks the pairing code, digit by digitPOST /api/pair {code} · 10/min/IPWS /ws joined to this call's sessionboth sides' transcripts as they happenlive transcript eventsshow_work / show_choices tool outputtyped message (length-capped)injectText → queued as a real turnElle answers OUT LOUD on the callsession dies with the call; codes are unusable after hangup
Two-way pairing: the phone drives the screen, and typed input flows back into the spoken conversation.

Owner-directed outreach

The private channel: two silent gates in, a compliant first text out, and a ledger that lets the SMS agent recognize the reply. Full context on web, SMS and intake.

Ownertelegram.tsTwilio SMSContactsms.ts agent"text Jane at +1 555… about the role"gate 1: secret_token header · gate 2: owner id; anyone else is dropped silentlyconfirms name, number, topic"yes, send it"compliance-complete first-contact SMSoutreach row · status = sentdelivered from the public numberreply textoutreach → replied · thread context injectedreply that already knows who they are and why"how’s the Jane thread going?" → status from the ledger
The owner directs; Elle executes and remembers. When the contact replies, the SMS agent already has the context.

Entity relationship diagram

The complete SQLite schema. Full context on data and knowledge.

callersphone_numberTEXTfirst_seen_atTEXTcallscall_sidTEXTFK phone_numberTEXTstarted_atTEXTended_atTEXTfactsidINTEGERFK phone_numberTEXTcategoryTEXTfactTEXTFK call_sidTEXTcreated_atTEXTtranscriptsidINTEGERFK call_sidTEXTroleTEXTcontentTEXTcreated_atTEXTbookingsidINTEGERFK phone_numberTEXTFK call_sidTEXTnameTEXTemailTEXTtopicTEXTstart_atTEXTstatusTEXTevent_id / event_linkTEXToutreachphone_numberTEXTnameTEXTtopicTEXTmessageTEXTstatusTEXTsms_consentphone_numberTEXTstatusTEXTsourceTEXTupdated_atTEXTsettingskeyTEXTvalueTEXTupdated_atTEXTplacesis described byrecordsproducesrequestsgrants texting viais targeted byNotationexactly onezero or oneone or manyzero or manyunderlined attribute = primary key · FK = correlational key (convention, not enforced)many-to-many: a foot on both ends, resolved through a junction table.None exists in this schema; every relationship hangs off one person or one call.
The full schema in crow's-foot notation. A caller places calls, calls record transcripts and produce facts, and one-row-per-number tables (consent, outreach) hang directly off the person.

Cost crossovers by volume

Where each architecture is cheapest, and the two division-driven crossover points. Full context on architectures by volume.

1K10K100K1M10M$10$100$1K$10K$100Ktalking minutes per month (log)replaceable infra $/month (log)single-GPU ceiling80K min/mo: first GPU pencils4.2M min/mo: fleet wins outrighthosted APIs (cached cascade)one self-hosted GPUself-hosted fleet
Where each architecture is cheapest (log-log, schematic). Below the first dot, hosted APIs win by an order of magnitude; between the dots, hybrids pencil; past the second, a fleet wins outright. Telephony (~$0.009/min) is excluded; every option pays it equally.

Roadmap: the director escalation

The future architecture, fusing the listening and judgment layers while keeping inference and the house voice separate. Full context on architecture.

CallerDuplex talkerInference brainHouse TTSaudio: the talker hears AND judges turns nativelyroutine turn: answered directly via house TTShard question (pricing, policy, tool work)ask_brain(question), a tool call not a transferpre-synthesized backchannel: “let me check that”sentence 1 streamed while still generatingfirst audio ≈ 1 s, same house voicespoken text fed back as the talker's contexttext bottleneck preserved: verbatim compliance wording + swappable voice survive
The future architecture keeps inference and TTS separate on purpose: cost tiering, tool reliability, and verbatim delivery all live at that seam.