Cost engineering

A voice agent's bill is a latency diagram with prices on the edges. This page covers what a conversation costs at list price, the three-layer caching design that removes most of the LLM line, and the crossover math for when self-hosting beats APIs.

Marginal cost per talking minute

For comparison, managed voice-agent orchestration platforms charge roughly 2.5× that for the layer this codebase implements itself (the pacer, barge-in, echo handling, and turn orchestration). SMS runs about a cent per message each way; web chat is LLM tokens only. Fixed monthly costs (an always-warm Cloud Run instance, the phone number, the messaging campaign fee) dominate at portfolio volume.

The problem caching solves

A voice call re-sends its entire prompt on every turn: persona, inline knowledge base, and the whole conversation so far. Without caching, a 20-turn call pays for the same several-thousand-token prefix twenty times. Three mechanisms attack this, layered so each protects the next:

Layer 1: the cross-call static block

The system prompt is split. The static part, persona, knowledge base, greeting note, is byte-identical across every call, so the provider's prompt cache shares one entry across all calls: the first call in a window writes it, every later call (and every later turn) reads it at a tenth of the input price. Per-call context (caller facts, pairing code, clock) rides in a separate dynamic block after the static one, where it cannot break the shared prefix:

let instructionsDynamic = personaContext(facts, this.session?.pairingCode);
...
// Greeting text is config-derived, so this stays in the STATIC block.
const instructions =
  `${personaCore("voice", "inline")}\n\n` +
  `You have already opened the call by saying: ...`;

Layer 2: the conversation breakpoint

Within a call, the fastest-growing prompt component is the conversation itself. A cache marker rides on the final message block of each request, so turn N+1 re-reads turns 1…N from cache instead of re-buying them:

// With `cacheLastMessage`, a cache_control breakpoint is attached to the
// final text/tool_result block, so the next turn re-reads the whole prior
// conversation from cache.
blocks.push({ type: "text", text: last.content,
              cache_control: { type: "ephemeral" } });

The cascade opts every voice turn in (cacheConversation: true); freezing the dynamic block at call start is what keeps each turn's breakpoint valid for the whole call.

Layer 3: turn batching

A caller talking in bursts queues several utterances; the drain loop merges them into one LLM call and one coherent answer instead of an API call (and a spoken reply) apiece. Fewer calls, fewer tokens, no stacked replies.

Proof, not vibes

Every LLM call logs its token accounting, so caching is verified in production, not assumed:

[llm] tokens: in=42 cacheRead=6210 cacheWrite=0 out=58

A healthy call shows cacheRead climbing turn over turn while in stays small. At Haiku list prices, $1/MTok raw input, $0.10 cached read, $1.25 cache write , a warm call bills the bulk of each turn's prompt at the cached rate, and the cross-call block means even a caller's first turn reads the persona from cache if any call ran recently.

When self-hosting wins: the crossover math

This section is the short version; the full band-by-band treatment, with the crossover chart and per-shift trade-offs, is on Architectures by volume.

APIs are pay-per-use; GPUs are pay-always. The break-even is a division, and it was modeled explicitly for this system's call-center variant (assumptions: a single mid-range cloud GPU around $600/month running an open STT + small-LLM + open-TTS stack, versus ≈$0.0075/min in replaced API spend per minute at the cached rates above):

The honest conclusion cuts both ways: at portfolio volume the hosted cascade is the right stack and self-hosting would multiply costs; at call-center volume the same architecture (and especially the fused-talker roadmap, which concentrates most turns on the cheapest component) is the difference between the LLM line dominating the bill and it being a rounding error next to telephony.

Cost controls that are really product decisions

• 120-token reply cap on voice turns, short replies are better phone conversation and the largest single output-cost lever.
• Inline grounding on the phone, a bigger prompt (nearly free once cached) buys the removal of a retrieval round-trip per turn: paying pennies for latency.
• The abuse tiers (security) cap the damage a hostile dialer can do to the bill; the blocked tier never engages the paid pipeline.
• The lazy Spanish TTS socket, bilingual capability costs nothing until a caller actually uses it.