Every voice agent — whether it's Vapi, a custom LiveKit stack, or something you build from scratch — shares the same five-layer architecture. If you understand these five layers and the orchestration problem between them, you can reason about any voice agent system.
Highlighted: Audio IN, LLM, and Audio OUT are the layers you architect. STT and TTS are mostly vendor choices.
| Layer | What it does | Key decision |
|---|---|---|
| Audio IN | Receives raw audio from a phone call or browser, handles codecs, buffers, and streaming | Twilio SIP vs. browser WebRTC vs. custom |
| STT | Converts raw audio → text in real time, with incremental (interim) results | Deepgram vs. Whisper vs. platform-bundled |
| LLM | Receives transcript, decides what to say next, optionally calls tools (database, calendar) | Self-hosted vs. API — the latency/cost/capability triangle |
| TTS | Converts LLM output text → streaming audio, often with voice cloning | ElevenLabs vs. Cartesia vs. platform-bundled |
| Audio OUT | Sends synthesized audio back to the caller, matching the original codec and transport | Same as Audio IN — they're paired |
Between these five boxes sits the real-time orchestration layer. This is where most of the engineering complexity lives, and it's what separates a voice agent from a simple "call this API, then that API" chain.
The orchestration layer owns three hard problems:
Is the user still talking? Are they done? Is that a pause between words or the end of a turn? VAD is the gatekeeper — it decides when audio gets shipped to STT and when the agent should respond. Get it wrong and the agent either interrupts constantly or waits awkwardly long.
The user starts talking while the agent is mid-sentence. The system must: stop TTS immediately, discard the rest of the queued audio, start processing the new input, and not feel jarring. Modern frameworks handle this at the audio level — they literally cut the outgoing audio stream and flush the buffer.
Multiple things happen simultaneously: TTS is streaming out while the next STT chunk arrives while the LLM is generating. The orchestration layer manages these concurrent streams, decides whose turn it is, and ensures only one "voice" speaks at a time.
End-to-end latency is the sum of every layer plus network. Here's a realistic budget for a well-tuned stack:
| Component | Latency | Cumulative |
|---|---|---|
| Network (caller → server) | 30–80ms | 30–80ms |
| VAD + audio buffering | 100–300ms | 130–380ms |
| STT (Deepgram streaming) | 200–400ms | 330–780ms |
| LLM inference | 300–800ms | 630ms–1.58s |
| TTS first chunk | 100–200ms | 730ms–1.78s |
| Network (server → caller) | 30–80ms | 760ms–1.86s |
Every production voice agent falls into one of two patterns:
VAD detects end of user speech → full utterance sent to STT → complete transcript sent to LLM → full response sent to TTS → audio played back. Simpler, higher latency, no true barge-in. Works fine for FAQ bots and simple booking flows where turns are short and predictable.
Audio streams continuously. STT emits interim results as the user speaks. LLM starts generating before the user finishes. TTS streams the first audio chunk before the full sentence is generated. Lower latency, true barge-in, more complex. Required for natural conversation.
The era of writing your own VAD from scratch ended in 2025. Modern frameworks provide:
Your job as the architect is choosing between these, then writing the LLM layer — the system prompt, the tool definitions, and the conversation logic. That's where the value lives.
Here's the full streaming flow for a single turn:
1. Caller speaks → audio packets stream over SIP/WebRTC
2. VAD detects speech start → audio frames sent to STT
3. STT emits interim: "I need to" → "I need to book" → "I need to book an appointment"
4. VAD detects 300ms silence → turn is "final"
5. Final transcript + conversation history → LLM
6. LLM streams: "Sure, let" → "me check" → "Dr. García's" → "availability"
7. TTS receives first chunk at "Sure, let" → starts playing audio
8. Caller hears agent responding while LLM is still generating
9. If caller interrupts at any point → TTS stops, go to step 1Before moving on, make sure these land: