Building voice agents with LLMs means wiring a streaming speech-to-text engine, a large language model, and a low-latency text-to-speech engine into a single real-time conversation loop. The cascaded STT → LLM → TTS pipeline remains the most flexible and debuggable architecture in 2026, letting you swap providers, add tool calling, and control latency stage by stage. VideoSDK's open-source AI Agent SDK provides this pipeline as a managed, production-ready foundation you can extend today.
Voice is the fastest interface a user will ever touch. Speaking a question takes seconds; typing it takes longer, and navigating an IVR menu tree takes longer still. That gap is why building voice agents with LLMs has moved from research demo to production budget line in a single product cycle.
The economics finally work. Streaming speech-to-text engines now transcribe conversational speech with word error rates in the single digits, LLM inference costs have fallen by an order of magnitude since 2023, and neural text-to-speech has crossed the threshold where most listeners cannot reliably distinguish synthetic speech from a human recording. Stack those three curves together and a voice agent that handles a real customer conversation for pennies per minute is now an engineering project, not a moonshot.
The use cases are concrete: customer support lines that resolve tier-one tickets without a human, outbound appointment reminders that accept a spoken "yes, move it to Thursday," and internal assistants that read a warehouse worker their next pick order while their hands stay busy. Telehealth intake, insurance claims, and debt collection follow the same pattern: structured, repetitive, high-volume conversations.
There are two broad architectural camps. The cascaded pipeline chains speech-to-text, an LLM, and text-to-speech as separate, swappable components. Speech-to-speech (S2S) models process audio in and produce audio out in a single network. This guide walks through both, then gets practical about provider selection, turn-taking, memory, and deployment.
Core Architecture of a Voice Agent
Every production voice agent built with LLMs today is a latency-budget orchestration problem. You have roughly 800 milliseconds to a full second before a conversation starts feeling broken to the caller, and every millisecond you spend in one stage is a millisecond you cannot spend in another. Understanding where time goes is the difference between an agent that feels present and one that feels like a bad phone line.
The Cascaded Pipeline Explained
The cascaded pipeline is defined as a chain of three independent models: a streaming speech-to-text (STT) engine converts the caller's audio into partial text hypotheses in real time, a large language model (LLM) reasons over that text to decide what to say or do, and a text-to-speech (TTS) engine converts the response into audio, ideally streamed so playback begins before the full sentence is generated.
The critical design property is that each stage is independently replaceable. If a cheaper STT provider matches your accuracy needs, you swap it without touching the LLM logic. If a new LLM drops your reasoning cost by half, the audio layers never know. This modularity is why the cascaded approach dominates production deployments despite the theoretical elegance of end-to-end models.
Tool calling slots in between the LLM and the TTS stage: when the LLM decides it needs external data, such as an order status or a calendar slot, it emits a structured request, your orchestration layer executes it, and the result feeds back into the LLM's context before speech is generated.
Alternative: Speech-to-Speech Models
Speech-to-speech models are defined as single neural networks that accept raw audio and emit raw audio, skipping the text intermediate representation entirely. OpenAI's Realtime API and Google's Gemini Live are the most prominent commercial examples, and they can achieve genuinely lower end-to-end latency because there is no handoff between three separate services.
The trade-offs are real. S2S models give you less visibility into what the model "heard" and "decided," which makes debugging, compliance logging, and transcript-based analytics harder. Tool calling support exists but is less mature than in text-native LLMs. Provider lock-in is stronger because the entire stack is one vendor's model, and cost per minute is typically higher than composing commodity STT and TTS around a small LLM.
In practice, teams building regulated or analytics-heavy workloads choose the cascaded pipeline for its transcript trail and control, while teams optimizing purely for conversational latency in consumer-facing assistants increasingly evaluate S2S. A hybrid pattern is also emerging: cascaded pipeline for the main flow, with an S2S model handling rapid backchannel acknowledgments like "mm-hm" and "sure."
Selecting the Right STT Provider
Streaming support is the single non-negotiable criterion when choosing a speech-to-text provider for a voice agent. Batch transcription after the user stops speaking adds hundreds of milliseconds of avoidable latency; streaming STT emits partial hypotheses while the caller is still talking, which lets your LLM begin reasoning before the turn ends.
Deepgram's streaming models are a common first choice for telephony-grade voice agents, with strong accuracy on conversational audio and low latency over WebSocket connections. OpenAI Whisper excels at batch and offline transcription and powers many post-call analytics pipelines, but its streaming variants are less latency-competitive for live calls. Cloudflare Workers AI offers an STT option that is attractive when your entire stack already runs at the edge, trading some accuracy for geographic proximity to the caller. Cartesia's Ink model targets the real-time segment with a focus on low-latency streaming transcription.
Language coverage matters more than most teams expect at the start. If your agent will serve callers in multiple languages, verify each provider's per-language accuracy independently, because a vendor's headline English numbers tell you little about its Hindi or Portuguese performance. Also confirm telephony audio support: models trained on studio-quality 16 kHz audio degrade noticeably on 8 kHz phone-line speech, so test with real call recordings, not podcast samples.
Choosing an LLM for Voice Agents
The LLM is the brain of the system, and for voice agents the selection criteria differ from chatbot selection in one crucial way: response length and generation speed matter more than raw reasoning depth, because every token generated is time the caller spends waiting.
Model Options
For most voice workloads, a fast, mid-tier model beats a flagship reasoning model. OpenAI's GPT-4o-mini class of models delivers conversational competence at a fraction of the cost and latency of the largest models, and Anthropic's Claude and Google's Gemini families both offer fast tiers well suited to spoken dialogue. The reasoning-heavy frontier models are worth their latency only when the conversation genuinely requires multi-step deduction, such as complex claims triage.
Self-hosting via vLLM changes the calculus for privacy-sensitive deployments. Running an open-weight model on your own GPU infrastructure eliminates per-token API costs and keeps caller data entirely inside your network, at the price of operational burden and capacity planning. A common production pattern is a fast hosted model for general turns with a fallback to a larger model for hard cases.
Enabling Tool Calling
Tool calling, also called function calling, is what turns a chatty model into a useful agent. Instead of only generating text, the LLM emits a structured request, such as "look up order 4471," your orchestration layer executes it against your real systems, and the result returns to the model's context to shape the spoken answer.
For voice, tool calling unlocks the interactions callers actually want: checking an order status, booking a calendar slot, rescheduling a delivery, or transferring to a human when the conversation exceeds the agent's scope. Two practices keep tool-augmented voice turns fast. First, keep tool responses compact, because a verbose API result bloating the context window directly slows the next generation. Second, pre-warm common lookups where possible, so the model's tool request resolves in tens of milliseconds rather than a full round trip.
VideoSDK's AI Agent SDK supports function tools natively, including parallel tool calling for fan-out requests, which matters when a single turn needs both an order lookup and a calendar availability check.
Text-to-Speech (TTS) Choices
TTS is the stage callers judge most directly, because they hear it. Voice quality, streaming behavior, and cost per character are the three axes to evaluate.
ElevenLabs remains the quality benchmark for expressive, human-sounding voices, and its streaming API lets playback begin before the full response is synthesized. Cartesia's Sonic models are engineered specifically for real-time conversation, with some of the lowest latency figures in the category, making them a strong fit when responsiveness outweighs maximum expressiveness. Cloudflare Workers AI provides a TTS option that pairs well with an edge-deployed stack and keeps audio generation close to the caller. Open-source options such as Piper and Coqui-class models enable fully self-hosted deployments where no caller audio or text may leave your infrastructure.
Cost compounds quickly at scale. A support line handling a million minutes per month will feel the difference between per-character pricing tiers acutely, so model your real volume before committing. A practical pattern is tiered voices: a premium voice for customer-facing brand moments and a cheaper, faster voice for internal or high-volume flows.
Implementing Barge-In and Turn-Taking
Barge-in is defined as the caller's ability to interrupt the agent mid-sentence and have the agent stop, listen, and respond to the new input, exactly as a human would. Without barge-in, a voice agent feels like a voicemail message; with it, the conversation feels alive.
The mechanism rests on voice activity detection (VAD). A lightweight VAD model runs continuously on the incoming audio stream and signals when the caller starts speaking. When the agent detects speech during its own playback, it must immediately cancel the TTS audio, discard the in-flight response, and hand the turn back to the STT and LLM stages.
Best practice is to make interruption cancellation atomic: audio stops within tens of milliseconds, not after the current sentence finishes. Equally important is what the LLM does next. A good barge-in recovery acknowledges the interruption naturally rather than restarting its previous monologue. VideoSDK's Agent SDK handles this loop with built-in turn detection, VAD, and preemptive response cancellation, which is otherwise one of the fiddliest pieces of the pipeline to hand-roll.
Managing Conversation State
A voice agent without memory repeats itself and loses the caller's context mid-call. A voice agent with unbounded memory becomes slow and expensive. State management is the discipline of staying useful between those failure modes.
Session state falls into three layers. Short-term history is the rolling transcript of the current call, kept as compact as possible because every token of history adds latency and cost to each new turn. Working state is the structured data extracted from the conversation so far, such as the caller's account ID, stated intent, and collected slot values, stored as typed fields rather than raw transcript. Long-term memory persists across sessions, enabling an agent to greet a returning caller by recognizing their history.
The standard technique for keeping context without blowing token limits is summarization and extraction: as the transcript grows, older turns are compressed into a summary while structured facts are lifted into the working state. VideoSDK's Agent SDK supports both memory and context window management primitives for exactly this pattern, and its Conversational Graph layer takes it further by storing conversation state in a typed schema with checkpointing, so a loan application or booking flow can pause, resume, and never lose a collected field.
Deployment Options and Scaling
Where your voice agent runs determines its latency floor, its privacy posture, and its cost curve. The choice between cloud-native and self-hosted is usually decided by data sensitivity and expected volume, not by feature checklists.
Cloud-Native Deployment
Cloud-native deployment means running your agent orchestration on managed infrastructure close to your callers. Cloudflare Workers is the extreme end of this: your logic executes at the edge, minimizing the network hop for both STT and TTS. Cartesia's managed offerings similarly target real-time workloads. The advantages are fast time-to-market, elastic scaling, and no GPU capacity planning. The trade-off is that caller audio and transcripts transit third-party infrastructure, which may be disallowed in healthcare, finance, or public-sector deployments.
VideoSDK's Agent Cloud offers a middle path: a managed runtime for the open-source Agent SDK where sessions, scaling, and observability are handled for you, with the SDK itself remaining open and portable if you later move it onto your own infrastructure.
Self-Hosted Deployment
Self-hosting stacks FreeSWITCH or another softswitch for telephony, Pipecat or VideoSDK's open-source Agent SDK for pipeline orchestration, and vLLM for serving open-weight LLMs on your own GPUs. The payoff is total data custody, per-call costs that fall with volume, and freedom from provider rate limits.
The cost is operational: GPU capacity planning, failover, monitoring, and keeping model versions current all become your team's job. Self-hosting typically wins above sustained high volume or under strict data residency rules; below that threshold, managed infrastructure is almost always cheaper in total cost of ownership.
Monitoring, Testing, and Optimization
You cannot tune what you do not measure, and voice agent latency is a sum of stage-level timings that each need their own gauge. The headline metric is time-to-first-audio: the interval between the caller finishing their turn and hearing the agent's first syllable. Beneath it, track STT finalization time, LLM time-to-first-token, tool call duration, and TTS time-to-first-audio, because a slow agent is always exactly one of these stages misbehaving.
Track error rates per stage as well: STT empty transcripts, LLM refusals or malformed tool calls, and TTS synthesis failures each degrade the caller experience differently and need distinct alerting. Load testing before launch matters because voice pipelines fail in non-linear ways under concurrency; a stack that handles ten simultaneous calls gracefully may collapse at fifty due to connection pool exhaustion or GPU queueing.
Tuning follows a simple hierarchy: cut unnecessary tokens from prompts and history first, enable streaming at every stage second, and move compute geographically closer to callers third. Pipeline observability tooling, such as the hooks built into VideoSDK's Agent SDK, gives you per-stage traces so optimization effort lands where the milliseconds actually are.
Future Trends in LLM Voice Agents
Three shifts are worth watching through 2026 and beyond. Speech-to-speech models will keep improving and will absorb more of the pipeline for latency-critical consumer experiences, though the cascaded approach retains the compliance and observability advantages that enterprise deployments require. Multimodal agents that see screen shares, camera input, or documents mid-conversation are already emerging, and voice becomes one channel among several in a single session. Edge AI is pushing inference onto devices, enabling fully offline voice agents for privacy-first and connectivity-poor scenarios. VideoSDK's support for vision and multi-modality in its Agent SDK is an early signal of where agent interactions are heading.
Definitions Glossary
Cascaded Pipeline: A voice agent architecture chaining streaming STT, an LLM, and streaming TTS as independent, swappable stages, with tool calling inserted between reasoning and speech generation.
Speech-to-Speech (S2S) Model: A single neural network that consumes audio and emits audio directly, offering lower theoretical latency but less transparency and weaker tool-calling maturity than cascaded pipelines.
Barge-In: The caller's ability to interrupt the agent mid-response, requiring immediate TTS cancellation and turn handoff driven by voice activity detection.
Tool Calling: The LLM mechanism for emitting structured requests to external systems, such as order lookups or calendar bookings, whose results feed back into the conversation context.
Time-to-First-Audio: The key latency metric for voice agents, measuring the interval between the end of the caller's turn and the start of the agent's audible response.
Agent Worker: The Python process in VideoSDK's AI Agent SDK that runs the voice pipeline and manages each session's lifecycle, deployable on Agent Cloud or self-hosted.
Key Takeaways
- The cascaded STT → LLM → TTS pipeline remains the dominant architecture for building voice agents with LLMs in 2026 because each stage is independently swappable, debuggable, and cost-optimizable.
- Streaming at every stage, from partial STT hypotheses to chunked TTS playback, is what keeps end-to-end latency inside the roughly one-second budget callers perceive as natural.
- Tool calling is the difference between a talking model and a working agent; keep tool responses compact and pre-warm common lookups to protect latency.
- Barge-in and turn-taking, driven by voice activity detection, are the make-or-break interaction details that separate a conversational agent from a voicemail.
- Choose cloud-native deployment for speed to market and elastic scaling, and self-hosting for data custody and high sustained volume; VideoSDK's open-source Agent SDK runs in both worlds.
Conclusion
The roadmap for building voice agents with LLMs is now well-trodden: pick a streaming STT engine, a fast LLM with tool calling, a low-latency TTS voice, wire in barge-in and state management, then deploy where your latency and privacy requirements point you. The fastest way to start is VideoSDK's open-source AI Agent SDK, which ships the full pipeline with turn detection, memory, and observability built in, deployable on Agent Cloud or your own infrastructure. Explore the code samples and join the VideoSDK Discord community of 3,000+ developers. What are you building with LLM voice agents? Drop a comment, I'd love to hear what use case you're working on.
Free $20 Balance for AI Voice Agents & Video Calls
FAQ