Key Moments
⏭️ Forward Deployed: Voice AI on what works in 2026
Latent Space PodcastWant to know something specific about what's covered?
We've already dissected every moment. Ask and we will deliver (with timestamps).
Key Moments
Voice agents achieve near-human interaction through cascaded models, but complex pipelines and latency trade-offs mean they're still not as seamless as demos suggest, especially for enterprise use cases.
Key Insights
The current state-of-the-art for voice agents uses a cascaded pipeline: Speech-to-Text (STT), Large Language Model (LLM), and Text-to-Speech (TTS), rather than a direct voice-to-voice model due to reliability concerns.
Building effective voice agents requires balancing response intelligence with latency, where more intelligent responses often mean longer wait times for the user.
Reliability is a major concern, necessitating fallback models and strategies to prevent voice agents from failing when primary LLM providers experience outages.
Turn-taking, determining when a human has finished speaking versus pausing, is a non-trivial problem that needs sophisticated models to manage the conversation flow.
While speech-to-speech models offer a more natural, asynchronous interaction, the cascaded approach currently provides better control over guardrails and interpretability for enterprise deployments.
Optimizing latency involves shaving off milliseconds at each stage of the pipeline and employing effective filler words during unavoidable delays, with cost being a significant factor in prompt design.
The cascaded pipeline: a necessary evil for voice agents
The dominant architecture for voice agents today is a cascaded pipeline, comprising three main stages: Speech-to-Text (STT), a Large Language Model (LLM) for processing and response generation, and Text-to-Speech (TTS) for output. This approach is favored over direct voice-to-voice models because the latter are currently less reliable. The STT component transcribes spoken words, often with enhancements like background noise removal and voice isolation. Crucially, a turn detection model is employed to discern when a speaker has finished their utterance, distinguishing between a thoughtful pause and the end of a sentence. This transcribed text is then fed to an LLM, which may infer tool calls or generate a textual response. Finally, the TTS engine converts this text back into speech, ideally streaming it back to the user faster than real-time to maintain a smooth experience. While this multi-step process is complex, it allows for greater control and reliability, which are paramount for enterprise applications.
Balancing intelligence with responsiveness: the latency dilemma
A core challenge in voice agent development is the inherent trade-off between the intelligence and the speed of responses. Users expect natural, human-like interactions, which often require sophisticated LLMs capable of complex reasoning. However, these advanced models can introduce significant latency, leading to delays that break the conversational flow and frustrate users. For instance, a highly intelligent response might take longer to generate, whereas a faster response might be less nuanced or accurate. Engineers must meticulously optimize each step of the pipeline, shaving off milliseconds wherever possible. This optimization extends to parallelizing tasks and strategically using filler words during unavoidable delays, such as waiting for an API call to a customer's system. The goal is to make the agent feel snappy and responsive, even when complex operations are occurring behind the scenes.
Ensuring reliability through fallback mechanisms
Enterprise voice agents must be exceptionally reliable, as system failures can have significant business implications, such as dropping customer calls. This necessitates robust fallback strategies. The underlying LLM providers, while powerful, can experience outages or performance degradation. To mitigate this, developers build 'waterfall' systems of models, where if a primary model fails, a secondary or tertiary model can seamlessly take over. This ensures the voice agent doesn't simply stop working. For inbound use cases, where the bot has limited context about the caller's intent, this becomes even more critical. A comprehensive system needs to handle a vast array of potential user inputs and queries, often requiring extensive knowledge bases and the ability to dynamically select relevant information without overwhelming the LLM, which can suffer from context window limitations.
The complexities of turn-taking and conversation flow
What seems like a simple aspect of human conversation – knowing when to speak – is a significant technical hurdle for voice agents. Accurately detecting when a user has finished speaking, versus pausing mid-sentence or mid-thought, is crucial for a natural interaction. If the agent interjects too soon, it interrupts the user. If it waits too long, the user might feel ignored. Sophisticated voice activity detection (VAD) and smart turn-taking models are employed to manage this. This also ties into the concept of asynchronous processing, where a human brain processes information while listening, potentially interrupting or taking notes. Advanced voice agents aim to mimic this by operating asynchronously, allowing for more dynamic and human-like conversational turns.
Speech-to-speech vs. cascaded: evolving architectures
While the cascaded model (STT-LLM-TTS) is prevalent for its control and interpretability, there's growing interest in speech-to-speech (S2S) models. S2S models aim to capture emotional nuances and operate more naturally, mimicking the human brain's asynchronous processing. However, S2S models can sometimes lag in accuracy or interpretability compared to their cascaded counterparts. A key challenge is maintaining accuracy while achieving naturalness. Many believe the future lies in hybrid approaches, where S2S handles the more fluid, immediate parts of a conversation, while cascaded systems are used for complex tasks or tool calls. This allows for leveraging the strengths of both architectures, potentially leading to more sophisticated and robust voice agents.
Handling multilingualism and accents
Supporting multiple languages and accents adds another layer of complexity. While STT models have improved, achieving high accuracy across diverse languages and phonetic variations, especially for less common ones or specific regional dialects, remains challenging. For instance, what works for English might not translate well to Japanese. Companies often leverage the flexibility of cascaded pipelines to swap out components, such as using custom text-to-speech servers for specific languages or dialects where off-the-shelf solutions fall short. This modularity allows for better adaptation to global markets and a wider range of customer needs, though it introduces further engineering complexity.
Prompt engineering and conversational context
The design of prompts for LLMs in voice agents is critical and can take various forms, from a single, large system prompt to a more dynamic, workflow-driven approach. For inbound use cases with predictable flows, a node or graph-based system can be effective. However, for outbound calls or unpredictable interactions, a single, comprehensive prompt might offer a more centralized 'brain' that can adapt to erratic user behavior. The challenge with large prompts is managing context effectively; LLMs can struggle to retain information from the middle of extensive text. This leads to strategies like context compaction, or the ongoing effort to make LLMs better at reasoning over long contexts, especially when dealing with specific parts of a conversation or a vast knowledge base. The best practice often involves iterative testing and evaluation, using LLMs themselves to judge the quality of responses.
Latency, cost, and continuous evaluation
Latency remains a primary concern, with expectations constantly rising. While zero latency might seem ideal, some 'filler words' can actually make an agent sound more natural by signaling thought processes. Effective filler words can mask delays caused by complex LLM reasoning or slow API calls. Cost is also a significant factor, especially for outbound calls where many users hang up early, and companies pay for tokens used in those brief, unsuccessful interactions. Splitting prompts and using specialized, smaller models can reduce both latency and cost. Continuous evaluation is paramount; using benchmarks for STT, LLM, and TTS components allows developers to test new models and infrastructure, ensuring ongoing performance improvements. Ultimately, building reliable voice agents involves an endless cycle of testing, analysis, and refinement, often involving LLMs as judges to assess reliability and performance.
Mentioned in This Episode
●Software & Apps
●Companies
Common Questions
A cascaded pipeline is the current state-of-the-art for building voice agents. It involves a three-step process: speech-to-text, processing by a Large Language Model (LLM), and then text-to-speech. This approach allows for more control and reliability compared to direct voice-to-voice models.
Topics
Mentioned in this video
Mentioned for its APIs that can spike in latency and cost, and for its models like GPT-4.1.
A company participating in the discussion on voice agents.
A company mentioned as a participant in the voice agents panel.
A company participating in the voice agents discussion, known for its speech-to-speech models and a product called Hydra.
A company mentioned as a potential transcriber for voice agents.
A company that released an impressive voice demo but whose current focus is unclear, possibly hardware-related.
Mentioned as a potential model for cascaded pipelines, balancing speed and intelligence.
Smallest AI's small language model, which can be fine-tuned for real-time voice use cases.
A company mentioned as a participant in the voice agents panel and as a sponsor.
A company run by one of the panelists, specializing in voice agent architecture.
Smallest AI's speech-to-speech model designed for asynchronous operation and multimodal input/output.
Praised for its speed in generating responses for voice agents.
Mentioned as a fast and good model for voice agent applications.
More from Latent Space
View all 245 summaries
84 min🔬 Why Transformers Hit a Wall the Moment Physics Shows Up — Anima Anandkumar, Caltech
48 minExo: Harnesses should see their own code and logs — Alex Krentsel
96 min🔬Biology Is Turning Into Software — Matt McPartlon and Neil Patil, Chai Discovery
103 minThe Inference Frontier: 10x Faster Models to Self-Optimizing AI — Philip Kiely & Ali Taha, Baseten
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free