Key Moments
Build Smarter Voice Agents
AssemblyAIWant to know something specific about what's covered?
We've already dissected every moment. Ask and we will deliver (with timestamps).
Key Moments
Voice agents built with cascading architectures offer customization and optimization but struggle with subjective quality, requiring "vibe checks" for TTS. Latency is key, with a 1-1.5 second budget, but LLM reasoning can be dialed down to cut costs.
Key Insights
99% of Retell's agents use a cascading architecture (ASR, LLM, TTS) for greater customization and easier individual layer optimization.
A typical latency budget for voice agents is between 1 and 1.5 seconds; less than 1 second can be unnerving for users.
Evaluating Text-to-Speech (TTS) models is challenging and primarily relies on subjective "vibe checks" by people.
To manage context in long conversations, systems like Super's "scratchpad" or Retell's "dynamic variables extraction" and mini-CRM are used to avoid repetitive questions, which cause 95% negative sentiment.
Testing in production is sometimes necessary to uncover edge cases, and these learnings form a company's "moat" or competitive advantage.
Open-source models are becoming as good as cloud-based alternatives for LLMs, but the high cost of GPUs for self-hosting remains a significant factor.
Cascading architecture offers flexibility and control
The majority of voice agents, around 99% according to Retell AI, are built using a cascading architecture. This approach separates the Automatic Speech Recognition (ASR), Large Language Model (LLM), and Text-to-Speech (TTS) layers. This separation provides significant advantages for developers, including enhanced customization options for end-users and the ability to individually optimize each component of the system. For instance, latency can be fine-tuned at any layer or through combinations of layers. While speech-to-text (S2T) all-in-one models exist, they are considered newer and more experimental, with less established support. Super, a voice layer for property management, also utilizes a cascaded pipeline, highlighting that managing voice infrastructure as core intellectual property (IP) often leads to building this deconstructed pipeline for granular control and the flexibility to integrate new models quickly.
Balancing latency and accuracy is a critical trade-off
A primary consideration in voice agent development is managing latency within a defined budget, typically between one and 1.5 seconds. This budget dictates the acceptable trade-offs for accuracy and the complexity of the LLM's reasoning. Exceeding this budget can lead to a poor user experience. Super aims to stay within this one to 1.5-second window, allowing for user-defined priorities like accuracy or specific LLM scripts. Retell, however, tends to prioritize lower latency across all layers. They find that minor inaccuracies in the LLM can often be mitigated with effective prompting or harness engineering. For TTS, latency is less of a concern as audio can be streamed upon receiving the first token. The main bottleneck becomes receiving that first token from the LLM promptly. Transcription layer latency is generally manageable, with P50 times around 500-700 milliseconds, providing ample leeway.
Evaluating voice agent components requires diverse methods
Evaluating different models for ASR, LLM, and TTS involves distinct strategies. For transcription, word error rate (WER) is a standard benchmark, but developers are moving beyond it to consider more critical errors, such as misinterpreting numbers, which can have a much larger impact. LLMs are evaluated using proprietary benchmarks that test performance in difficult scenarios, often employing another LLM as a judge to compare model capabilities. Text-to-Speech (TTS) evaluation is particularly challenging due to its subjective nature. Here, the primary method involves "vibe checks" conducted by humans, often referred to as VIP checks. This qualitative assessment is crucial for determining if the generated speech sounds natural and appropriate for the intended use case, though it can be time-consuming. The industry is also looking to experts like Hamill Hussein and Shrea Shankar for insights into robust AI system evaluations.
Natural turn-taking and conversation flow rely on subtle cues
Achieving natural turn-taking in voice agents is often a "vibe-driven" process, though quantitative metrics can also be incorporated. Key elements that improve conversational flow include acknowledgments, repetition of user phrases, and setting expectations. Super notes that even seemingly helpful additions, like a typing sound effect to mask latency, can be perceived as deceptive and frustrating by some users. Retell employs a multi-layered approach to enhance turn-taking. This includes leveraging built-in features from ASR providers like AssemblyAI and developing their own turn-taking models trained on historical conversations. These models help the agent discern when a user has finished speaking and it's appropriate to respond. Crucially, agents are tuned to avoid interrupting the human, to quickly yield to human speech if an interruption occurs, and to disregard filler words as interruptions, all contributing to a less robotic interaction.
Context management is vital to prevent user frustration
Maintaining context is paramount for a positive user experience, especially in lengthy or complex interactions. The impact of failing to remember previous information is significant; studies show a 95% negative sentiment when users are forced to repeat information. To combat this, systems employ strategies like Super's "scratchpad," a structured data store that logs key conversation details to build a customer profile. This allows agents to access necessary information for knowledge extraction without requiring the user to reiterate. Retell utilizes "dynamic variables extraction" to capture and reuse information within a conversation and has developed a mini-CRM that integrates with external platforms like HubSpot and Salesforce, enabling agents to personalize greetings and interactions based on persistent user data. This proactive context management is seen as potentially superior to existing customer service systems.
Production deployment uncovers real-world challenges
Transitioning voice agents from controlled demos to production environments presents significant hurdles. Real-world conditions like poor internet connectivity, variable audio quality, and background noise can degrade performance. Good engineering practices, including thorough measurement and understanding quantifiable metrics, are essential. Sometimes, testing directly in production is unavoidable to capture unforeseen edge cases, particularly when targeting specific demographics or geographic locations. These hard-earned lessons from production deployments become a company's competitive advantage, or "moat," as they inform product improvements. Retell addresses production failures by implementing fallback systems for ASR, LLM, and TTS providers. If a primary provider experiences issues, the system quickly switches to an alternative. They also employ latency routing, dynamically directing traffic to providers like OpenAI or Azure to ensure the lowest possible latency, though they prioritize fallback availability over caching tokens.
Future directions include self-improving agents and loop engineering
Looking ahead, there's significant excitement around self-evaluating and self-improving agents. The concept of "loop engineering," coined by Adi Yosmani, envisions defining achievable metrics and allowing agents to iteratively improve by self-assessing their progress against those metrics, rather than relying solely on prompt engineering by humans. This could free up engineering time currently spent on iterative skill and workflow development. Retell is exploring a similar idea where an agent could identify poorly performing calls, diagnose issues, and automatically improve itself without human intervention. This aligns with the idea of agents becoming more autonomous in their development lifecycle.
Cost optimization and open-source adoption are key considerations
Cost is a major concern, especially for large-scale deployments. Measuring token spend is the first step in identifying savings opportunities. Developers can optimize costs by choosing less expensive models, potentially by reducing LLM reasoning parameters, which are a significant cost driver. Utilizing older or open-source models can also offer savings, though the current high cost of GPUs for self-hosting open-source solutions is a trade-off to consider against pay-as-you-go provider costs. For TTS and ASR, while open-source options are emerging, many providers still rely on third-party services. The flexibility of a cascading architecture allows for swapping out components as better or cheaper options become available, whether they are proprietary or open-source.
Mentioned in This Episode
●Software & Apps
●Companies
●People Referenced
Building Smarter Voice Agents: Dos and Don'ts
Practical takeaways from this episode
Do This
Avoid This
Common Questions
A cascading architecture separates ASR, LLM, and TTS components, allowing for individual optimization and customization. Speech-to-speech integrates these, potentially offering convenience but less flexibility.
Topics
Mentioned in this video
A company that provides a platform for building voice agents, emphasizing a cascading architecture for customization and optimization.
A CRM platform that can be integrated with voice agents for persistence of customer information.
A CRM platform that can be integrated with voice agents for persistence of customer information.
A provider for LLM models and TTS services, mentioned in the context of latency routing and fallback systems.
A platform that enables teachers and professors to build AI learning experiences, including voice agent experiences, for students.
Mentioned as a provider of AI voice technology where conversation data is stored locally with the provider.
An ASR provider mentioned as part of the tech stack for voice agents, used for transcription services.
An ASR provider mentioned as part of the tech stack for voice agents, used for transcription services.
A company represented by Chenza, who asks a question about the biggest success factor for voice agent calls.
A cloud platform mentioned as a provider for LLM models and ASR services, used for latency routing and fallback systems.
Mentioned as the former workplace of Adi Yosmani, who coined 'loop engineering'.
Mentioned as an example of an AI voice mode where conversation data is stored locally with the provider.
A recently released open-source model that is considered powerful and suitable for use in voice agents.
A new GPT model that provides a high-quality conversational experience, though it's suggested to still be best utilized within a cascading stack.
More from AssemblyAI
View all 52 summaries
60 minBuild a Voice Agent in an Hour with Claude Code | AssemblyAI Workshop
56 minVoice AI: Beyond Transcription with Granola, CoLoop & EdgeTier
53 minYour Ground Truth Is Wrong: Evaluating STT with truth files & semantic WER | AssemblyAI Workshop
1 minUniversal-3 Pro Streaming: Subway test
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free