Key Moments
Building Voice Agents with Vapi & AssemblyAI
AssemblyAIWant to know something specific about what's covered?
We've already dissected every moment. Ask and we will deliver (with timestamps).
Key Moments
AI voice agents can now conduct complex customer service calls and training simulations, but issues with hallucination and noisy environments still pose challenges.
Key Insights
The performance, speed, and cost of transcription, LLMs, and text-to-speech models have improved significantly, reaching human-like performance when orchestrated together.
Vapi's workflow platform allows for step-by-step conversational logic, mitigating the hallucination issues common with long, complex prompts, especially with smaller models like GPT 3.5.
The Vapi platform integrates with various LLMs (e.g., Anthropic's Claude 3, OpenAI models) and text-to-speech providers (e.g., 11 Labs), allowing users to bring their own API keys or use Vapi's preferred pricing.
The demo showcased a voice agent simulating a dental office appointment booking, with a total latency across transcription, LLM, and text-to-speech of approximately 1200-1500 milliseconds.
Handling interruptions in voice agents relies on fast transcription (under 300ms) and voice activity detection to gracefully back off and allow the user to speak.
Background voice cancellation remains a challenge, as transcription models are tuned to pick up all speech, requiring custom models or smarter transcription that can ignore background noise.
The rise of human-like AI voice agents
The recent surge in AI voice agents is driven by dramatic improvements in underlying AI models. Transcription, Large Language Models (LLMs), and text-to-speech (TTS) technologies have become significantly faster, cheaper, and more performant. This convergence has reached a point where AI can orchestrate these models to achieve human-level conversational abilities, effectively passing the Turing test in many interactions. Companies are investing heavily because users naturally prefer interacting with systems that communicate like humans. This technological leap has unlocked new possibilities across various sectors, making sophisticated voice interactions a reality for businesses and consumers alike.
Unforeseen use cases and core functionalities
While initially focused on enterprise solutions like appointment scheduling, AI voice agents have found surprising value in niche areas. Jordan, CEO of Vapi, highlighted applications in training and coaching, such as role-playing scenarios for call center agents and sales teams. Entertainment applications, like conversing with fictional characters, have also emerged. However, the most significant value remains in customer service, where the sheer volume of annual phone calls represents a massive opportunity to reduce costs and improve customer experience by minimizing hold times. Vapi aims to serve both enterprise needs and consumer-facing applications.
Real-time interaction and platform integration
Effective voice agents require more than just understanding; they need to process speech in real-time. AssemblyAI's streaming API offers high-accuracy transcription with low latency, typically within a few hundred milliseconds, which is crucial for fluid conversations. Vapi acts as the bridge between these powerful models and production-ready voice agents. Their platform allows users to configure and deploy agents, integrating components like AssemblyAI's streaming transcription, LLMs (including options from OpenAI and Anthropic), and TTS providers like 11 Labs. The demo illustrated this by showcasing an assistant configured for a dental office, capable of gathering appointment details and pushing them to a spreadsheet.
Managing latency and cost across the stack
In real-time conversational applications, minimizing latency is paramount to maintaining conversational flow. Vapi's platform provides visibility into costs and latency across the entire stack. For a typical agent, transcription might take 100-300 milliseconds, an LLM like GPT-4 Mini could add 300 milliseconds, and TTS might add another 300 milliseconds. The goal is to keep the total latency between 1200-1500 milliseconds. This tight control over latency and cost is essential for creating seamless user experiences, and Vapi allows users to either use their own API keys or leverage Vapi's preferred pricing with underlying providers.
From prompts to workflows for reliable logic
A significant challenge with complex AI agents is prompt engineering, where long or intricate prompts can lead models to hallucinate or go off-track, especially with smaller, lower-latency models. Vapi addresses this by introducing a workflow platform. Instead of relying solely on prompts, users can design step-by-step conversational flows. This ensures that the agent follows a predetermined logic, such as confirming details, making API requests, or transferring calls, guaranteeing a more predictable and secure business process. Users can build these workflows from scratch using various block types, including actions, information gathering, API calls, and conditional logic.
Handling interruptions and maintaining context
User interruptions are a critical part of natural conversation. Vapi handles this through a combination of Voice Activity Detection (VAD) models to identify speech and rapid transcription services. When speech is detected and transcribed within milliseconds, the agent can intelligently back off or adjust its response. Maintaining conversational context is also key. While prompt-based systems might lose context over long conversations due to token limits, Vapi's workflow platform is developing 'global state' or 'global memory' capabilities. This allows agents to save and recall information from earlier in the conversation, ensuring continuity even if a specific workflow step lacks direct context.
Navigating noisy environments and hybrid human-AI models
Real-world environments are rarely quiet, presenting challenges for voice agents. While background noise cancellation is a relatively solved problem, background *voice* cancellation is more complex. Transcription models are designed to capture all speech, including background conversations. Vapi employs custom background voice cancellation models and filters audio before transcription, but smarter, more intelligent transcription models that can be instructed to ignore background speech are still needed. Furthermore, the adoption of AI voice agents is often not a 'rip and replace' scenario. It's more common for AI agents to augment human agents, handling routine tasks (like IVR systems or repetitive transactional calls) to free up humans for higher-value work or escalations. Vapi facilitates smooth handoffs, including 'warm transfers' where the AI can brief the human agent before the caller is connected.
Future of voice agents: Speech-to-speech and regulated industries
Looking ahead, the most exciting development is the advent of native speech-to-speech models. These models can natively process audio input and generate audio output, eliminating the latency and context loss associated with chaining separate transcription, LLM, and TTS models. This end-to-end approach allows agents to better interpret nuances like frustration and respond more empathetically. While progress has been slower than anticipated, these models are expected to significantly improve latency and conversational quality. In regulated industries like telehealth, where data privacy is paramount, Vapi provides contractual guarantees and architectural designs to ensure no patient data is stored or trained on, making their platform suitable for highly sensitive applications due to the high ROI potential.
Mentioned in This Episode
●Software & Apps
●Companies
Building and Deploying Voice Agents with Vapi
Practical takeaways from this episode
Do This
Avoid This
Common Questions
AI voice agents are gaining traction because transcription, LLM, and text-to-speech models have become significantly faster, cheaper, and more performant. This advancement allows them to approach human-level performance when orchestrated together, making voice interactions with AI feel more natural and accessible.
Topics
Mentioned in this video
A large language model from Anthropic supported by Vapi, offering advanced capabilities for voice agents.
A specific large language model mentioned as part of the latency budget calculation for voice agents, offering a balance of performance and speed.
Mentioned as a smaller, lower-latency model that might 'go off the rails' in complex prompt-based conversations, contrasting with the more robust Vapi workflows.
A company providing AI solutions, specifically their streaming API for real-time transcription, which is integrated into Vapi's platform.
A company providing text-to-speech models, used in the Vapi demo for voice generation and for creating a voice clone of Jordan.
A platform that simplifies building, testing, and deploying AI voice agents, enabling real-time voice interactions.
A provider of large language models, including their Claude 3.5 Sonnet, which Vapi supports for voice agent development.
A leading AI research company whose models, including their real-time API, are supported by Vapi for building voice agents.
More from AssemblyAI
View all 53 summaries
47 minBuild Smarter Voice Agents
60 minBuild a Voice Agent in an Hour with Claude Code | AssemblyAI Workshop
56 minVoice AI: Beyond Transcription with Granola, CoLoop & EdgeTier
53 minYour Ground Truth Is Wrong: Evaluating STT with truth files & semantic WER | AssemblyAI Workshop
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free