What is the primary goal when designing a voice agent for patient care?

The primary goal for healthcare voice agents is often not to delight the user, but to ensure they achieve their end goal without becoming unhappy or complaining to the clinic. This means focusing on clear, efficient communication and avoiding medical advice.

How does Bordy differ from other AI assistants in its approach?

Unlike general AI assistants that focus solely on helping the user, Bordy must consider its impact on its entire network of 200,000 users. It cannot make an introduction or recommendation that might disadvantage others in the network.

What strategies are used to build trust with users interacting with voice agents?

For Flaggo Health, upfront disclosure that the agent is an AI (e.g., 'I'm Sarah, an AI from this clinic') is crucial to avoid breaking trust later. Bordy, on the other hand, breaks the ice by acknowledging it's an AI and then focusing on a charismatic and engaging conversation.

What are the main technical challenges in developing advanced voice agents?

Key challenges include handling turn-taking and interruptions smoothly, especially in multi-participant video meetings where determining when the AI should speak is critical. Achieving sub-second latency for natural conversation flow and managing complex integrations without delays are also significant hurdles.

How do voice agents manage context and personalization across conversations?

While LLMs have large context windows for single conversations, maintaining context and memory across multiple channels (voice, text, email) and over long periods (years) is a significant challenge. Bordy is revamping its voice capabilities to better handle compaction and long-term memory.

What are the biggest surprises when scaling voice agents from demo to production?

One surprise is the relative stability of providers, considering the complexity of voice AI. A major challenge, however, is effectively verifying and simulating voice agent performance, as human interaction is often different from text-based simulations or even agent-to-agent testing.

What advancements are needed in voice AI for builders in the next 12 months?

Builders are looking for better platforms for transcription and analysis. They anticipate more advanced speech-to-speech models, improved intelligence in voice agents, and front-end/back-end architectures that allow agents to speak continuously while performing background tasks, similar to full-duplex communication.

Key Moments

Event Recap: Build Smarter Voice Agents - New York Edition

AssemblyAI

Science & Technology5 min read30 min video

Sep 14, 2026|401 views|2

Save to Pod

Want to know something specific about what's covered?

We've already dissected every moment. Ask and we will deliver (with timestamps).

Key Moments

TL;DR

Voice agents can now participate in video meetings, but managing turn-taking and avoiding interruptions is a major technical hurdle. This requires complex architecture for sub-second response times to feel natural.

Key Insights

Boardy uses voice AI to make introductions between people, with over 50% of intro suggestions now converting to actual meetings.

Flagler Health's voice agent calls patients before procedures, handling instructions and medication reminders, with over 50% of patients preferring phone communication.

Designing for regulated healthcare environments requires strict adherence to flowcharts and guardrails, unlike the more conversational and relationship-driven approach of Boardy.

Boardy's AI must consider a network of 200,000 people, meaning it cannot help one user at the expense of others in the network.

Testing voice agents in production is challenging, with human interaction proving significantly different and harder to simulate than text-based interactions or agent-to-agent testing.

The future of voice AI is moving towards speech-to-speech models and full-duplex capabilities, allowing agents to speak while performing background tasks, reducing latency.

Participating in video meetings: a complex frontier for voice AI

A significant technical challenge highlighted is enabling voice agents to actively participate in video meetings, such as Google Meets. While basic turn-taking in one-on-one voice conversations is largely solved, managing interactions when multiple humans and an AI are present is exceptionally difficult. False negatives (the AI not responding when it should) and false positives (the AI interrupting inappropriately) are both catastrophic. To achieve natural interaction, sub-second latency is crucial. This requires preemptively generating response pipelines so the AI can speak immediately when its turn comes, rather than waiting for the entire pipeline to process after a sentence ends. This advanced capability, though technically demanding, is seen as essential for moving AI from a software tool to a true teammate.

Boardy: voice AI as a conversational matchmaker

Owen Gretzinger from Boardy described their product as akin to a personal LinkedIn, facilitating introductions between individuals like founders, investors, and talent. Voice plays a crucial role because it allows Boardy to understand users more deeply than text alone. By having phone calls or joining meetings, the AI can better grasp priorities and goals, leading to more effective matchmaking. Notably, Boardy aims to be charismatic, funny, and self-aware, often breaking the ice by acknowledging its AI nature upfront. This transparency, coupled with engaging conversation, helps users quickly become comfortable, leading to a more enjoyable and insightful experience. Their intro conversion rate, where a suggested intro leads to an actual meeting, has surpassed 50%, indicating success in their core function.

Flagler Health: voice AI in regulated healthcare

Dylan Wight from Flagler Health discussed their voice agent's role in musculoskeletal clinics. The AI calls patients before procedures to provide instructions and medication information. This application is necessary because many healthcare operations still rely heavily on phone communication, especially with an older demographic that prefers calls over online links. Unlike Boardy's conversational approach, Flagler Health's agent operates within a structured, flowchart-like process with strict guardrails. The primary goal is not to delight the user but to ensure they reach their objective without becoming unhappy or complaining. A critical aspect is preventing the AI from giving medical advice, which could have serious consequences. Ensuring patients actually hear and understand instructions, even when faced with interruptions or voicemails, presents a significant challenge.

Balancing user goals with network considerations

A key distinction for Boardy is that its AI must operate within the context of a large network (200,000 people). This means the AI cannot prioritize one user's needs if it compromises the interests of others in the network. For instance, it won't introduce a user to a top investor unless that user is also of comparable caliber. This constraint shapes the AI's decision-making process, differing significantly from general-purpose assistants that aim to fulfill any user request.

The challenge of building trust with AI

Both speakers emphasized the importance of trust. For Flagler Health, upfront disclosure that the AI is an AI (e.g., 'Hi, I'm Sarah, an AI at this clinic') is crucial. Failing to disclose or delaying it can break trust when users eventually figure it out. For Boardy, while also upfront, the focus is on creating such a good conversation that users almost forget they're talking to an AI. Addressing user skepticism early, perhaps by acknowledging it's their first call with an AI, helps ease tension and facilitates a more natural interaction.

Observability and testing in production

Measuring the success of voice agents involves both quantitative and qualitative metrics. Boardy tracks intro conversion rates and uses agent observability tools to monitor user sentiment (e.g., no yelling or swearing). Flagler Health initially relied heavily on human review of call transcripts to ensure goals were met and to identify issues. Both companies utilize specialized tools for testing and observability, such as BrainTrust and Voicebase, though simulating real-world human interaction remains a significant challenge. Verifying voice agents in production is more difficult than text-based agents, as human responses can be unpredictable and hard to replicate in simulations or agent-to-agent tests.

Handling messy real-world voice interactions

Dealing with practical issues like voicemails, non-standard greetings, and the timing of call connections are significant hurdles. For Flagler Health, correctly identifying voicemails and restarting the correct instruction blocks if interrupted is essential. They also need to ensure the system correctly detects when a patient picks up to start timers accurately. Boardy faces similar challenges with integrations and the potential for slow or unreliable software interfaces. For instance, ensuring accurate collection of email addresses or insurance information without errors is critical.

Future directions: Speech-to-speech and full-duplex AI

Looking ahead, the speakers are excited about advancements in speech-to-speech models, which bypass the need for transcription-to-text pipelines, reducing potential errors and latency. The development of full-duplex communication, where an AI can continue speaking while simultaneously processing new information or performing background tasks, is also seen as a major step forward. This capability is expected to make voice agents more intelligent, responsive, and integrated, moving closer to the ideal of a seamless AI teammate.

Mentioned in This Episode

●Software & Apps

●Companies

Common Questions

In healthcare, voice agents like Flaggo Health's are often highly structured and focused on utility, aiming to avoid errors and deliver specific instructions. In contrast, agents for professional networking, like Bordy, are designed to be more conversational, relationship-driven, and adaptable to a wider range of topics.

Topics

Ai-Ethics AI & Machine Learning Technology & Innovation Business & Entrepreneurship Voice Agents User Experience AI Development AI Applications Conversational AI AI Product Management

Mentioned in this video

Companies

Bordy

A company building a voice agent that acts as a personal matchmaker, connecting people for professional networking, introductions, and other collaborations.

Mentioned as a comparison for Bordy's function, suggesting Bordy is like a personal, callable version of LinkedIn.

Flaggo Health

A company that develops AI solutions for musculoskeletal clinics, including a voice agent for patient communication before procedures.

Twilio

Mentioned as a platform used for managing outbound calls, particularly for ensuring phone numbers are recognized and not flagged as spam.

Brain Trust

A tool used by Flaggo Health for studying and gaining observability into all their calls.

Software & Apps

Codex

Mentioned as an example of a general AI assistant that aims to help the user with whatever they want.

Claude

Mentioned as an example of a general AI assistant focused on fulfilling user requests.

Brockbot

Mentioned as an example of a chatbot that handles compaction, long-term memory, and other complex interactions, similar to Bordy's text messaging capabilities.

Poly Code

Mentioned as a type of AI or capability that is improving and leading users to expect more from voice agents.

Ask anything from this episode.

Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.

Get Started Free