How does Ambience's AI assistant work with EHR systems?

Ambience uses a mobile app that doctors take into patient rooms. The app listens to conversations, transcribes them, and uses fine-tuned language models to generate documentation, which is then automatically written back into the electronic health record (EHR) systems like Epic or Cerner via their APIs.

What is Reinforcement Learning from Human Feedback (RLHF) and why is it useful in medicine?

RLHF is a method for fine-tuning language models to improve their reasoning. In medicine, it's valuable for objective tasks and complex medical reasoning that standard models struggle with. It optimizes for the end objective rather than just proxy metrics like loss, and is more sample-efficient than supervised learning.

What are common issues with RLHF, like 'reward hacking'?

Reward hacking occurs when models find ways to maximize scores unrealistically. In Ambience's physical exam use case, the model inflated findings or used layman's terms. This was addressed by updating the grader to constrain the model and add style weighting.

How does Ambience evaluate the performance of its AI models, specifically for tasks like ICD-10 coding?

Ambience evaluated RLHF for ICD-10 coding by recruiting physicians and comparing their performance against expert annotators. They found that RLHF significantly improved model performance, pushing results from around 40% F1 score to 57% with a small model, indicating potential for complex tasks.

Beyond scribing, what other AI applications is Ambience developing for healthcare?

Ambience is expanding into patient-facing agents to help with medication adherence and appointment follow-ups. They also offer 'pre-charting' to summarize past patient visits for specialists like oncologists and cardiologists, saving them time before appointments.

What are the major challenges with AI in healthcare, particularly regarding hallucinations and data?

Base AI models struggle with robustness in healthcare, a patient-safety critical environment. Realistic clinical data is often locked in EHRs or contains 'tribal knowledge' not found online. Hallucinations are a concern, and models can make incorrect inferences or diagnoses if not properly fine-tuned on domain-specific data.

What kind of talent is Ambience looking for to drive innovation in clinical AI?

Ambience seeks top-tier machine learning research talent and a unique 'clinician researcher' archetype. This ideal candidate combines domain knowledge with an experimentalist mindset and a startup operating system, a rare but crucial combination for tackling frontier AI use cases in healthcare.

⚡️Using RFT to Build Clinical Superintelligence

Latent Space Podcast

Science & Technology7 min read27 min video

Jul 29, 2025|1,071 views|34|1

Save to Pod

Key Moments

On this page

TL;DR

AI in healthcare: Ambience AI uses RFT to reduce doctor note-taking, improve patient care, and expand AI applications.

Key Insights

Ambience AI employs a mobile app for doctors to record patient interactions, which are then transcribed and used to generate medical documentation.

The company integrates with major EHR systems like Epic and Cerner to streamline the documentation process, aiming to save doctors significant time.

Reinforcement Fine-Tuning (RFT) is a key technology for Ambience, enabling models to optimize for objective tasks and end-user goals rather than just imitating data.

RFT's sample efficiency and ability to 'hill climb' towards maximizing scores make it valuable for complex medical reasoning and objective tasks like ICD-10 coding.

Challenges in RFT include reward hacking and tone degradation, which Ambience addresses by refining its scoring functions and incorporating style evaluations.

Ambience is expanding beyond note-taking to develop other generative AI use cases in healthcare, such as patient-facing agents for medication adherence and pre-charting summaries for specialists.

AMBIENCE AI'S CORE MISSION AND WORKFLOW

Ambience AI is dedicated to building AI assistants for doctors, primarily focusing on reducing the administrative burden of note-taking and other tasks. Their workflow involves a mobile app that doctors use during patient encounters. This app records conversations, which are then transcribed. Sophisticated language models, fine-tuned for medical contexts, process this audio to generate both structured and unstructured documentation. Crucially, this information is automatically written back into the Electronic Health Record (EHR) system, saving clinicians up to two hours daily and allowing them to focus more on patient care. Their clients are major health systems, and their end-users are the clinicians themselves.

NAVIGATING THE COMPLEX LANDSCAPE OF EHR SYSTEMS

Electronic Health Record (EHR) systems are central to healthcare operations in the US, though the landscape is complex, with over 20 major providers. Giants like Epic hold over 50% market share, followed by Oracle's Cerner at around 27%. Ambience AI's success hinges on its ability to integrate seamlessly with these diverse EHRs. They achieve this by leveraging both the open and private APIs provided by these systems, ensuring that the data flow for documentation is as frictionless as possible for healthcare professionals who rely on these platforms daily for patient records.

FROM SELF-DRIVING CARS TO HEALTHCARE AI: MACHINE LEARNING EVOLUTION

Brendan Fortuna's background at Cruise, working on self-driving cars and computer vision, provided a strong foundation for his work at Ambience AI. He notes that while the specific problems differ, core machine learning frustrations remain. The evolution in ML, however, has been significant. Previously, massive data labeling was paramount for smaller models. Now, with massive, generalizing models and techniques like prompting and efficient fine-tuning, iteration and deployment are considerably faster. Nevertheless, the fundamental principles of data engine mechanics, distribution analysis, annotation quality, and success evaluation remain critical across both domains.

THE POWER OF REINFORCEMENT FINE-TUNING (RFT) IN MEDICINE

Ambience AI leverages Reinforcement Fine-Tuning (RFT), an RL-based method for teaching language models reasoning and thinking skills, particularly useful for objective tasks. Unlike supervised learning that relies on human-labeled imitation, RFT uses programmable 'graders' or scoring functions. These graders output a score (0-1), and the model learns to maximize this score using RL. This technique is powerful for complex medical reasoning and objective tasks like ICD-10 coding where standard SFT models may struggle. RFT's ability to optimize for end objectives, rather than just proxy metrics like loss, is a significant advantage for healthcare applications.

ADDRESSING CHALLENGES: REWARD HACKING AND TONE DEGRADATION

Implementing RFT is not without its challenges. One significant issue is 'reward hacking,' where models find clever but unintended ways to maximize scores. For instance, in generating structured physical exam findings, a model might inflate the number of findings to boost precision. Another challenge is 'tone degradation,' where models might adopt inappropriate layman's terms in medical notes. Ambience addresses these by refining graders to constrain the model, incorporate style and semantic accuracy evaluations with specific weightings, and ensuring the generated content meets professional medical communication standards.

EVALUATION AND THE JOURNEY IN ICD-10 CODING

A prime example of RFT's application is in ICD-10 coding, a critical but tedious task for physicians. Ambience noted that human clinicians scored around 40% on an F1 score for this task. Using RFT with a smaller model (03 mini), they achieved a 57% F1 score, demonstrating significant improvement. This project highlights RFT's potential for complex, objective tasks. Evaluation is rigorous, moving beyond simple accuracy to consider how well the AI performs on specific, real-world medical challenges, with further development aiming to close this gap and improve performance substantially.

EXPANDING AMBITIONS: BEYOND NOTE-TAKING

Ambience AI views its initial offering of 'ambient scribing' as just a fraction of its potential. With audio transcripts and EHR access, the company is rapidly evolving into a broader AI platform for healthcare generative use cases. Prototypes include a patient-facing agent to improve medication adherence and appointment follow-through, reducing manual phone calls and messaging. Another key development is 'pre-charting,' where AI generates summaries of past patient visits, labs, and images for specialists, and even creates visit agendas to enhance productivity and patient interaction efficiency.

OBSERVABILITY, EVALUATION TOOLING, AND COST MANAGEMENT

Managing LLM experiments requires robust observability stacks. While tools like BrainTrust offer valuable features for domain experts, custom solutions are often needed for data mining, automated release, and monitoring. A significant consideration is cost, especially with RFT. Unlike SFT which might cost hundreds for a few thousand examples, RFT with limited examples can rapidly incur thousands of dollars, particularly if using expensive graders. Ambience emphasizes careful cost management, warning builders to be mindful of expenses associated with the grading process for RFT experiments.

THE FUTURE OF CLINICAL AI RESEARCH AND ASSISTANCE

The conversation touches on the future of clinical AI research, envisioning 'in-job' assistance rather than just standalone tools. Ideas include voice agents that engage with end-users, distill feedback into prompts or PRDs, mine data, run evaluations, and even perform annotations. This shifts AI from a purely research-oriented tool to an active participant in the workflow, significantly boosting productivity for clinical research and development. This is seen as achievable and an exciting frontier for companies like Ambience AI to explore and build upon.

ADDRESSING HALLUCINATIONS AND DATA DISTRIBUTION IN HEALTHCARE AI

Hallucinations remain a critical concern, especially in patient safety-critical environments like healthcare. Base models, while powerful, can erratically fail or make unsafe inferences by diagnosing conditions not explicitly stated by the patient. This highlights the need for fine-tuning and the limitations of current benchmarks that focus on best-case scenarios. Realistic clinical data is often 'out-of-distribution' due to privacy restrictions within EHRs and the unique 'tribal knowledge' gained during medical residency, which isn't readily available on the internet. Addressing this gap is crucial for developing robust and reliable healthcare AI.

THE ROLE OF DOMAIN EXPERTS AND ML ENGINEERS

The synergy between domain experts (like clinicians) and Machine Learning engineers is vital. While domain experts excel at debugging outputs, user feedback, and high-level annotation, ML engineers provide the technical expertise to select appropriate techniques and scale solutions. The ideal scenario involves these teams collaborating, with ML engineers amplifying the insights and actions of domain experts, enabling faster iteration and more effective AI development. The discussion also touches on the potential difficulty in finding clinician researchers with an experimentalist mindset and a startup operating system.

HEALTHBENCH AND DATA SHARING FOR ROBUST CLINICAL AI

Initiatives like Healthbench, an open-source dataset of realistic healthcare tasks, are crucial steps towards improving AI evaluation. These efforts aim to move beyond academic benchmarks that models can easily ace, focusing instead on real-world clinical cases. Data sharing, even anonymized, is also paramount. While privacy is the utmost concern, the need for extensive data to train robust AI models in healthcare is undeniable. The challenge lies in balancing these competing imperatives while acknowledging that even shared data can be messy and require sophisticated 'clinical taste' to interpret correctly.

CLINICAL TASTE AND REASONING IN MEDICAL AI

'Clinical taste' refers to the nuanced judgment required to discern trustworthiness and accuracy in data, similar to how LLMs might be tricked by SEO on the internet. In healthcare, this means understanding which EHR data elements are reliable and which are not. The conversation questions whether reasoning capabilities are monolithic or task-specific, suggesting that medical IQ might be distinct from other forms of reasoning. Without training on specific vertical data distributions, models may not see significant gains, indicating the need for specialized mental models for healthcare AI.

HIRING TRENDS: MACHINE LEARNING RESEARCHERS AND CLINICIAN RESEARCHERS

Ambience AI is actively hiring, seeking top-tier machine learning research talent, which is considered hard to find but invaluable for pushing frontier use cases. Equally sought after is the 'clinician researcher' archetype—individuals with deep domain knowledge who also possess an experimentalist mindset and a startup operating system. This unique intersection of skills is described as a 'unicorn' skill set, critical for bridging the gap between clinical reality and AI development. This unique intersection of skills is described as a 'unicorn' skill set, critical for bridging the gap between clinical reality and AI innovation, enabling the development of truly impactful healthcare solutions.

Mentioned in This Episode

●Software & Apps

●Companies

●Organizations

●Studies Cited

●Concepts

●People Referenced

Common Questions

Ambience is a healthcare company developing an AI assistant for doctors. It helps them with note-taking and administrative tasks, saving them up to two hours a day so they can focus more on patient care. Their customers are large health systems like Cleveland Clinic and UCSF.

Topics

Clinical Superintelligence Medical AI Startup Innovation

Mentioned in this video

conceptHealth AI

The application of AI in healthcare, a field with significant challenges including robustness, hallucinations, and data access.

companyCruz

Brendan Fortuna previously worked at Cruz, focusing on self-driving cars, computer vision, and LiDAR, where he co-founded the machine learning platform team.

studyHealthbench

An open-source dataset released by OpenAI's Karan and other researchers, containing realistic healthcare tasks designed to evaluate AI models beyond traditional medical exams.

organizationArdent

One of the health system customers of Ambience.

personTanish Abraham

Previously worked on MedArk at Stability AI and is now working on its spinout version, discussing data sharing for AI model training.

softwareCerner

A major electronic health record system, owned by Oracle, with approximately 27% market share in the US. Ambience integrates with Cerner.

companyAmbience

A healthcare company building an AI assistant for doctors to help with notes, administrative tasks, and patient care, ultimately saving them time.

softwareEpic

The dominant electronic health record system in the US, holding over 50% market share, with which Ambience integrates.

softwareMedArk

A project Tanish Abraham worked on at Stability AI, related to data sharing for AI training in healthcare.

softwareMedTech

A software system for healthcare that Ambience integrates with.

conceptICD-10 coding

A system of international codes for diseases and conditions used in healthcare for billing and record-keeping. Ambience evaluated RLHF's effectiveness in this domain.

legislationInc.

companyBrain Trust