Why is maintaining user agency important in AI interactions?

Maintaining user agency is crucial, especially in personal advisory roles, as it ensures users remain active participants in their own decision-making processes. Overly prescriptive AI can deny users this agency, potentially leading to unhelpful outcomes and a feeling of disempowerment.

What is 'qualitative context' in the context of AI coaching?

Qualitative context refers to personal factors like a user's goals, values, and motivations, which are central to understanding their reasons for change. Eliciting this context through natural language is key for AI to provide non-prescriptive, personalized support.

How does GPT Coach promote non-prescriptive support?

GPT Coach uses prompt chains based on motivational interviewing strategies to guide its communication. It avoids unsolicited advice, asks open-ended questions to elicit user motivations, and integrates quantitative data with qualitative context to offer personalized, affirming support.

What was the impact of the Bloom application on user experience?

The Bloom application, integrating an AI coach with behavior change interactions, led to more positive participant mindsets and increased engagement. Users felt a greater sense of agency and control, attributing their progress to the system's personalized and non-prescriptive approach.

How can AI learn to navigate uncertainty in user interactions?

AI can learn to navigate uncertainty by using reinforcement learning, specifically Bayesian adaptive MDPs. This involves eliciting beliefs about the user's latent state, ranking potential strategies based on expected returns, and then generating an utterance that is context-aware and strategically chosen.

What are the key design principles for promoting agency in human-AI interaction?

The key principles are non-prescriptiveness, eliciting qualitative context, and navigating uncertainty. These principles help design AI systems that augment users, empower them to make their own decisions, and foster a sense of control over their interactions and outcomes.

Key Moments

Stanford CS547 HCI Seminar | Spring 2026 | Promoting Agency in Human-AI Interaction

Stanford Online

Education5 min read45 min video

Jul 22, 2026|773 views|34

Stanford Stanford Online LLM Human-AI Interaction

Save to Pod

Want to know something specific about what's covered?

We've already dissected every moment. Ask and we will deliver (with timestamps).

Key Moments

TL;DR

AI coaches can be personalized to promote user agency by eliciting qualitative context and navigating uncertainty, but they must avoid the pitfalls of being overly prescriptive or verbose to truly empower users.

Key Insights

Non-work-related messages on ChatGPT increased from 53% in 2024 to over 70% in 2025, indicating a significant shift towards using LLMs for personal advice.

Health and well-being advice is the most common topic users turn to LLMs for, making it an ideal test bed for studying augmentative human-AI interaction.

In user studies, GPT Coach used non-prescriptive communication strategies over 93% of the time, significantly outperforming a vanilla GPT-4 agent.

Participants in the Bloom study reported that the LLM coach helped them feel in control and have agency over their choices, fostering more positive mindset effects.

The novel zero-shot Baze adaptive planning algorithm for LLMs demonstrated substantial policy improvement by eliciting user beliefs and ranking high-level strategies.

When using the zero-shot Baze adaptive planning algorithm, a smaller GPT-4 mini model recovered performance levels similar to a frontier model like GPT-5 in specific environments.

The shift from AI assistants to AI advisors

The increasing use of Large Language Models (LLMs) like ChatGPT for personal advice on health, careers, and relationships signals a shift from their initial design as task-automating assistants to that of AI advisors. OpenAI reported that non-work-related messages on ChatGPT grew from 53% in 2024 to over 70% in 2025. This evolution requires a new approach to designing and evaluating AI systems. While assistants perform tasks on behalf of users, advisors augment users by influencing their beliefs and understanding to enable better actions. This mirrors the historical vision of human-computer interaction, exemplified by Douglas Engelbart's work in the 1960s, which moved computers beyond calculators to tools for personal augmentation. The challenge is to design these advisor interactions responsibly, ensuring they preserve user agency and mitigate harm.

Eliciting qualitative context over prescriptive advice

Many health experts and coaches advocate for a non-prescriptive, facilitative approach where clients drive their own behavior change. They emphasize refraining from unsolicited advice, likening their role to a passenger providing direction rather than a driver dictating actions. This aligns with behavior change theory, which suggests promoting autonomy and intrinsic motivation is more effective than simply telling people what to do. Standard LLMs, however, are prone to offering prescriptive advice, as seen in examples where ChatGPT suggests actions without understanding the user's context or struggles. This not only can be unhelpful but also undermines user agency by denying them the opportunity to actively participate in their change process. The solution lies in eliciting qualitative context—users' goals, values, and motivations—which can be effectively captured in natural language by LLMs. This qualitative context is crucial for providing effective support aligned with the client's reasons for change, a concept explored through the development of GPT Coach.

GPT Coach: An LLM-based physical activity coach

To address the limitations of prescriptive LLMs, GPT Coach was developed, an LLM-based physical activity coaching chatbot. It integrates principles from motivational interviewing and utilizes data from wearable devices. The system employs three prompt chains: a dialogue state chain for topic management, a motivational interviewing chain for communication style, and a tool use chain for data visualization. In a lab study with 16 participants, GPT Coach demonstrated strong adherence to non-prescriptive strategies, used consistent or neutral conversational strategies over 93% of the time, and significantly outperformed a vanilla GPT-4 agent in employing supportive communication. Participants reported overwhelmingly positive experiences, appreciating the actionable, personalized approach, and feeling comfortable and supported. Qualitative feedback highlighted the importance of understanding user context (e.g., work, schedule, values) for effective support, and the non-threatening nature of the AI's approach compared to some human providers.

Bloom: Augmenting behavior change interactions with LLMs

Building upon GPT Coach, the Bloom application further integrates LLM-augmented behavior change interactions into an iOS app. Bloom features a 'Bibo' bee avatar that guides users through a weekly goal-setting conversation and personalizes various behavior change interventions, including ambient displays, activity tracking summaries, and push notifications. The system combines quantitative data from wearables with qualitative context elicited by the coaching agent to personalize these interactions. A 4-week randomized field study with 54 participants compared the full Bloom app to a control condition with a simplified GUI and templated notifications. Treatment participants showed more specific and nuanced changes in their mindset towards physical activity, attributing these shifts to increased agency and control. They also spent over five times longer in the app compared to the control group, exhibiting increased engagement across all features, often speaking about Bibo in relational terms.

Navigating uncertainty with reinforcement learning

A key limitation of earlier systems like GPT Coach was the hard-coding of non-prescriptive behaviors into prompts. To overcome this, a technical approach using reinforcement learning (RL) was developed to enable LLM agents to deliberately navigate uncertainty about user goals and states. This involves treating conversations as Markov decision processes (MDPs) where LLMs learn policies to maximize rewards. By considering latent uncertainty over user attributes (e.g., goals, stage of change), the framework extends to Baze adaptive MDPs. The algorithm, termed 'zero-shot Baze adaptive planning,' allows LLMs to elicit beliefs about users, rank high-level strategies (e.g., ask a question, make a recommendation), and then generate utterances accordingly, all without prior training data for specific users. This approach aims to create more adaptive and less rigid AI interactions.

Algorithmic validation in simulated environments

The zero-shot Baze adaptive planning algorithm was evaluated across three simulated environments: exercise recommendation, medical diagnosis (MedicQ), and negotiation (Deal or No Deal). Results showed that LLMs can be prompted to elicit beliefs that converge towards a user's true attributes over time. Crucially, these elicited beliefs improved the LLM's ability to rank different actions and strategies. When integrated into the full pipeline, the algorithm led to substantial policy improvements across models and environments. For instance, a smaller model like GPT-4 mini recovered performance levels comparable to a strong frontier model like GPT-5 in certain settings, demonstrating the effectiveness of this uncertainty-navigating approach for promoting agency through strategic interaction.

Future directions and broader implications

The research presented highlights three core principles for promoting agency: non-prescriptiveness, qualitative context, and navigating uncertainty. While initially focused on health behavior change, these principles are broadly applicable to designing augmentative AI systems for various domains. Future work includes longitudinal clinical RCTs for systems like Bloom in cardiac rehabilitation and expanding coaching applications to areas like sleep and nutrition. There is also significant potential to apply these principles to tackle societal issues such as skill-skilling, disempowerment, and over-reliance on AI beyond health. Technical advancements in representing user context and quantifying uncertainty are seen as critical for building more capable and non-prescriptive AI.

Mentioned in This Episode

●Software & Apps

●Companies

Promoting Agency in Human-AI Interaction: Key Principles

Practical takeaways from this episode

Do This

Design AI systems to augment, not just automate.

Focus on eliciting qualitative context (goals, values, motivations).

Employ non-prescriptive support strategies.

Navigate user uncertainty to personalize interactions.

Use LLMs in a facilitative, rather than directive, manner.

Incorporate user feedback throughout the design cycle.

Avoid This

Avoid unsolicited advice.

Do not impose solutions or assumptions on the user.

Do not design AI solely as task-automating assistants for personal topics.

Avoid rigid or overly prescriptive AI responses.

Do not neglect qualitative factors in favor of purely quantitative data.

Common Questions

AI assistants primarily automate tasks based on direct commands, like writing code. AI advisors, on the other hand, engage users on personal topics such as health or relationships, aiming to augment the user's understanding and decision-making rather than executing tasks for them.

Topics

Reinforcement Learning Health & Longevity Mindset & Self-Improvement AI & Machine Learning Technology & Innovation Behavior Change Human-AI Interaction Health Technology User Agency AI Coaching Non-prescriptive AI Qualitative Context

Mentioned in this video

Software & Apps

ChatGPT

An LLM developed by OpenAI, noted for its increasing use in personal advice and its potential to be prescriptive.

Companies

Microsoft

A company mentioned in the context of reports on user engagement with LLMs.

Ask anything from this episode.

Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.

Get Started Free