Key Moments
Stanford CS193T Thinking with AI | Autumn 2026 | Managing Sources
Stanford OnlineKey Moments
AI models can be tricked into ignoring safety instructions or executing malicious commands through "prompt injection," a vulnerability that arises because AI systems struggle to distinguish between user prompts and system instructions. This means that even uploaded documents or web searches can be weaponized to hijack AI behavior, posing significant security risks, especially with emerging agentic AI.
Key Insights
Users can customize AI behavior through settings, system prompts, and preferences, such as adjusting tone, formatting, or direct commands like "act less flattering" or "speak like a pirate."
AI models remember past conversations and can inject summaries of this information into new contexts, which users can often view and manage, including migrating these memories between different AI services like ChatGPT and Gemini.
Retrieval-Augmented Generation (RAG) allows AI models to access external knowledge bases, significantly expanding context beyond the model's fixed window, but the quality of information in the RAG database disproportionately affects outputs.
AI models can exhibit "adaptive chameleon" or "stubborn sloth" behaviors when faced with conflicting information between their training data and a RAG system, leading to either over-compliance with flawed data or rejection of correct user-provided context.
While AI models are designed to prioritize RAG data, this is not a guarantee, and prompt injection attacks can occur when malicious instructions embedded in web documents or uploaded files are misinterpreted by the AI as user commands.
Skills and complex prompts can be saved as reusable 'skills' or 'gems' within AI platforms like Claude and Gemini, allowing users to store and share custom AI behaviors and workflows, though OpenAI is deprecating a similar feature.
Personalizing AI interactions through customizable settings and prompts
Modern AI tools like ChatGPT, Claude, and Gemini offer extensive customization options to tailor user experiences beyond default settings. This includes adjusting conversational tone, preferred response length, frequency of emojis, and the use of headings or lists. For instance, ChatGPT's customization settings allow users to select preferences like 'less' for emoji usage or to input specific system prompts to guide the AI's persona, such as requesting it to be less flattering and more challenging. Claude's system prompt also dictates its default behavior, which can be overridden or modified through user-defined instructions. This enables users to inject personal preferences, like requesting more frequent use of bold text and bullet points, or even whimsical requests like responding in a pirate style every third turn. These persistent settings and preferences eliminate the need for repetitive instructions in each new interaction, creating a more personalized and efficient user experience.
AI memory and cross-session context persistence
AI models automatically store and recall information revealed in past conversations, such as personal details like location or occupation. This information can be injected into the context window of subsequent interactions, influencing AI responses even when the current topic is unrelated. For example, if an AI learns about your job in one chat, it might adjust its tone or recommendations in a future, unrelated conversation. Users can often access and manage these 'memories.' ChatGPT allows users to enable or disable memory collection and view stored memories. Recognizing the desire to migrate personalized AI states, Gemini offers a feature to import memories from ChatGPT by generating a specific prompt that instructs ChatGPT to list all its remembered facts about the user. This exported data can then be imported into Gemini, facilitating a seamless transition between AI platforms for users who heavily rely on personalized AI states.
Expanding context with Retrieval-Augmented Generation (RAG)
When the amount of information exceeds the capacity or practical usability of the context window, Retrieval-Augmented Generation (RAG) provides a solution. RAG involves an external data store that can hold significantly more information than what fits into the AI's immediate context window. When a user makes a request, the RAG system performs a search within this data store to find relevant documents. Only the content from these retrieved documents is then placed into the context window. This selective retrieval allows the AI to access a vast amount of custom information without overwhelming the context window. However, the quality of the sources within the RAG database is paramount, as AI systems are designed to heavily prioritize this information over their general training data. Poorly curated RAG data can disproportionately and negatively impact the AI's output, leading to incorrect or biased responses. This emphasizes the need for careful selection and management of RAG data sources.
Navigating conflicting knowledge: 'Adaptive Chameleon' vs. 'Stubborn Sloth'
Research, such as the paper 'Adaptive Chameleon or Stubborn Sloth,' highlights how AI models behave when faced with conflicting information between their training data and a RAG system. The 'adaptive chameleon' behavior occurs when the AI excessively trusts and adopts incorrect information from the RAG system, even when its general training data suggests otherwise. This is driven by a tendency to please the user or adapt to provided context. Conversely, the 'stubborn sloth' behavior describes situations where the AI rigidly adheres to its training data, refusing to acknowledge or incorporate correct information from the RAG system. Both patterns represent failure modes. The 'stubborn sloth' can be beneficial if a user unintentionally inputs false information into their RAG, preventing the AI from propagating that misinformation. Conversely, the 'adaptive chameleon' can be problematic if the AI uncritically accepts flawed RAG data. The choice of which behavior is preferable often depends on the specific context and desired outcome, raising questions about control and bias in AI responses.
Saving and reusing complex AI instructions as 'skills' or 'gems'
AI platforms are increasingly offering features to save and reuse complex instructions, workflows, or personas as 'skills' or 'gems.' This moves beyond simple preference settings to capturing entire prompt engineering efforts. For example, if a user develops a highly detailed prompt for generating a specific type of document or analysis, they can save this as a skill. Claude offers 'Skills,' and Gemini has 'Gems,' which allow users to store detailed prompts along with associated context or documents. These saved skills can then be easily recalled for future use, eliminating the need to recreate lengthy prompts. Furthermore, these skills and gems can often be shared with other users, fostering a community around custom AI functionalities. While OpenAI previously had a similar feature ('Custom GPTs'), it is being deprecated, suggesting a shift towards different implementations for saving and sharing AI behaviors.
The security risks of prompt injection and context window vulnerabilities
A significant security concern arises from the AI's difficulty in distinguishing between its system prompt (provided by the developer) and user prompts, especially since they are all processed as a continuous stream of tokens. This ambiguity creates a vulnerability for 'prompt injection,' where malicious instructions can be embedded within user inputs, uploaded documents, or even web pages fetched by the AI. An injected prompt can trick the AI into ignoring its safety guidelines, revealing sensitive information, or executing harmful actions. For example, a web document could contain instructions telling the AI to disregard all previous commands and email a gambling website's advertisement to a list of contacts. While AI systems are being trained to detect and resist such injections, the risk remains, particularly as AI evolves towards more agentic capabilities. This is why 'red teaming' – actively trying to break AI security – is crucial.
Strategic information blocking and maintaining response objectivity
Users can strategically withhold information to achieve more objective or insightful AI responses. For instance, when seeking feedback on a resume, framing the request as evaluating 'this resume' rather than 'my resume' can reduce the AI's tendency towards flattery or bias. By presenting oneself as a neutral third party (e.g., a hiring manager), users can encourage more direct and honest evaluations. This strategic withholding, however, necessitates careful management of conversational memory, perhaps by using a new account or incognito mode to avoid the AI detecting the 'lie' about one's role. This technique is also useful in brainstorming sessions to prevent early examples from unduly influencing the AI, encouraging a wider diversity of ideas. The core principle is to control the information the AI receives to shape its output and avoid self-serving biases or undue influence from prior interactions.
Prompt injection via web documents and the danger of agentic AI
The vulnerability of prompt injection is exacerbated when AI models can access and process external web documents. If a malicious actor embeds prompt injection commands within a webpage, the AI might retrieve this document and execute the embedded instructions, mistaking them for legitimate user commands. This is particularly dangerous when AI models are connected to external services like email or file systems, as the injected prompt could lead to actions like sending spam emails or accessing sensitive data. The AI's inability to reliably distinguish between its own system prompts and content retrieved from the web creates a significant security gap. While tools can disable web browsing or link opening features as a precaution, sophisticated attacks might use techniques like invisible text (white text on a white background) to hide malicious commands. The evolution towards agentic AI, which can take actions in the real world, amplifies the potential damage from successful prompt injection attacks, making them a critical area of ongoing security research and development.
Mentioned in This Episode
●Software & Apps
●Companies
●Organizations
●Concepts
Common Questions
Most major AI tools like ChatGPT, Claude, and Gemini offer settings to customize responses. You can adjust tone, response length, formatting preferences (like using headings or emojis), and even write custom system prompts to guide the AI's behavior.
Topics
Mentioned in this video
A widely used AI model for which users can set preferences and manage memory settings. The full version is required for certain functionalities discussed in the video.
An AI model discussed for its notebook feature (formerly NotebookLM) and its capabilities in memory import and managing custom skills (Gems).
The previous name for Gemini notebook, a product for managing AI memory and context.
An AI model discussed in relation to its system prompts, which can influence its output style and adherence to formatting guidelines. Users can also customize its behavior through settings.
An AI model that has become good at identifying prompt injection attempts within skills.
Two behavioral patterns observed in large language models when knowledge conflicts: the 'Adaptive Chameleon' overly trusts new information, while the 'Stubborn Sloth' rigidly adheres to its training data.
A more advanced form of AI that can take actions in the real world, increasing the risks associated with prompt injection.
A game or media referenced in the context of Gemini Gems, used for entertainment purposes.
A game or media referenced in the context of Gemini Gems, used for entertainment purposes.
A tabletop role-playing game that can be simulated using Gemini Gems, showcasing the entertainment potential of AI skills.
The company behind ChatGPT, mentioned in the context of its AI models and potential use of user conversation history for training future models.
A social media platform where a user shared over 30 RPG Gemini Gems, highlighting the creative use of AI skills.
The company that developed Claude, mentioned in relation to its system prompts and the user's ability to override them.
More from Stanford Online
View all 173 summaries
77 minStanford CS193T Thinking with AI | Autumn 2026 | Core Prompting
78 minLLMs for Agent Builders: Context Engineering | Stanford CS329Z Lecture 2 with Michael Ryan
70 minStanford CS329Z Introduction — What Are Agentic Systems?
73 minStanford CS193T Thinking with AI | Autumn 2026 | Introduction to AI
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free