Key Moments
AI Costs: Tokens, Energy, Model Choice & API Keys | Stanford CS193T Lecture 7 with Cynthia Bailey
Key Moments
AI data centers are facing significant voter backlash due to resource consumption, with 76% opposing them locally and 40 out of 41 ballot measures failing in recent elections, impacting political campaigns and even prompting Elon Musk to suggest space-based data centers.
Key Insights
Caching can reduce input token costs by up to 90% by reusing precomputed attention mechanism calculations.
Image generation can be 28-40x the energy cost of text generation, while video generation is 1,000-3,000x, and reasoning can be 30-700x.
Even if not used, enabling skills for an agent incurs an overhead cost as their prompt text is injected into the context window for every prompt.
In a recent election, 40 out of 41 ballot measures related to AI data centers were decided against them, indicating strong public opposition.
76% of voters in a CBS exit poll stated they would oppose a data center in their own community.
Over $380 million was spent on TV ads where data centers were a central topic, highlighting their political salience.
Analyzing AI usage reports for cost and efficiency
Understanding the costs associated with using AI models is crucial, especially as projects scale. A sample usage report from Claude Code breaks down costs into total dollar amount, duration (API and wall clock time), and token usage (input and output). API duration encompasses the time spent waiting for a response, including computation, network transit, and potential data center delays. Wall clock time is the total elapsed time since the session began. Input tokens refer to everything the model processes, including the prompt, system instructions, and tool configurations. Output tokens are generated by the LLM, including internal model reasoning. Paying close attention to these metrics helps in optimizing AI usage and managing expenses.
The power of caching to reduce token consumption
Caching is a critical mechanism for saving costs related to input tokens. The attention mechanism, a core component of transformer models, requires significant computation. When parts of the context window, such as system prompts or settings, remain constant across multiple prompts within a session, the precomputed results of the attention mechanism can be saved. Accessing these cached results provides a significant discount, potentially up to 90%, compared to processing raw input tokens. This is because the expensive linear algebra and matrix operations for calculating attention are performed only once and then reused. Effectively leveraging caching can substantially lower the overall cost of running AI models.
Skill overhead and strategic model selection
When building agents, enabling skills, while useful for reusable tasks, comes with an inherent cost. The text of these reusable prompts is injected into the agent's context window for every single prompt, contributing to input token usage. Even if a skill is not actively used during a session, its inclusion in the context window incurs an overhead cost. Therefore, it is important to declutter and only enable necessary skills to avoid unnecessary expenses. Furthermore, advice from usage reports often suggests using cheaper, smaller models for general-purpose sub-agents, like assigning Claude Haiku instead of the more expensive Claude Opus, to manage costs effectively. This highlights the importance of thoughtful model selection based on task requirements and budget constraints.
Environmental impact of AI: Water and electricity consumption
Beyond token and dollar costs, AI usage has a significant real-world impact on resources like electricity and water. While precise public data is scarce, researchers estimate these impacts. For a median text prompt in 2025, Google reported usage of 0.26 milliliters of water and 0.24 watts of electricity. However, these figures can escalate dramatically with different task types. Image generation is estimated to be 28-40 times more energy-intensive than text generation, video generation is 1,000-3,000 times more, and reasoning or thinking modes can be 30 times more, with some models reaching up to 700 times. Long context windows also increase costs; for instance, 10,000 input tokens incur a significant cost, and 100,000 tokens even more so. These figures underscore the environmental considerations when choosing and using AI models.
Latency and its critical role in multi-agent workflows
Model size directly correlates with both capabilities and latency, the time it takes for a model to process a request. While larger models might be more capable, they also introduce longer delays. This latency is a sum of individual prompt processing times and becomes particularly critical in multi-agent workflows, where a small delay at each step can multiply significantly. For example, an AI tutoring system with input and output guardrail agents can iterate multiple times, making cumulative latency a major concern for user experience. Choosing the right model for each agent, balancing capability with speed and cost, is therefore essential for efficient and responsive AI systems.
Strategic model selection for different agent roles
When designing multi-agent systems, it's often beneficial to use different models for different roles, optimizing for cost, capability, and latency. For instance, an input guardrail agent designed to simply check for blatant requests for answers might utilize a smaller, cheaper model. In contrast, an output guardrail agent responsible for nuanced analysis of context and ensuring accurate citations might require a more sophisticated and powerful model. The choice depends on the complexity and objectivity of the task. A simple, objective task like checking for consecutive lines of code can be handled by the cheapest models, whereas a task requiring nuanced interpretation and comparison against policies demands a more capable model, even if both are labeled as 'guardrail agents'.
API keys: enabling integration and managing billing
API keys function as login credentials for programmatic access to AI models, allowing integration into custom workflows outside of vendor-specific chat interfaces. They enable billing to be tracked and charged to a specific account, similar to how a human logs in. This is particularly useful for developers building applications. An API key can be associated with a personal account, allowing the developer to absorb costs, or individual users can be required to provide their own API keys, distributing the cost. This distributed cost-sharing model allows for creative development of AI tools without the creator bearing the entire financial burden. However, API keys must be treated with extreme caution, akin to passwords, to prevent unauthorized use and associated charges.
The critical security risk of exposing API keys
Exposing API keys in code, especially in publicly accessible web applications, presents a severe security risk. If an API key is hardcoded into the source code of a website, anyone can view it by inspecting the page's source. This allows malicious actors to use the key to access AI services, incurring costs billed to the owner of the key. A common pitfall is embedding API keys directly into client-side code for web apps. Secure practices involve storing API keys in environment variables or secure configuration files that are not exposed to the public. Proper code review and security checks are essential to prevent such leaks, which can lead to unexpected expenses and misuse of AI resources.
The growing political and social opposition to AI data centers
The expansion of AI data centers is becoming a significant political issue, driven by concerns over resource consumption, particularly water and electricity, and their environmental impact. In recent elections, AI data centers were a high-profile issue, with 40 out of 41 ballot measures related to them decided against their construction. A CNBC exit poll indicated that 50% of voters considered AI data centers a crucial or major factor in their voting decisions, and a CBS poll revealed that 76% of voters would oppose a data center in their own community. The substantial spending on TV ads ($380 million) focusing on data centers further illustrates the public's strong emotions regarding this issue. This backlash is influencing political races and has even led prominent figures like Elon Musk to suggest relocating data centers to space, signaling a growing global resistance to their on-Earth deployment.
Mentioned in This Episode
●Software & Apps
●Companies
●Organizations
●Books
●People Referenced
AI Usage and Cost Management Best Practices
Practical takeaways from this episode
Do This
Avoid This
AI Task Energy Consumption Multipliers (Estimated)
Data extracted from this episode
| Task Type | Multiplier (x text generation) | Source/Notes |
|---|---|---|
| Image Generation | 28-40x | Estimated, based on study in MIT Technology Review |
| Video Generation | 1,000-3,000x | Estimated, based on study in MIT Technology Review |
| Reasoning/Thinking Mode | 30x (average, up to 150x or 700x for some models) | Estimated, based on study in MIT Technology Review |
AI Model Selection Considerations for Agents
Data extracted from this episode
| Agent Type | Example Considerations | Recommended Model Characteristics |
|---|---|---|
| Input Guardrail Agent (AI Tutor) | Checks for blatant asks for answers. | Likely requires a smaller, cheaper model due to objective, unambiguous task. |
| Response Generation Agent (AI Tutor) | Generates thoughtful, kind responses with citations and analogies. | May require a more capable model, but latency is a key concern in multi-step workflows. |
| Output Guardrail Agent (AI Tutor) | Checks if the response provides answers directly. | Needs to be objective and unambiguous (e.g., code line limits); smaller model suitable. |
| Company Policy Q&A Chatbot Output Guardrail | Checks generated response and citations for accuracy against company handbooks. | Requires a nuanced, capable model due to complex analysis and context matching; higher cost and latency may be acceptable. |
Common Questions
Input tokens are the data the AI model reads (your prompt, system instructions, etc.), while output tokens are what the AI generates. Both contribute to the cost of using AI models, as they consume computational resources and processing time.
Topics
Mentioned in this video
A tool mentioned alongside Claude Code, likely related to AI and code generation or analysis.
A platform or tool used for code-related tasks, likely involving AI models, where usage reports can be accessed.
A specific, more expensive AI model within the Claude family, often used for complex tasks.
A cheaper and likely less powerful AI model within the Claude family, suggested for simpler tasks to save costs.
A resource management tool that helps users choose and manage different AI models for various tasks.
The AI tutor tool used in a Stanford computer science course (CS106B), featuring a multi-agent system with guardrails to prevent students from simply getting answers.
A tool that assists in selecting and managing different AI models for the same task, potentially offering cost or performance comparisons.
A generation of OpenAI models mentioned in the context of agent selection, with different sizes like Soul, Luna, and Astra.
Google's AI model, for which self-reported energy and water consumption data for a median text prompt was presented.
An older generation of OpenAI models, mentioned for comparison with GPT-6 models regarding capabilities, size, and cost.
The Governor of Texas, whose challenger highlighted opposition to AI data centers as a key issue during their election campaign.
Challenger to Texas Governor Greg Abbott, who used the issue of AI data centers as a central campaign point.
CEO of SpaceX and Tesla, who commented on building data centers in space after noting local opposition to terrestrial data centers, particularly in Texas.
A celebrity investor and Shark Tank personality involved in a controversial proposal to build a large AI data center in Utah, facing local opposition due to land use and water consumption.
More from Stanford Online
View all 186 summaries
74 minTerminal-Bench: Evaluating AI Agents | Stanford CS329Z Guest Lecture with Alex Shaw & Ryan Marten
81 minOptimizing AI Agents: Prompts vs. Weights (GEPA, GRPO) | Stanford CS329Z Lecture 9 with Michael Ryan
59 minHow AI Agents Work: Connectors, MCP & Agent Safety | Stanford CS193T Lecture 5 with Cynthia Bailey
70 minVibe Coding, AI Risk & AI Evaluation | Stanford CS193T Lecture 6 with Cynthia Bailey
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free