Key Moments

AI Costs: Tokens, Energy, Model Choice & API Keys | Stanford CS193T Lecture 7 with Cynthia Bailey

Stanford OnlineStanford Online
Education6 min read49 min video
Nov 19, 2026|6,144 views|74
Save to Pod
TL;DR

AI data centers are facing significant voter backlash due to resource consumption, with 76% opposing them locally and 40 out of 41 ballot measures failing in recent elections, impacting political campaigns and even prompting Elon Musk to suggest space-based data centers.

Key Insights

1

Caching can reduce input token costs by up to 90% by reusing precomputed attention mechanism calculations.

2

Image generation can be 28-40x the energy cost of text generation, while video generation is 1,000-3,000x, and reasoning can be 30-700x.

3

Even if not used, enabling skills for an agent incurs an overhead cost as their prompt text is injected into the context window for every prompt.

4

In a recent election, 40 out of 41 ballot measures related to AI data centers were decided against them, indicating strong public opposition.

5

76% of voters in a CBS exit poll stated they would oppose a data center in their own community.

6

Over $380 million was spent on TV ads where data centers were a central topic, highlighting their political salience.

Analyzing AI usage reports for cost and efficiency

Understanding the costs associated with using AI models is crucial, especially as projects scale. A sample usage report from Claude Code breaks down costs into total dollar amount, duration (API and wall clock time), and token usage (input and output). API duration encompasses the time spent waiting for a response, including computation, network transit, and potential data center delays. Wall clock time is the total elapsed time since the session began. Input tokens refer to everything the model processes, including the prompt, system instructions, and tool configurations. Output tokens are generated by the LLM, including internal model reasoning. Paying close attention to these metrics helps in optimizing AI usage and managing expenses.

The power of caching to reduce token consumption

Caching is a critical mechanism for saving costs related to input tokens. The attention mechanism, a core component of transformer models, requires significant computation. When parts of the context window, such as system prompts or settings, remain constant across multiple prompts within a session, the precomputed results of the attention mechanism can be saved. Accessing these cached results provides a significant discount, potentially up to 90%, compared to processing raw input tokens. This is because the expensive linear algebra and matrix operations for calculating attention are performed only once and then reused. Effectively leveraging caching can substantially lower the overall cost of running AI models.

Skill overhead and strategic model selection

When building agents, enabling skills, while useful for reusable tasks, comes with an inherent cost. The text of these reusable prompts is injected into the agent's context window for every single prompt, contributing to input token usage. Even if a skill is not actively used during a session, its inclusion in the context window incurs an overhead cost. Therefore, it is important to declutter and only enable necessary skills to avoid unnecessary expenses. Furthermore, advice from usage reports often suggests using cheaper, smaller models for general-purpose sub-agents, like assigning Claude Haiku instead of the more expensive Claude Opus, to manage costs effectively. This highlights the importance of thoughtful model selection based on task requirements and budget constraints.

Environmental impact of AI: Water and electricity consumption

Beyond token and dollar costs, AI usage has a significant real-world impact on resources like electricity and water. While precise public data is scarce, researchers estimate these impacts. For a median text prompt in 2025, Google reported usage of 0.26 milliliters of water and 0.24 watts of electricity. However, these figures can escalate dramatically with different task types. Image generation is estimated to be 28-40 times more energy-intensive than text generation, video generation is 1,000-3,000 times more, and reasoning or thinking modes can be 30 times more, with some models reaching up to 700 times. Long context windows also increase costs; for instance, 10,000 input tokens incur a significant cost, and 100,000 tokens even more so. These figures underscore the environmental considerations when choosing and using AI models.

Latency and its critical role in multi-agent workflows

Model size directly correlates with both capabilities and latency, the time it takes for a model to process a request. While larger models might be more capable, they also introduce longer delays. This latency is a sum of individual prompt processing times and becomes particularly critical in multi-agent workflows, where a small delay at each step can multiply significantly. For example, an AI tutoring system with input and output guardrail agents can iterate multiple times, making cumulative latency a major concern for user experience. Choosing the right model for each agent, balancing capability with speed and cost, is therefore essential for efficient and responsive AI systems.

Strategic model selection for different agent roles

When designing multi-agent systems, it's often beneficial to use different models for different roles, optimizing for cost, capability, and latency. For instance, an input guardrail agent designed to simply check for blatant requests for answers might utilize a smaller, cheaper model. In contrast, an output guardrail agent responsible for nuanced analysis of context and ensuring accurate citations might require a more sophisticated and powerful model. The choice depends on the complexity and objectivity of the task. A simple, objective task like checking for consecutive lines of code can be handled by the cheapest models, whereas a task requiring nuanced interpretation and comparison against policies demands a more capable model, even if both are labeled as 'guardrail agents'.

API keys: enabling integration and managing billing

API keys function as login credentials for programmatic access to AI models, allowing integration into custom workflows outside of vendor-specific chat interfaces. They enable billing to be tracked and charged to a specific account, similar to how a human logs in. This is particularly useful for developers building applications. An API key can be associated with a personal account, allowing the developer to absorb costs, or individual users can be required to provide their own API keys, distributing the cost. This distributed cost-sharing model allows for creative development of AI tools without the creator bearing the entire financial burden. However, API keys must be treated with extreme caution, akin to passwords, to prevent unauthorized use and associated charges.

The critical security risk of exposing API keys

Exposing API keys in code, especially in publicly accessible web applications, presents a severe security risk. If an API key is hardcoded into the source code of a website, anyone can view it by inspecting the page's source. This allows malicious actors to use the key to access AI services, incurring costs billed to the owner of the key. A common pitfall is embedding API keys directly into client-side code for web apps. Secure practices involve storing API keys in environment variables or secure configuration files that are not exposed to the public. Proper code review and security checks are essential to prevent such leaks, which can lead to unexpected expenses and misuse of AI resources.

The growing political and social opposition to AI data centers

The expansion of AI data centers is becoming a significant political issue, driven by concerns over resource consumption, particularly water and electricity, and their environmental impact. In recent elections, AI data centers were a high-profile issue, with 40 out of 41 ballot measures related to them decided against their construction. A CNBC exit poll indicated that 50% of voters considered AI data centers a crucial or major factor in their voting decisions, and a CBS poll revealed that 76% of voters would oppose a data center in their own community. The substantial spending on TV ads ($380 million) focusing on data centers further illustrates the public's strong emotions regarding this issue. This backlash is influencing political races and has even led prominent figures like Elon Musk to suggest relocating data centers to space, signaling a growing global resistance to their on-Earth deployment.

AI Usage and Cost Management Best Practices

Practical takeaways from this episode

Do This

Pay attention to input and output tokens, as well as API duration and wall clock time.
Utilize caching to reduce input token consumption and associated costs.
Be mindful of the cost incurred by enabled skills, even if not actively used.
Start fresh with 'clear' or a new chat when switching tasks to manage context.
Curate the context window carefully to avoid unnecessary token usage.
Assign cheaper models to general-purpose sub-agents.
Consider the environmental impact (water, electricity, carbon) of AI model usage.
Balance model size and capabilities with latency for better user experience, especially in multi-agent systems.
Treat API keys with the same security as passwords; do not expose them publicly.
Securely store API keys outside of public code, especially in web applications.

Avoid This

Do not let enabled skills inflate context window costs if they are not necessary.
Do not assume larger models are always better; consider latency and user experience.
Do not embed API keys directly into publicly accessible website code.
Do not share API keys or treat them less securely than personal passwords.

AI Task Energy Consumption Multipliers (Estimated)

Data extracted from this episode

Task TypeMultiplier (x text generation)Source/Notes
Image Generation28-40xEstimated, based on study in MIT Technology Review
Video Generation1,000-3,000xEstimated, based on study in MIT Technology Review
Reasoning/Thinking Mode30x (average, up to 150x or 700x for some models)Estimated, based on study in MIT Technology Review

AI Model Selection Considerations for Agents

Data extracted from this episode

Agent TypeExample ConsiderationsRecommended Model Characteristics
Input Guardrail Agent (AI Tutor)Checks for blatant asks for answers.Likely requires a smaller, cheaper model due to objective, unambiguous task.
Response Generation Agent (AI Tutor)Generates thoughtful, kind responses with citations and analogies.May require a more capable model, but latency is a key concern in multi-step workflows.
Output Guardrail Agent (AI Tutor)Checks if the response provides answers directly.Needs to be objective and unambiguous (e.g., code line limits); smaller model suitable.
Company Policy Q&A Chatbot Output GuardrailChecks generated response and citations for accuracy against company handbooks.Requires a nuanced, capable model due to complex analysis and context matching; higher cost and latency may be acceptable.

Common Questions

Input tokens are the data the AI model reads (your prompt, system instructions, etc.), while output tokens are what the AI generates. Both contribute to the cost of using AI models, as they consume computational resources and processing time.

Topics

Mentioned in this video

More from Stanford Online

View all 186 summaries

Ask anything from this episode.

Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.

Get Started Free